When inserting text, we should not use void elements (such as <br>) as the target for insertion but rather move up the tree to its parent. I suspect we need to add a step between step 6 and 7 of the inserttext command stating that if node is a void element, set offset to node's index and node to node's parent.
Test case:
<!DOCTYPE html>
<div id="editor" contenteditable>
<div><br></div>
</div>
<button onclick="test()">Insert "hello" at selection inside <br></button>
<pre id="result"></pre>
<script>
function test() {
const br = editor.querySelector('br');
getSelection().collapse(br, 0); // collapse selection inside the <br>
document.execCommand('insertText', false, 'hello');
result.textContent = 'Result: ' + editor.innerHTML;
}
</script>