[X86] Ignore REX prefixes not immediately before opcode (llvm#117299)

The Intel X86 Architecture Manual says the following:

> A REX prefix is ignored, as are its individual bits, when it is not
needed
> for an instruction or when it does not immediately precede the opcode
byte or
> the escape opcode byte (0FH) of an instruction for which it is needed.
This
> has the implication that only one REX prefix, properly located, can
affect an
> instruction.

We currently do not handle these cases in the disassembler, leading to
incorrect disassembly. This patch rectifies the situation by treating
REX prefixes as standard prefixes rather than only expecting them before
the Opcode.

The motivating test case added as a test was fuzzer generated.

Author: boomanaiden154

llvm/lib/Target/X86/Disassembler/X86Disassembler.cpp

@@ -329,6 +329,14 @@ static int readPrefixes(struct InternalInstruction *insn) {
       break;
     }
 
+    if (isREX(insn, byte)) {
+      insn->rexPrefix = byte;
+      isPrefix = true;
+      LLVM_DEBUG(dbgs() << format("Found REX prefix 0x%hhx", byte));
+    } else if (isPrefix) {
+      insn->rexPrefix = 0;
+    }
+
     if (isPrefix)
       LLVM_DEBUG(dbgs() << format("Found prefix 0x%hhx", byte));
   }
@@ -506,11 +514,6 @@ static int readPrefixes(struct InternalInstruction *insn) {
     LLVM_DEBUG(dbgs() << format("Found REX2 prefix 0x%hhx 0x%hhx",
                                 insn->rex2ExtensionPrefix[0],
                                 insn->rex2ExtensionPrefix[1]));
-  } else if (isREX(insn, byte)) {
-    if (peek(insn, nextByte))
-      return -1;
-    insn->rexPrefix = byte;
-    LLVM_DEBUG(dbgs() << format("Found REX prefix 0x%hhx", byte));
   } else
     --insn->readerCursor;
 

llvm/test/MC/Disassembler/X86/x86-64.txt

@@ -770,3 +770,9 @@
 
 # CHECK: prefetchit1 (%rip)
 0x0f,0x18,0x35,0x00,0x00,0x00,0x00
+
+# Check that we correctly ignore a REX prefix that is not immediately before
+# the opcode. REX prefixes not immediately preceding the Opcode are ignored
+# according to Section 2.2.1 of the Intel 64 Architecture Manual.
+# CHECK: orw $25659, %ax
+0x66 0x4c 0x64 0x0d 0x3b 0x64