[x86][MC] Fail to decode some long multi-byte NOPs

[venkyqz]

Work environment

Questions	Answers
OS/arch/bits	x86_64 Ubuntu 20.04
Architecture	x86_64
Source of Capstone	git clone, default on master branch.
Version/git commit	llvm-20git, f08278

minimum PoC disassembler

#include <llvm-c/Disassembler.h>
#include <llvm-c/Target.h>

int main(int argc, char *argv[]){
    /*
       some input sanity check of hex string from argv
    */
    // Initialize LLVM after input validation
    LLVMInitializeAllTargetInfos();
    LLVMInitializeAllTargets();
    LLVMInitializeAllTargetMCs();
    LLVMInitializeAllDisassemblers();

    LLVMDisasmContextRef disasm = LLVMCreateDisasm("x86_64", NULL, 0, NULL, NULL);
    if (!disasm) {
        errx(1, "Error: LLVMCreateDisasm() failed.");
    }

    // Set disassembler options: print immediates as hex, use Intel syntax
    if (!LLVMSetDisasmOptions(disasm, LLVMDisassembler_Option_PrintImmHex |
                                        LLVMDisassembler_Option_AsmPrinterVariant)) {
        errx(1, "Error: LLVMSetDisasmOptions() failed.");
    }

    char output_string[MAX_OUTPUT_LENGTH];
    uint64_t address = 0;
    size_t instr_len = LLVMDisasmInstruction(disasm, raw_bytes, bytes_len, address,
                                             output_string, sizeof(output_string));

    if (instr_len > 0) {
        printf("%s\n", output_string);
    } else {
        printf("Error: Unable to disassemble the input bytes.\n");
    }
}

Instruction bytes giving faulty results

0f 1a de

Expected results

It should be:

nop esi, ebx

Actually results

$./min_llvm_disassembler "0f1ade"
Error: Unable to disassemble the input bytes.

Other cases seem to work

$./min_llvm_disassembler "0f1f00"
nop     dword ptr [rax]

Additional Logs, screenshots, source code, configuration dump, ...

Instructions with opcodes ranging from 0f 18 to 0f 1f are defined as multi-byte NOP (No Operation) instructions in x86 ISA. Please refer to the StackOverflow post for more details. It should be decoded in the following logic.

• "0x0f 0x1a" is extended opcode.
• The ModR/M byte DE translates to binary 11011110 (0xde).
  • Bits 7-6 (Mod): 11 (binary) = 3 (decimal)
    Indicates register-direct addressing mode.
  • Bits 5-3 (Reg): 011 (binary) = 3 (decimal)
    Corresponds to the EBX (or RBX in 64-bit mode) register.
  • Bits 2-0 (R/M): 110 (binary) = 6 (decimal)
    Corresponds to the ESI (or RSI in 64-bit mode) register.

XED also translates "0f 1a de" into "nop esi, ebx".

[llvmbot]

@llvm/issue-subscribers-backend-x86

Author: VinkyQZ (Mar3yZhang)

### Work environment

Questions	Answers
OS/arch/bits	x86_64 Ubuntu 20.04
Architecture	x86_64
Source of Capstone	git clone, default on master branch.
Version/git commit	llvm-20git, f08278

<!-- INCORRECT DISASSEMBLY BUGS -->

minimum PoC disassembler

#include <llvm-c/Disassembler.h>
#include <llvm-c/Target.h>

int main(int argc, char *argv[]){
    /*
       some input sanity check of hex string from argv
    */
    // Initialize LLVM after input validation
    LLVMInitializeAllTargetInfos();
    LLVMInitializeAllTargets();
    LLVMInitializeAllTargetMCs();
    LLVMInitializeAllDisassemblers();

    LLVMDisasmContextRef disasm = LLVMCreateDisasm("x86_64", NULL, 0, NULL, NULL);
    if (!disasm) {
        errx(1, "Error: LLVMCreateDisasm() failed.");
    }

    // Set disassembler options: print immediates as hex, use Intel syntax
    if (!LLVMSetDisasmOptions(disasm, LLVMDisassembler_Option_PrintImmHex |
                                        LLVMDisassembler_Option_AsmPrinterVariant)) {
        errx(1, "Error: LLVMSetDisasmOptions() failed.");
    }

    char output_string[MAX_OUTPUT_LENGTH];
    uint64_t address = 0;
    size_t instr_len = LLVMDisasmInstruction(disasm, raw_bytes, bytes_len, address,
                                             output_string, sizeof(output_string));

    if (instr_len > 0) {
        printf("%s\n", output_string);
    } else {
        printf("Error: Unable to disassemble the input bytes.\n");
    }
}

Instruction bytes giving faulty results

0f 1a de

Expected results

It should be:

nop esi, ebx

Actually results

$./min_llvm_disassembler "0f1ade"
Error: Unable to disassemble the input bytes.

Other cases seem to work

$./min_llvm_disassembler "0f1f00"
nop     dword ptr [rax]

<!-- ADDITIONAL CONTEXT -->

Additional Logs, screenshots, source code, configuration dump, ...

Instructions with opcodes ranging from 0f 18 to 0f 1f are defined as multi-byte NOP (No Operation) instructions in x86 ISA. Please refer to the StackOverflow post for more details. It should be decoded in the following logic.

• "0x0f 0x1a" is extended opcode.
• The ModR/M byte DE translates to binary 11011110 (0xde).
  • Bits 7-6 (Mod): 11 (binary) = 3 (decimal)
    Indicates register-direct addressing mode.
  • Bits 5-3 (Reg): 011 (binary) = 3 (decimal)
    Corresponds to the EBX (or RBX in 64-bit mode) register.
  • Bits 2-0 (R/M): 110 (binary) = 6 (decimal)
    Corresponds to the ESI (or RSI in 64-bit mode) register.

XED also translates "0f 1a de" into "nop esi, ebx".

[PhilippRados]

@RKSimon I've looked into this (I'm pretty new to llvm) and "fixed" the mentioned issue by adding the corresponding opcodes to the X86InstrMisc.td file that encode a nop:

let hasSideEffects = 0, SchedRW = [WriteNop] in {
  def NOOP : I<0x90, RawFrm, (outs), (ins), "nop", []>;
  def NOOPW : I<0x1f, MRMXm, (outs), (ins i16mem:$zero),
                "nop{w}\t$zero", []>, TB, OpSize16;
  def NOOPL_A : I<0x1a, MRMXm, (outs), (ins i32mem:$zero), // <- new def, would need to be added for all 0x19-0x1e
                "nop{l}\t$zero", []>, TB, OpSize32;
  def NOOPL : I<0x1f, MRMXm, (outs), (ins i32mem:$zero),
                "nop{l}\t$zero", []>, TB, OpSize32;
  def NOOPQ : RI<0x1f, MRMXm, (outs), (ins i64mem:$zero),
                "nop{q}\t$zero", []>, TB, Requires<[In64BitMode]>;
...

Is this a fine approach or is there some less verbose way?

This change should only affect the disassembler though (The tests are still running) as the existing code issues the regular nop (0x1f) as recommended by the intel software developers manual (https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html).

While testing the faulty example from above "0f 1a de" the disassembler only emitted "nop %esi" without the second register from the R/M section. It seems like this format isn't supported by LLVM right?

Another thing I encountered were other instructions like "bndldx" that would override the nop as shown in this table from the intel manual mentioned above:
However none of these seem currently supported by LLVM so it seems fine to just emit the remaining defs to the code from above.

I'm just writing down my findings so any guidance/input is appreciated.

[PhilippRados]

After further reading I am not sure if these should be implemented as the Intel manual (October 2024) mentioned above says:

Appendix A, A.1: Software should not use opcodes corresponding blank cells or cells marked “Reserved-NOP” nor depend on the current behavior of those opcodes.

I'm not sure if this only means that these shouldn't be emitted or that they also shouldn't be implemented by a disassembler.

[phoebewang]

However none of these seem currently supported by LLVM so it seems fine to just emit the remaining defs to the code from above.

It was supported by LLVM and removed by c2d4fe5 after it's discontinued.

Appendix A, A.1: Software should not use opcodes corresponding blank cells or cells marked “Reserved-NOP” nor depend on the current behavior of those opcodes.

I'm not sure if this only means that these shouldn't be emitted or that they also shouldn't be implemented by a disassembler.

I suggest we shouldn't implement them. GAS doesn't decode them well either.