[MC][x86-64] Clang silently mis-assembles instructions with high-byte registers and `EVEX/VEX` prefixes in release mode and crashes with assertion enabled

[venkyqz]

Description

When using Clang's integrated assembler (LLVM-MC), an interesting issue arises when assembling an instruction that both requires an EVEX or VEX prefix for encoding and uses a high-byte register (ah, ch, dh, or bh). The fundamental incompatibility between these two encoding requirements leads to a silent misassembling bug in the release mode of Clang.

This results in a dangerous discrepancy between different build modes:

• In debug/assertion builds: The assembler's internal checks correctly identify the invalid instruction. This leads to a crash with an informative error message: "LLVM ERROR: Cannot encode high byte register in VEX/EVEX-prefixed instruction." This behavior, although it causes a crash, is useful for development as it clearly signals a fatal error.

• In release builds: The internal checks are disabled. The assembler silently proceeds with the invalid instruction, resulting in a silent mis-assembly. This behavior is problematic and dangerous as it produces a defective executable without any warning or error. The resulting binary's behavior is unpredictable, potentially leading to security vulnerabilities or silent data corruption.

Impact and Scope

• This bug is not limited to a single instruction but affects a broad range of opcodes. Any instruction that the assembler attempts to encode with an EVEX/VEX prefix will exhibit this issue if a high-byte register is used. This includes:

• Instructions that natively use EVEX/VEX: The entire ccmp* and ctest* family of instructions from AVX-512 are directly affected.

• Standard instructions that can be implicitly encoded with EVEX/VEX: Many traditional opcodes (add, sub, and, or, etc.) can be affected when the assembler chooses to use these prefixes for specific operand combinations or modern CPU features.

• The list of potentially affected opcodes is extensive, including but not limited to:

adc, add, and

All ccmp* and ctest* instructions

dec, inc, neg, not

or, sbb, shl, shr

All setzu* instructions

sub, xor

Expected Behavior

An assembler should always be robust and predictable. The correct behavior is to consistently fail with an explicit error message when given an impossible instruction, regardless of the build mode. This is demonstrated by GAS binutils, which rejects the instruction with the clear error: "Error: can't encode register 'ah' in an instruction requiring EVEX prefix." The silent mis-assembly in release mode is an unacceptable form of Undefined Behavior.

Ways to Reproduce

Godbolt Link: https://godbolt.org/z/jhP3o4v8q

Observation

• Clang 17.0.1 is the latest version that does not have that problem.
• The behaviour of trunk binutils GAS is correct.

[llvmbot]

@llvm/issue-subscribers-llvm-mc

Author: Venky Qi Zhang (venkyqz)

# Description When using Clang's integrated assembler (LLVM-MC), a critical issue arises when assembling an instruction that both requires an `EVEX or VEX prefix` for encoding and uses a high-byte register (`ah`, `ch`, `dh`, or `bh`). The fundamental incompatibility between these two encoding requirements leads to a silent misassembling bug in the release mode of

This results in a dangerous discrepancy between different build modes:

• In debug/assertion builds: The assembler's internal checks correctly identify the invalid instruction. This leads to a crash with an informative error message: "LLVM ERROR: Cannot encode high byte register in VEX/EVEX-prefixed instruction." This behavior, although it causes a crash, is useful for development as it clearly signals a fatal error.

• In release builds: The internal checks are disabled. The assembler silently proceeds with the invalid instruction, resulting in a silent mis-assembly. This behavior is problematic and dangerous as it produces a defective executable without any warning or error. The resulting binary's behavior is unpredictable, potentially leading to security vulnerabilities or silent data corruption.

Impact and Scope

• This bug is not limited to a single instruction but affects a broad range of opcodes. Any instruction that the assembler attempts to encode with an EVEX/VEX prefix will exhibit this issue if a high-byte register is used. This includes:

• Instructions that natively use EVEX/VEX: The entire ccmp* and ctest* family of instructions from AVX-512 are directly affected.

• Standard instructions that can be implicitly encoded with EVEX/VEX: Many traditional opcodes (add, sub, and, or, etc.) can be affected when the assembler chooses to use these prefixes for specific operand combinations or modern CPU features.

• The list of potentially affected opcodes is extensive, including but not limited to:

> adc, add, and
>
> All ccmp* and ctest* instructions
>
> dec, inc, neg, not
>
> or, sbb, shl, shr
>
> All setzu* instructions
>
> sub, xor

Expected Behavior

An assembler should always be robust and predictable. The correct behavior is to consistently fail with an explicit error message when given an impossible instruction, regardless of the build mode. This is demonstrated by GAS binutils, which rejects the instruction with the clear error: "Error: can't encode register 'ah' in an instruction requiring EVEX prefix." The silent mis-assembly in release mode is an unacceptable form of Undefined Behavior.

Ways to Reproduce

Godbolt Link: https://godbolt.org/z/jhP3o4v8q

Observation

• Clang 17.0.1 is the latest version that does not have that problem.
• The behaviour of trunk binutils GAS is correct.

[llvmbot]

@llvm/issue-subscribers-backend-x86

Author: Venky Qi Zhang (venkyqz)

# Description When using Clang's integrated assembler (LLVM-MC), a critical issue arises when assembling an instruction that both requires an `EVEX or VEX prefix` for encoding and uses a high-byte register (`ah`, `ch`, `dh`, or `bh`). The fundamental incompatibility between these two encoding requirements leads to a silent misassembling bug in the release mode of

This results in a dangerous discrepancy between different build modes:

• In debug/assertion builds: The assembler's internal checks correctly identify the invalid instruction. This leads to a crash with an informative error message: "LLVM ERROR: Cannot encode high byte register in VEX/EVEX-prefixed instruction." This behavior, although it causes a crash, is useful for development as it clearly signals a fatal error.

• In release builds: The internal checks are disabled. The assembler silently proceeds with the invalid instruction, resulting in a silent mis-assembly. This behavior is problematic and dangerous as it produces a defective executable without any warning or error. The resulting binary's behavior is unpredictable, potentially leading to security vulnerabilities or silent data corruption.

Impact and Scope

• This bug is not limited to a single instruction but affects a broad range of opcodes. Any instruction that the assembler attempts to encode with an EVEX/VEX prefix will exhibit this issue if a high-byte register is used. This includes:

• Instructions that natively use EVEX/VEX: The entire ccmp* and ctest* family of instructions from AVX-512 are directly affected.

• Standard instructions that can be implicitly encoded with EVEX/VEX: Many traditional opcodes (add, sub, and, or, etc.) can be affected when the assembler chooses to use these prefixes for specific operand combinations or modern CPU features.

• The list of potentially affected opcodes is extensive, including but not limited to:

> adc, add, and
>
> All ccmp* and ctest* instructions
>
> dec, inc, neg, not
>
> or, sbb, shl, shr
>
> All setzu* instructions
>
> sub, xor

Expected Behavior

An assembler should always be robust and predictable. The correct behavior is to consistently fail with an explicit error message when given an impossible instruction, regardless of the build mode. This is demonstrated by GAS binutils, which rejects the instruction with the clear error: "Error: can't encode register 'ah' in an instruction requiring EVEX prefix." The silent mis-assembly in release mode is an unacceptable form of Undefined Behavior.

Ways to Reproduce

Godbolt Link: https://godbolt.org/z/jhP3o4v8q

Observation

• Clang 17.0.1 is the latest version that does not have that problem.
• The behaviour of trunk binutils GAS is correct.

[efriedma-quic]

Probably fix should be somewhere in

llvm-project/llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp

Line 3948 in 04cd39a

bool X86AsmParser::validateInstruction(MCInst &Inst, const OperandVector &Ops) {

[llvmbot]

Hi!

This issue may be a good introductory issue for people new to working on LLVM. If you would like to work on this issue, your first steps are:

1. Check that no other contributor is working on this issue. If someone is assigned to the issue or claimed to be working on it, ping the person. After one week without a response, the assignee may be changed.
2. Leave a comment indicating that you are working on the issue, or just create a pull request after following the steps below. Mention this issue in the description of the pull request.
3. Fix the issue locally.
4. Run the test suite locally. Remember that the subdirectories under test/ create fine-grained testing targets, so you can e.g. use make check-clang-ast to only run Clang's AST tests.
5. Create a Git commit.
6. Run git clang-format HEAD~1 to format your changes.
7. Open a pull request to the upstream repository on GitHub. Detailed instructions can be found in GitHub's documentation. Mention this issue in the description of the pull request.

If you have any further questions about this issue, don't hesitate to ask via a comment in the thread below.

[llvmbot]

@llvm/issue-subscribers-good-first-issue

Author: Venky Qi Zhang (venkyqz)

# Description When using Clang's integrated assembler (LLVM-MC), a critical issue arises when assembling an instruction that both requires an `EVEX or VEX prefix` for encoding and uses a high-byte register (`ah`, `ch`, `dh`, or `bh`). The fundamental incompatibility between these two encoding requirements leads to a silent misassembling bug in the release mode of

This results in a dangerous discrepancy between different build modes:

• In debug/assertion builds: The assembler's internal checks correctly identify the invalid instruction. This leads to a crash with an informative error message: "LLVM ERROR: Cannot encode high byte register in VEX/EVEX-prefixed instruction." This behavior, although it causes a crash, is useful for development as it clearly signals a fatal error.

• In release builds: The internal checks are disabled. The assembler silently proceeds with the invalid instruction, resulting in a silent mis-assembly. This behavior is problematic and dangerous as it produces a defective executable without any warning or error. The resulting binary's behavior is unpredictable, potentially leading to security vulnerabilities or silent data corruption.

Impact and Scope

• This bug is not limited to a single instruction but affects a broad range of opcodes. Any instruction that the assembler attempts to encode with an EVEX/VEX prefix will exhibit this issue if a high-byte register is used. This includes:

• Instructions that natively use EVEX/VEX: The entire ccmp* and ctest* family of instructions from AVX-512 are directly affected.

• Standard instructions that can be implicitly encoded with EVEX/VEX: Many traditional opcodes (add, sub, and, or, etc.) can be affected when the assembler chooses to use these prefixes for specific operand combinations or modern CPU features.

• The list of potentially affected opcodes is extensive, including but not limited to:

> adc, add, and
>
> All ccmp* and ctest* instructions
>
> dec, inc, neg, not
>
> or, sbb, shl, shr
>
> All setzu* instructions
>
> sub, xor

Expected Behavior

An assembler should always be robust and predictable. The correct behavior is to consistently fail with an explicit error message when given an impossible instruction, regardless of the build mode. This is demonstrated by GAS binutils, which rejects the instruction with the clear error: "Error: can't encode register 'ah' in an instruction requiring EVEX prefix." The silent mis-assembly in release mode is an unacceptable form of Undefined Behavior.

Ways to Reproduce

Godbolt Link: https://godbolt.org/z/jhP3o4v8q

Observation

• Clang 17.0.1 is the latest version that does not have that problem.
• The behaviour of trunk binutils GAS is correct.

[woruyu]

Happy to take this on. Any pointers are welcome.

[woruyu]

I just check .intel_syntax noprefix add ah, ah, ah, which will compile success in latest release, but failed in latest debug and llvmorg-17.0.1

In latest debug, error msg is LLVM ERROR: Cannot encode high byte register in VEX/EVEX-prefixed instruction, the related code is in emitVEXOpcodePrefix function:

#ifndef NDEBUG
  unsigned NumOps = MI.getNumOperands();
  for (unsigned I = NumOps ? X86II::getOperandBias(Desc) : 0; I != NumOps;
       ++I) {
    const MCOperand &MO = MI.getOperand(I);
    if (!MO.isReg())
      continue;
    MCRegister Reg = MO.getReg();
    if (Reg == X86::AH || Reg == X86::BH || Reg == X86::CH || Reg == X86::DH)
      report_fatal_error(
          "Cannot encode high byte register in VEX/EVEX-prefixed instruction");
  }
#endif

In llvmorg-17.0.1, error msg is error: invalid operand for instruction add ah, ah, ah, the reason is there are no suitable opcode for add ah, ah, ah in tablegen function MatchInstructionImpl, the log like this

Trying to match opcode ADD64mi32
  Matching formal operand class MCK_Mem64 against actual operand at index 1 (Reg:ah): Opcode result: multiple operand mismatches, ignoring this opcode
Trying to match opcode ADD8mr
  Matching formal operand class MCK_Mem8 against actual operand at index 1 (Reg:ah): Opcode result: multiple operand mismatches, ignoring this opcode
Trying to match opcode ADD8mi
  Matching formal operand class MCK_Mem8 against actual operand at index 1 (Reg:ah): Opcode result: multiple operand mismatches, ignoring this opcode
test.s:2:2: error: invalid operand for instruction
       add ah, ah, ah

However in latest debug, the log like this

Trying to match opcode ADD8mi
  Matching formal operand class MCK_Mem8 against actual operand at index 1 (Reg:ah): Opcode result: multiple operand mismatches, ignoring this opcode
Trying to match opcode ADD8rr_ND
  Matching formal operand class MCK_GR8 against actual operand at index 1 (Reg:ah): match success using generic matcher
  Matching formal operand class MCK_GR8 against actual operand at index 2 (Reg:ah): match success using generic matcher
  Matching formal operand class MCK_GR8 against actual operand at index 3 (Reg:ah): match success using generic matcher
  Matching formal operand class InvalidMatchClass against actual operand at index 4: actual operand index out of range
Opcode result: complete match, selecting this opcode

The related .td file

// in llvmorg-17.0.1
    let Constraints = "$src1 = $dst" in {
      let isCommutable = CommutableRR in {
        let isConvertibleToThreeAddress = ConvertibleToThreeAddressRR in {
          def NAME#8rr  : BinOpRR_RF<BaseOpc, mnemonic, Xi8 , opnodeflag>;
          def NAME#16rr : BinOpRR_RF<BaseOpc, mnemonic, Xi16, opnodeflag>;
          def NAME#32rr : BinOpRR_RF<BaseOpc, mnemonic, Xi32, opnodeflag>;
          def NAME#64rr : BinOpRR_RF<BaseOpc, mnemonic, Xi64, opnodeflag>;
        } // isConvertibleToThreeAddress
      } // isCommutable

// in latest debug
multiclass ArithBinOp_RF<bits<8> BaseOpc, bits<8> BaseOpc2, bits<8> BaseOpc4,
                         string mnemonic, Format RegMRM, Format MemMRM,
                         SDNode opnodeflag, SDNode opnode,
                         bit CommutableRR, bit ConvertibleToThreeAddress,
                         bit ConvertibleToThreeAddressRR> {
  let isCommutable = CommutableRR,
      isConvertibleToThreeAddress = ConvertibleToThreeAddressRR in {
    let Predicates = [NoNDD] in {
      def 8rr  : BinOpRR_RF<BaseOpc, mnemonic, Xi8 , opnodeflag>;
      def 16rr : BinOpRR_RF<BaseOpc, mnemonic, Xi16, opnodeflag>, OpSize16;
      def 32rr : BinOpRR_RF<BaseOpc, mnemonic, Xi32, opnodeflag>, OpSize32;
      def 64rr : BinOpRR_RF<BaseOpc, mnemonic, Xi64, opnodeflag>;
    }
    let Predicates = [HasNDD, In64BitMode] in {
      def 8rr_ND  : BinOpRR_RF<BaseOpc, mnemonic, Xi8 , opnodeflag, 1>;
      def 16rr_ND : BinOpRR_RF<BaseOpc, mnemonic, Xi16, opnodeflag, 1>, PD;
      def 32rr_ND : BinOpRR_RF<BaseOpc, mnemonic, Xi32, opnodeflag, 1>;
      def 64rr_ND : BinOpRR_RF<BaseOpc, mnemonic, Xi64, opnodeflag, 1>;
      def 8rr_NF_ND  : BinOpRR_R<BaseOpc, mnemonic, Xi8, 1>, EVEX_NF;
      def 16rr_NF_ND : BinOpRR_R<BaseOpc, mnemonic, Xi16, 1>, EVEX_NF, PD;
      def 32rr_NF_ND : BinOpRR_R<BaseOpc, mnemonic, Xi32, 1>, EVEX_NF;
      def 64rr_NF_ND : BinOpRR_R<BaseOpc, mnemonic, Xi64, 1>, EVEX_NF;
    }

So in latest debug llvm, we add ADD8rr_ND instruction, so it pass MatchInstructionImpl and get into emitVEXOpcodePrefix function, so this is not a regression error, but a bug that was introduced after NDD without considering the release. So how should it be fixed? Just delete #ifndef NDEBUG?

Friendly ping @KanRobert .