`send_transaction` is returning `TransactionMetadata` with invalid `inner_instructions`

[MicaiahReid]

Hi LiteSVM team! Thanks again for the awesome tool, I really appreciate the work you've done in making this.

I have a case where I am calling send_transaction and the transaction is successfully processed. However, the returned TransactionMetadata contains an inner_instructions field with a programIdIndex that is outside the bounds of the actual list of account keys.

Below are the details of the transaction and corresponding meta, but in short the programIdIndex returned is 13, while the list of account keys is 13 (so the highest possible index is 12).

Transaction & Transaction Meta Details

Example transaction from debug logs:

{ 
    signatures: [5B8qMybxMvs8d8G8PBxgt8WztoXfwo1dEaX3W7hF6DUagFHucNAs6fPf9Ds47v5epxMWHQUdcaQtovaLPCfprVqe], 
    message: V0(Message { 
        header: MessageHeader { num_required_signatures: 1, num_readonly_signed_accounts: 0, num_readonly_unsigned_accounts: 7 }, 
        account_keys: [
            txtxuHfqa62pfYscB1ScDBtAkabYLgqeyrpwKrXC9Cn,
            9s6wKTcZ5zznsF3UrWXunC4XViTgv4yv9wMW3kRst5zk, 
            9G5fDFmWz5CwitWoUEENBrx4724Dw79PYUNgNqozfDCo, 
            noVa9ymAeArErQjmrSARScuhtKoNFG32XBDuVWEr8AG, 
            5cy7wj1kR28cCM5dhxMPzzGhzuBc8CQwjyWTMcV9feaS, 
            A45GmTp4z8EzxxGG3iPwDkV8UEpZfAXicovErx9brdjy, 
            ComputeBudget111111111111111111111111111111, 
            SQDS4ep65T869zMMBKyuUq6aD6EgTu8psMjkvj52pCf, 
            5CTP4NwgNDheZwNXJ9AJR7Qr2F1ppQH5q4JD5wFk2Bad, 
            CEVVdcCWUGfEUyzbMNJx4GWBLeqjwvQ7XHmEU3AutnaU, 
            BPFLoaderUpgradeab1e11111111111111111111111, 
            SysvarC1ock11111111111111111111111111111111, 
            SysvarRent111111111111111111111111111111111
        ], 
        recent_blockhash: 2gDbZtuSkW5jBisEWacTNBkSMucNDt2bGgEa3RR1QkeV, 
        instructions: [
            CompiledInstruction { program_id_index: 6, accounts: [], data: [3, 136, 19, 0, 0, 0, 0, 0, 0] }, 
            CompiledInstruction { program_id_index: 6, accounts: [], data: [2, 64, 13, 3, 0] }, 
            CompiledInstruction { program_id_index: 7, accounts: [8, 1, 9, 0, 2, 3, 0, 4, 5, 10, 11, 12], data: [194, 8, 161, 87, 153, 164, 25, 171] }
        ], 
        address_table_lookups: [] 
    }) 
}

The transaction has 13 account keys. Then the transaction is processed with a successful result, but the meta has:

TransactionMetadata { 
    signature: 5B8qMybxMvs8d8G8PBxgt8WztoXfwo1dEaX3W7hF6DUagFHucNAs6fPf9Ds47v5epxMWHQUdcaQtovaLPCfprVqe, 
    logs: [
        "Program ComputeBudget111111111111111111111111111111 invoke [1]", 
        "Program ComputeBudget111111111111111111111111111111 success", 
        "Program ComputeBudget111111111111111111111111111111 invoke [1]", 
        "Program ComputeBudget111111111111111111111111111111 success", 
        "Program SQDS4ep65T869zMMBKyuUq6aD6EgTu8psMjkvj52pCf invoke [1]", 
        "Program log: Instruction: VaultTransactionExecute", 
        "Program BPFLoaderUpgradeab1e11111111111111111111111 invoke [2]", 
        "Upgraded program noVa9ymAeArErQjmrSARScuhtKoNFG32XBDuVWEr8AG", 
        "Program BPFLoaderUpgradeab1e11111111111111111111111 success", 
        "Program SQDS4ep65T869zMMBKyuUq6aD6EgTu8psMjkvj52pCf consumed 34305 of 199700 compute units", 
        "Program SQDS4ep65T869zMMBKyuUq6aD6EgTu8psMjkvj52pCf success"
    ], 
    inner_instructions: [
        [], 
        [], 
        [
            InnerInstruction { 
                instruction: CompiledInstruction { program_id_index: 13, accounts: [5, 3, 4, 0, 12, 11, 2], data: [3, 0, 0, 0] }, stack_height: 2 
            }
        ]
    ], 
    compute_units_consumed: 34605, 
    return_data: TransactionReturnData { program_id: BPFLoaderUpgradeab1e11111111111111111111111, data: [] } 
}

The inner_instructions' program_id_index is 13, which is out of bounds of the transaction's account_keys list. You can see from the program logs, however, that we are successfully invoking the inner instruction by calling the BPFLoaderUpgradeab1e11111111111111111111111 program. This doesn't seem to be an off-by-one bug, because if so I'd expect the program called by the inner instruction (BPFLoaderUpgradeab1e11111111111111111111111) to be the last in the account keys list, but it's not. The correct program_id_index for the inner instruction is actually 10.

Reproduction

This is consistently reproducible, but is not easy to reproduce. The transaction is a program upgrade signed by a squad multisig.

This surfpool issue has reproduction steps using txtx runbooks, but I don't currently have a rust snippet to reproduce purely with litesvm. It would be quite intensive to do so - you'd need to pull in the squads multisig contract, create a squad, create a program upgrade transaction, and sign with all squad members.

With a lack of straightforward reproduction, this probably is an edge case that you don't care about - just wanted to bring it up in case you know what could be causing this.

[Aursen]

We are using this in LiteSVM: https://github.com/anza-xyz/agave/blob/473089c19cd5e92da0da58cab0aadc4f0514c0d3/svm/src/transaction_processor.rs#L996