SIMD-0266: Efficient Token program

[febo]

Replace the current version of SPL Token (TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA) program by a CU-optimized one (p-token).

[zfedoran]

First of all, awesome work getting to this point.

As an optimization this makes a lot of sense. It means more compute for other things. I’m on board with the idea as it seems to be a natural progression beyond making SVM optimizations specifically for the token program, this seems way cleaner.

If I had to provide a critique, it would be that I think the changes and additions could be a separate SIMD. I think there is a lot more opportunity to improve the existing design.

Slightly off topic but I like that you’ve pointed out that logging is expensive and could be removed. Perhaps this could be more effective as a different mechanism that may be useful for other programs.

Anyways, I like this proposal. I’m sure there will be opposition as we are talking about a closed program.

[lostintime101]

I ran some queries to try and find the amount of SOL bricked in token mint accounts.

~1.5 million token mint accounts hold more SOL than the rent exempt amount of 0.0014616.
https://dune.com/queries/4885823/8090505?category=abstraction&blockchain=solana&namespace=tokens_solana

Total SOL bricked in all token mint accounts is 167,282.47 ($21.6 million USD at today's price)
https://dune.com/queries/4886129?category=materialized_views&id=dune.heliuslabs.result_token_mint_accounts_sol_balances

[pooriagg]

We cannot assure that this program is 100% safe, it is too risky to change the current Token-Program! Furthermore current token-programs are in good state and they are mush more readable (developer-friendly) than "P-Token-Program".

[deanmlittle]

We cannot assure that this program is 100% safe, it is too risky to change the current Token-Program! Furthermore current token-programs are in good state and they are mush more readable (developer-friendly) than "P-Token-Program".

You also cannot assure the SPL token program is 100% safe. What you can do is create a comprehensive test suite proving equivalence. That's exactly what PToken has done. Concerns about "developer friendliness" are only really relevant if you're planning to fork the program, as all existing APIs and SDKs still work as usual, they just burn far fewer CUs, and of course, even with a PToken upgrade, you can still fork and use SPL. Have you actually looked into the code and identified any areas of concern or tried running the tests? It's already very convincing, and I can assure you that this will never make it anywhere near mainnet activation without extremely heavy scrutiny.

[fikunmi-ap]

We cannot assure that this program is 100% safe, it is too risky to change the current Token-Program! Furthermore current token-programs are in good state and they are mush more readable (developer-friendly) than "P-Token-Program".

Developers use this program through APIs. They have no business looking at what's underneath. The developer friendliness concern is unfounded.

[stegaBOB]

Also assuming it's 1:1 API wise, older clients would still work perfectly fine from a devex perspective. I am definitely in favor of this change, minus the additional instructions. While I'm probably ultimately in favor of those as well, I think they should be in a separate SIMD.

[pooriagg]

We cannot assure that this program is 100% safe, it is too risky to change the current Token-Program! Furthermore current token-programs are in good state and they are mush more readable (developer-friendly) than "P-Token-Program".

You also cannot assure the SPL token program is 100% safe. What you can do is create a comprehensive test suite proving equivalence. That's exactly what PToken has done. Concerns about "developer friendliness" are only really relevant if you're planning to fork the program, as all existing APIs and SDKs still work as usual, they just burn far fewer CUs, and of course, even with a PToken upgrade, you can still fork and use SPL. Have you actually looked into the code and identified any areas of concern or tried running the tests? It's already very convincing, and I can assure you that this will never make it anywhere near mainnet activation without extremely heavy scrutiny.

Thanks for the reply BUT i don't think we can compare the safety of SPL Token Programs to the P-Token, SPL-Token programs all are battle-tested for years. The only think i want to mention is that the Token-Programs are the backbone of the Solana and it has to be secure almost 100%. I totally agree with you.

[mertimus]

LGTM

[deanmlittle]

Also assuming it's 1:1 API wise, older clients would still work perfectly fine from a devex perspective. I am definitely in favor of this change, minus the additional instructions. While I'm probably ultimately in favor of those as well, I think they should be in a separate SIMD.

While I agree in principle that they are separate matters, I also think it's quite disruptive and impractical to have token program upgrades every other month. While most people appear to be on board, it's still contentious, causes uncertainty and should happen as infrequently as possible. I believe we all agree that the north star is IBRL. While recovering lamports may not be so relevant to that, the batch instruction is the most IBRL instruction in the whole program. I think it is far more practical for anyone who has any reasonable concerns about the 2 additional instructions, especially evidence-based ones, to address them thoroughly here than it is to fork the conversation.

[micheltriana]

This is a no-brainer. Let’s go!

[aeyakovenko]

Thanks for the reply BUT i don't think we can compare the safety of SPL Token Programs to the P-Token, SPL-Token programs all are battle-tested for years. The only think i want to mention is that the Token-Programs are the backbone of the Solana and it has to be secure almost 100%. I totally agree with you.

spl-token has been hard fork upgraded to fix security bugs before already. You can look at the git history. Equivalence test plus formal verification would make this version as secure as spl-token itself based on all available information.

[devtonic0]

Thanks for the reply BUT i don't think we can compare the safety of SPL Token Programs to the P-Token, SPL-Token programs all are battle-tested for years. The only think i want to mention is that the Token-Programs are the backbone of the Solana and it has to be secure almost 100%. I totally agree with you.

This is a very interesting perspective, but also very anti improvement..

While it may be a risk to upgrade a program that has been battle tested for years, that in no case means that it should never be done. Realistically, token extensions was intended to become the new standard in tokens on Solana, but it didn't. The SPL token program is more used than ever, and token extensions are very rarely used. In my personal opinion the SPL token program is the better program, mostly due to sheer simplicity, and the lack of need to over complicate tokens on Solana.

The spl token program is in need of upgrades though, this program shows that not only can it be optimized in a lot of ways that will reduce load on the entire network.. but it also solves issues that have been ignored for a very long time. There are millions of transactions each and every day using the spl token program, it's one of the most if not the most used program on Solana. If we can reduce the CUs in any way, there is absolutely no reason why we should not support this. It can be tested, security can be guaranteed to be equivalent to the current program.

Avoiding it because of concerns like that is senseless, and with thinking like that there wouldn't have even been a token program in the first place.

[mschneider]

if the program could log the destination account owner on transfer, it would probably reduce off-chain indexing cost even more dramatically than it increases block space.

[febo]

if the program could log the destination account owner on transfer, it would probably reduce off-chain indexing cost even more dramatically than it increases block space.

Having the log of the owner adds between 104 to 208 extra CUs to the transfer instruction, depending whether only the pubkey is logged (one log message) or a "description" + pubkey (two log messages). This would mean at least a 40% increase in CUs for transfers – that is not necessary a small increase.

How much easier/efficient would it make indexing?

[febo]

The latest pinocchio improvements have lowered the CUs of the instructions further when logging is disabled:

Instruction	p-token -logs (previous)	p-token -logs (new)
InitializeMint	118	112
InitializeAccount	170	143
Transfer	146	131
MintTo	144	126
Burn	202	135
CloseAccount	152	132

Logging the name of the instruction ("Instruction: <name>") takes about ~103 CUs, which is about 40% of the CUs of a transfer. Since these logs are not necessarily very reliable – e.g., they can be truncated, you could "log-inject" matching messages that can "confuse" the parsing – perhaps we should consider disabling them.

Does anyone see a blocker for doing this?

[stevenlusonggao]

Is there a SPL-2022 version of this aswell?

[wjthieme]

Great to see! I fully support this SIMD but here's a comment from the perspective of a dapp dev:

While I fully agree with the first point, I don't think dapp devs will benefit much from this (by itself).

• Large/complex Transaction are usually constrained by the tx size limit and not the CU limit. ALTs are in most cases an imperfect solution (yeey for SIMD-0296).
• When transactions are constrained by the CU limit this is usually because an instruction is doing complex/repetitive math (e.g. crossing a lot of ticks/bins in a CLAMM). The token transfer part of the transaction only consumes a very small percentage of the CUs so decreasing that won't have much impact.

[mschneider]

well you gotta start somewhere, a win is a win.

[devtonic0]

My input on this is that its not about that, its more about the network as a whole.

Spl token program is the most used program on Solana, a huge percentage of each transaction is using this program. Reducing the CUs consumed by each instruction significantly means reducing the consumed CUs across the board. Allowing for higher speeds, lower costs, and more

[mschneider]

Does anyone see a blocker for doing this?

🚢🚢🚢

[topointon-jump]

Firedancer are very supportive of this effort 🚀

[iownducks]

approved by ducks..

[SOELTH]

Temporal is strongly in favor

[ZeroTimeDrift]

we test this in production

[devtonic0]

We almost there cant wait to see @febo hard work in action. The average user cannot comprehend the load that will be lifted from this. Significant network improvement

[Mantistc]

ship it!

[febo]

Updated the SIMD with the latest CU consumption values, indicated that we are omitting instruction name logs and added details of a new unwrap_lamports instruction.

[0xDeeep]

LGTM!

[itsmodsiw]

Great upgrade 🕸️

[MonteCrypto999]

Yo why this is not already on the devnet? Need this

[imduchuyyy]

Great update. Do we have it on testnet?

[simd-bot]

Thanks, topointon-jump!

⚠️ Status: Cannot merge yet

• Missing approval from: Anza (t-nelson, sakridge, or bw-solana)

[simd-bot]

Thanks, topointon-jump!

⚠️ Status: Cannot merge yet

• Missing approval from: Anza (t-nelson, sakridge, or bw-solana)