stake-pool: Add / remove validators from the reserve (#3714)

* stake-pool: Create validator stake accounts from reserve

* Update accounting to take minimum into account

* Refactor withdraw tests

* Add ability to remove validators during withdraw stake

* Add seed to validator stake account address derivation

* Update CLI with new instruction formats

* Update documentation and helper scripts for new funding

* Update Python bindings

* Try to fix flakey test

* Support token-2022

* Condense tests for CI

* Reduce huge pool limit post-rebase

* Avoid draining the whole reserve

* Restrict account extensions on allow-list

* Update comments to say "lamports"

* Fixup code comments and variable names

* Fix clippy in tests

Author: joncinque

Cargo.lock

@@ -261,7 +261,11 @@ In order to accommodate large numbers of user deposits into the stake pool, the
 stake pool only manages one stake account per validator. To add a new validator
 to the stake pool, the staker must use the `add-validator` command.
 
-Let's add some random validators to the stake pool.
+The SOL used to add validators to the pool comes from the stake pool's reserve
+account. If there is insufficient SOL in the reserve, the command will fail.
+Be sure to use the `deposit-sol` command to move some SOL into the pool.
+
+With 10 SOL in the pool, let's add some random validators to the stake pool.
 
 ```console
 $ spl-stake-pool add-validator Zg5YBPAk8RqBR9kaLLSoN5C8Uv7nErBz1WC63HTsCPR 38DYMkwYCvsj8TC6cNaEvFHHVDYeWDp1qUgMgyjNqZXk
@@ -285,18 +289,18 @@ We can see the status of a stake account using the Solana command-line utility.
 
 ```console
 $ solana stake-account 5AaobwjccyHnXhFCd24uiX6VqPjXE3Ry4o92fJjqqjAr
-Balance: 0.00328288 SOL
+Balance: 1.00228288 SOL
 Rent Exempt Reserve: 0.00228288 SOL
-Delegated Stake: 0.001 SOL
+Delegated Stake: 1 SOL
 Active Stake: 0 SOL
-Activating Stake: 0.001 SOL
+Activating Stake: 1 SOL
 Stake activates starting from epoch: 5
 Delegated Vote Account Address: J3xu64PWShcMen99kU3igxtwbke2Nwfo8pkZNRgrq66H
 Stake Authority: DS3AyFN9dF1ruNBcSeo8XXQR8UyVMhcCPcnjU5GnY18S
 Withdraw Authority: DS3AyFN9dF1ruNBcSeo8XXQR8UyVMhcCPcnjU5GnY18S
 ```
 
-The stake pool creates these special staking accounts with 0.001 SOL as the required
+The stake pool creates these special staking accounts with 1 SOL as the required
 minimum delegation amount. The stake and withdraw authorities are the stake pool
 withdraw authority, program addresses derived from the stake pool's address.
 
@@ -312,29 +316,28 @@ Stake Deposit Fee: none
 SOL Deposit Fee: none
 SOL Deposit Referral Fee: none
 Stake Deposit Referral Fee: none
-Reserve Account: EN4px2h4gFkYtsQUi4yeCYBrdRM4DoRxCVJyavMXEAm5   Available Balance: ◎0.000000000
-Vote Account: EhRbKi4Vhm1oUCGWHiLEMYZqDrHwEd7Jgzgi26QJKvfQ      Balance: ◎0.000000000 Last Update Epoch: 4
-Vote Account: J3xu64PWShcMen99kU3igxtwbke2Nwfo8pkZNRgrq66H      Balance: ◎0.000000000  Last Update Epoch: 4
-Vote Account: 38DYMkwYCvsj8TC6cNaEvFHHVDYeWDp1qUgMgyjNqZXk      Balance: ◎0.000000000  Last Update Epoch: 4
-Total Pool Stake: ◎0.000000000
-Total Pool Tokens: 0.00000000
+Reserve Account: EN4px2h4gFkYtsQUi4yeCYBrdRM4DoRxCVJyavMXEAm5   Available Balance: ◎6.99315136
+Vote Account: EhRbKi4Vhm1oUCGWHiLEMYZqDrHwEd7Jgzgi26QJKvfQ      Balance: ◎1.002282880 Last Update Epoch: 4
+Vote Account: J3xu64PWShcMen99kU3igxtwbke2Nwfo8pkZNRgrq66H      Balance: ◎1.002282880 Last Update Epoch: 4
+Vote Account: 38DYMkwYCvsj8TC6cNaEvFHHVDYeWDp1qUgMgyjNqZXk      Balance: ◎1.002282880 Last Update Epoch: 4
+Total Pool Stake: ◎10.000000000
+Total Pool Tokens: 10.00000000
 Current Number of Validators: 3
 Max Number of Validators: 1000
 ```
 
 To make reading easier, the tool will not show balances that cannot be touched by
-the stake pool. The stake account `5AaobwjccyHnXhFCd24uiX6VqPjXE3Ry4o92fJjqqjAr`,
-delegated to `J3xu64PWShcMen99kU3igxtwbke2Nwfo8pkZNRgrq66H`, actually has a balance
-of 0.00328288 SOL, but since this is the minimum required amount, it is
-not shown by the CLI.
+the stake pool. The reserve stake account `EN4px2h4gFkYtsQUi4yeCYBrdRM4DoRxCVJyavMXEAm5`
+actually has an additional balance of 0.002282881 SOL, but since this is the minimum
+required amount, it is not shown by the CLI.
 
 ### Remove validator stake account
 
 If the stake pool staker wants to stop delegating to a vote account, they can
 totally remove the validator stake account from the stake pool.
 
 As with adding a validator, the validator stake account must have exactly
-0.00328288 SOL (0.001 SOL delegated, 0.00228288 SOL for rent exemption) to be removed.
+1.00228288 SOL (1 SOL delegated, 0.00228288 SOL for rent exemption) to be removed.
 
 If that is not the case, the staker must first decrease the stake to that minimum amount.
 Let's assume that the validator stake account delegated to 
@@ -356,29 +359,18 @@ Creating account to receive stake nHEEyey8KkgHuVRAUDzkH5Q4PkA4veSHuTxgG6C8L2G
 Signature: 4XprnR768Ch6LUvqUVLTjMCiqdYvtjNfECh4izErqwbsASTGjUBz7NtLZHAiraTqhs7b9PoSAazetdsgXa6J4wVu
 ```
 
-Unlike a normal withdrawal, the validator stake account is totally moved from
-the stake pool and into a new account belonging to the administrator.
-
-Note: since removal is only possible when the validator stake is at the minimum
-amount of 0.00328288, the administrator does not get any control of user funds,
-and only recovers the amount contributed during `add-validator`.
-
-The authority for the withdrawn stake account can also be specified using the
-`--new-authority` flag:
-
-```console
-$ spl-stake-pool remove-validator Zg5YBPAk8RqBR9kaLLSoN5C8Uv7nErBz1WC63HTsCPR J3xu64PWShcMen99kU3igxtwbke2Nwfo8pkZNRgrq66H --new-authority 4SnSuUtJGKvk2GYpBwmEsWG53zTurVM8yXGsoiZQyMJn
-Signature: 5rrQ3xhDWyiPkUTAQkNAeq31n6sMf1xsg2x9hVY8Vj1NonwBnhxuTv87nADLkwC8Xzc4CGTNCTX2Vph9esWnXk2d
-```
+Unlike a normal withdrawal, the validator stake account is deactivated, and then
+merged into the reserve during the next epoch.
 
-We can check the removed stake account:
+We can check the deactivating stake account:
 
 ```console
 $ solana stake-account nHEEyey8KkgHuVRAUDzkH5Q4PkA4veSHuTxgG6C8L2G
-Balance: 0.003282880 SOL
-Rent Exempt Reserve: 0.00328288 SOL
-Delegated Stake: 0.001000000 SOL
-Active Stake: 0.001000000 SOL
+Balance: 1.002282880 SOL
+Rent Exempt Reserve: 0.00228288 SOL
+Delegated Stake: 1.000000000 SOL
+Active Stake: 1.000000000 SOL
+Stake deactivates starting from epoch: 10
 Delegated Vote Account Address: J3xu64PWShcMen99kU3igxtwbke2Nwfo8pkZNRgrq66H
 Stake Authority: 4SnSuUtJGKvk2GYpBwmEsWG53zTurVM8yXGsoiZQyMJn
 Withdraw Authority: 4SnSuUtJGKvk2GYpBwmEsWG53zTurVM8yXGsoiZQyMJn
@@ -812,11 +804,28 @@ be able to withdraw their funds.
 
 To get around this case, it is also possible to withdraw from the stake pool's
 reserve, but only if all of the validator stake accounts are at the minimum amount of
-`0.001 SOL + stake account rent exemption`.
+`1 SOL + stake account rent exemption`.
 
 ```console
 $ spl-stake-pool withdraw-stake Zg5YBPAk8RqBR9kaLLSoN5C8Uv7nErBz1WC63HTsCPR 5 --use-reserve
 Withdrawing ◎5.000000000, or 5 pool tokens, from stake account J5XB7mWpeaUZxZ6ogXT57qSCobczx27vLZYSgfSbZoBB
 Creating account to receive stake 51XdXiBSsVzeuY79xJwWAGZgeKzzgFKWajkwvWyrRiNE
 Signature: yQH9n7Go6iCMEYXqWef38ZYBPwXDmbwKAJFJ4EHD6TusBpusKsfNuT3TV9TL8FmxR2N9ExZTZwbD9Njc3rMvUcf
 ```
+
+#### Special case: removing validator from the pool
+
+Since the funds used to add validators to the pool come from outside deposits,
+it's possible for a delinquent or malicious staker to make it impossible for
+users to reclaim their SOL by keeping everything at the minimum amount.
+
+To get around this case, it is also possible to remove a validator from the stake pool
+but only if all of the validator stake accounts are at the minimum amount of
+`1 SOL + stake account rent exemption`.
+
+```console
+$ spl-stake-pool withdraw-stake Zg5YBPAk8RqBR9kaLLSoN5C8Uv7nErBz1WC63HTsCPR 1.00228288 SOL
+Withdrawing ◎1.00228288 or 1.00228288 pool tokens, from stake account J5XB7mWpeaUZxZ6ogXT57qSCobczx27vLZYSgfSbZoBB
+Creating account to receive stake 51XdXiBSsVzeuY79xJwWAGZgeKzzgFKWajkwvWyrRiNE
+Signature: yQH9n7Go6iCMEYXqWef38ZYBPwXDmbwKAJFJ4EHD6TusBpusKsfNuT3TV9TL8FmxR2N9ExZTZwbD9Njc3rMvUcf
+```

docs/src/stake-pool/cli.md

@@ -32,7 +32,7 @@ This document is intended for the main actors of the stake pool system:
 
 * manager: creates and manages the stake pool, earns fees, can update the fee, staker, and manager
 * staker: adds and removes validators to the pool, rebalances stake among validators
-* user: provides staked SOL into an existing stake pool
+* user: provides liquid or staked SOL into an existing stake pool
 
 In its current iteration, the stake pool accepts active stakes or SOL, so
 deposits may come from either an active stake or SOL wallet. Withdrawals
@@ -56,17 +56,16 @@ providers for direct SOL deposits.
 
 ## Operation
 
-A stake pool manager creates a stake pool, and the staker includes validators that will
-receive delegations from the pool by adding "validator stake accounts" to the pool
-using the `add-validator` instruction. In this command, the stake pool creates
-a new stake account and delegates it to the desired validator.
+A stake pool manager creates a stake pool. At this point, users can immediately
+participate with SOL deposits with the `deposit-sol` instruction, moving funds
+into the reserve in exchange for pool tokens.
 
-At this point, users can participate with deposits. They can directly deposit
-SOL into the stake pool using the `deposit-sol` instruction. Within this instruction,
-the stake pool will move SOL into the pool's reserve account, to be redistributed
-by the staker.
+Using those SOL deposits, the staker includes validators that will receive
+delegations from the pool by adding "validator stake accounts" to the pool
+using the `add-validator` instruction. In this command, the stake pool uses
+reserve funds to create a new stake account and delegate it to the desired validator.
 
-Alternatively, users can deposit a stake account into the pool.  To do this,
+At this point, users can also deposit a stake account into the pool.  To do this,
 they must delegate a stake account to the one of the validators in the stake pool.
 If the stake pool has a preferred deposit validator, the user must delegate their
 stake to that validator's vote account.
@@ -101,10 +100,8 @@ The stake pool staker can add and remove validators, or rebalance the pool by
 decreasing the stake on a validator, waiting an epoch to move it into the stake
 pool's reserve account, then increasing the stake on another validator.
 
-The staker operation to add a new validator requires 0.00328288 SOL to create
-the stake account on a validator, so the stake pool staker will need liquidity
-on hand to fully manage the pool stakes.  The SOL used to add a new validator
-is recovered when removing the validator.
+The staker operation to add a new validator requires 1.00228288 SOL to create
+the stake account on a validator, so the stake pool reserve needs liquidity.
 
 ### Funding restrictions
 
@@ -153,6 +150,7 @@ When processing withdrawals, the order of priority goes:
 * validator stake accounts
 * transient stake accounts
 * reserve stake account
+* removing validator stake accounts entirely
 
 If there is preferred withdraw validator, and that validator stake account has
 any SOL, a user must withdraw from that account.
@@ -165,7 +163,7 @@ staker decreases the stake on all validators at once, then the user must withdra
 from any transient stake account.
 
 If all transient stake accounts are empty, then the user must withdraw from the
-reserve.
+reserve or completely remove a validator stake account.
 
 In this way, a user's funds are never at risk, and always redeemable.
 

docs/src/stake-pool/overview.md

@@ -117,28 +117,50 @@ Carefully read through the [Fees](fees.md) for more information about fees and
 best practices.
 
 In our example, we will use fees of 0.3%, a referral fee of 50%, opt to *not*
-set a deposit authority, and have the maximum number of validators (2,950).  Next,
-run the script:
+set a deposit authority, and have the maximum number of validators (2,350).  Next,
+run the script with the amount of SOL to deposit. We'll use 15 SOL:
 
 ```bash
-$ ./setup-stake-pool.sh
+$ ./setup-stake-pool.sh 15
 Creating pool
-+ spl-stake-pool create-pool --epoch-fee-numerator 3 --epoch-fee-denominator 1000 --withdrawal-fee-numerator 3 --withdrawal-fee-denominator 1000 --deposit-fee-numerator 3 --deposit-fee-denominator 1000 --referral-fee 50 --max-validators 2950 --pool-keypair keys/stake-pool.json --validator-list-keypair keys/validator-list.json --mint-keypair keys/mint.json --reserve-keypair keys/reserve.json
++ spl-stake-pool create-pool --epoch-fee-numerator 3 --epoch-fee-denominator 1000 --withdrawal-fee-numerator 3 --withdrawal-fee-denominator 1000 --deposit-fee-numerator 3 --deposit-fee-denominator 1000 --referral-fee 50 --max-validators 2350 --pool-keypair keys/stake-pool.json --validator-list-keypair keys/validator-list.json --mint-keypair keys/mint.json --reserve-keypair keys/reserve.json
 Creating reserve stake 4tvTkLB4X7ahUYZ2NaTohkG3mud4UBBvu9ZEGD4Wk9mt
 Creating mint BoNneHKDrX9BHjjvSpPfnQyRjsnc9WFH71v8wrgCd7LB
 Creating associated token account DgyZrAq88bnG1TNRxpgDQzWXpzEurCvfY2ukKFWBvADQ to receive stake pool tokens of mint BoNneHKDrX9BHjjvSpPfnQyRjsnc9WFH71v8wrgCd7LB, owned by 4SnSuUtJGKvk2GYpBwmEsWG53zTurVM8yXGsoiZQyMJn
 Creating pool fee collection account DgyZrAq88bnG1TNRxpgDQzWXpzEurCvfY2ukKFWBvADQ
 Signature: 51yf2J6dSGAx42KPs2oTMTV4ufEm1ncAHyLPQ6PNf4sbeMHGqno7BGn2tHkUnrd7PRXiWBbGzCWpJNevYjmoLgn2
 Creating stake pool Zg5YBPAk8RqBR9kaLLSoN5C8Uv7nErBz1WC63HTsCPR with validator list 86VZZCuqiz7sDJpFKjQy9c9dZQN9vwDKbYgY8pcwHuaF
 Signature: 47QHcWMEa5Syg13C3SQRA4n88Y8iLx1f39wJXQAStRUxpt2VD5t6pYgAdruNRHUQt1ZBY8QwbvEC1LX9j3nPrAzn
+Depositing SOL into stake pool
+Update not required
+Using existing associated token account DgyZrAq88bnG1TNRxpgDQzWXpzEurCvfY2ukKFWBvADQ to receive stake pool tokens of mint BoNneHKDrX9BHjjvSpPfnQyRjsnc9WFH71v8wrgCd7LB, owned by 4SnSuUtJGKvk2GYpBwmEsWG53zTurVM8yXGsoiZQyMJn
+Signature: 4jnS368HcofZ1rUpsGZtmSK9kVxFzJRndSX5VS7eMV3kVgzyg9efA4mcgd2C6BoSNksTmTonRGXTVM1WMywFpiKq
 ```
 
 Your stake pool now exists! For the largest number of validators, the cost for
-this phase is ~2.02 SOL.
+this phase is ~2.02 SOL, plus 15 SOL deposited into the pool in exchange for
+pool tokens.
+
+## Step 2: Deposit SOL into the pool
+
+Now that the pool exists, let's deposit some SOL in exchange for some pool tokens.
+
+SOL will likely be the most attractive form of deposit, since it's the easiest
+for everyone to use. Normally, this will likely be done from a DeFi app or
+wallet, but in our example, we'll do it straight from the command line.
+
+We already deposited 15 SOL during creation of the pool, but let's deposit 
+another 10 SOL into the pool:
+
+```
+$ spl-stake-pool deposit-sol Zg5YBPAk8RqBR9kaLLSoN5C8Uv7nErBz1WC63HTsCPR 10
+Using existing associated token account DgyZrAq88bnG1TNRxpgDQzWXpzEurCvfY2ukKFWBvADQ to receive stake pool tokens of mint BoNneHKDrX9BHjjvSpPfnQyRjsnc9WFH71v8wrgCd7LB, owned by 4SnSuUtJGKvk2GYpBwmEsWG53zTurVM8yXGsoiZQyMJn
+Signature: 4AJv6hSznYoMGnaQvjWXSBjKqtjYpjBx2MLezmRRjWRDa8vUaBLQfPNGd3kamZNs1JeWSvnzczwtzsMD5WkgKamA
+```
 
-## Step 2: Add validators to the pool
+## Step 3: Add validators to the pool
 
-Now that the pool exists, we need to add validators to it.
+Now that the pool has some SOL, we need to add validators to it.
 
 Using `add-validators.sh`, we'll add each of the validators created during step 0
 to the stake pool. If you are running on another network, you can create your own
@@ -153,34 +175,17 @@ Signature: 3XtmYu9msqnMeKJs9BopYjn5QTc5hENMXXiBwvEw6HYzU5w6z1HUkGwNW24io4Vu9WRKF
 ... (something similar repeated 9 more times)
 ```
 
-This operation costs 0.00328288 SOL per validator. This amount is totally recoverable
-by removing the validator from the stake pool.
+This operation moves 1.00228288 SOL from the reserve to a stake account on a given
+validator.  This means you'll need over 1 SOL for each validator that you want to add.
 
-## Step 3: Deposit into the pool
+## Step 4: Deposit stakes into the pool
 
-Now that your pool has validators, it needs some SOL or stake accounts for you
+Now that your pool has validators, it can accept stake accounts for you
 to manage. There are two possible sources of deposits: SOL or stake accounts.
+In step 2, we deposited SOL directly, so now we'll deposit stake accounts.
 
-### a) Depositing SOL
-
-This will likely be the most attractive form of deposit, since it's the easiest
-for everyone to use. Normally, this will likely be done from a DeFi app or
-wallet, but in our example, we'll do it straight from the command line.  Let's
-deposit 10 SOL into our pool:
-
-```
-$ spl-stake-pool deposit-sol Zg5YBPAk8RqBR9kaLLSoN5C8Uv7nErBz1WC63HTsCPR 100
-Using existing associated token account DgyZrAq88bnG1TNRxpgDQzWXpzEurCvfY2ukKFWBvADQ to receive stake pool tokens of mint BoNneHKDrX9BHjjvSpPfnQyRjsnc9WFH71v8wrgCd7LB, owned by 4SnSuUtJGKvk2GYpBwmEsWG53zTurVM8yXGsoiZQyMJn
-Signature: 4AJv6hSznYoMGnaQvjWXSBjKqtjYpjBx2MLezmRRjWRDa8vUaBLQfPNGd3kamZNs1JeWSvnzczwtzsMD5WkgKamA
-```
-
-Now there will be some SOL for us to work with.
-
-### b) Depositing stake accounts
-
-Alternatively, users can deposit stake accounts into the pool. This option is
-particularly attractive for users that already have a stake account, and either
-want stake pool tokens in return, or to diversify their stake more.
+This option is particularly attractive for users that already have a stake
+account, and either want stake pool tokens in return, or to diversify their stake more.
 
 The `deposit.sh` script gives an idea of how this works with the CLI.
 
@@ -194,7 +199,7 @@ $ ./deposit.sh keys/stake-pool.json local_validators.txt 10
 Note: This is a bit more finnicky on a local network because of the short epochs, and
 may fail. No problem, you simply need to retry.
 
-## Step 4: Rebalance stake in the pool
+## Step 5: Rebalance stake in the pool
 
 Over time, as people deposit SOL into the reserve, or as validator performance
 varies, you will want to move stake around. The best way to do this will be
@@ -216,7 +221,7 @@ to make sure that this is valid.
 $ ./rebalance.sh keys/stake-pool.json local_validators.txt 1
 ```
 
-## Step 5: Withdraw from the stake pool
+## Step 6: Withdraw from the stake pool
 
 Finally, if a user wants to withdraw from the stake pool, they can choose to
 withdraw SOL from the reserve if it has enough SOL, or to withdraw from one of

docs/src/stake-pool/quickstart.md

@@ -69,5 +69,3 @@ echo "If you are running on localnet with 32 slots per epoch, wait 12 seconds...
 sleep 12
 echo "Depositing stakes into stake pool"
 deposit_stakes "$stake_pool_pubkey" "$validator_list" $authority
-echo "Depositing SOL into stake pool"
-$spl_stake_pool deposit-sol "$stake_pool_pubkey" "$sol_amount"

stake-pool/cli/scripts/deposit.sh

@@ -5,27 +5,28 @@
 
 cd "$(dirname "$0")" || exit
 command_args=()
+sol_amount=$1
 
 ###################################################
 ### MODIFY PARAMETERS BELOW THIS LINE FOR YOUR POOL
 ###################################################
 
 # Epoch fee, assessed as a percentage of rewards earned by the pool every epoch,
 # represented as `numerator / denominator`
-command_args+=( --epoch-fee-numerator 0 )
-command_args+=( --epoch-fee-denominator 0 )
+command_args+=( --epoch-fee-numerator 1 )
+command_args+=( --epoch-fee-denominator 100 )
 
 # Withdrawal fee for SOL and stake accounts, represented as `numerator / denominator`
-command_args+=( --withdrawal-fee-numerator 0 )
-command_args+=( --withdrawal-fee-denominator 0 )
+command_args+=( --withdrawal-fee-numerator 2 )
+command_args+=( --withdrawal-fee-denominator 100 )
 
 # Deposit fee for SOL and stake accounts, represented as `numerator / denominator`
-command_args+=( --deposit-fee-numerator 0 )
-command_args+=( --deposit-fee-denominator 0 )
+command_args+=( --deposit-fee-numerator 3 )
+command_args+=( --deposit-fee-denominator 100 )
 
 command_args+=( --referral-fee 0 ) # Percentage of deposit fee that goes towards the referrer (a number between 0 and 100, inclusive)
 
-command_args+=( --max-validators 2950 ) # Maximum number of validators in the stake pool, 2950 is the current maximum possible
+command_args+=( --max-validators 2350 ) # Maximum number of validators in the stake pool, 2350 is the current maximum possible
 
 # (Optional) Deposit authority, required to sign all deposits into the pool.
 # Setting this variable makes the pool "private" or "restricted".
@@ -68,3 +69,9 @@ $spl_stake_pool \
   --validator-list-keypair "$validator_list_keyfile" \
   --mint-keypair "$mint_keyfile" \
   --reserve-keypair "$reserve_keyfile"
+
+set +ex
+echo "Depositing SOL into stake pool"
+stake_pool_pubkey=$(solana-keygen pubkey "$stake_pool_keyfile")
+set -ex
+$spl_stake_pool deposit-sol "$stake_pool_pubkey" "$sol_amount"