Friction report for setting up a Fuji subnet with custom vm

[tactical-retreat]

I recently set up a subnet on Fuji that uses a custom VM. I'm not using the turnkey stuff in the CLI that spins up nodes on cloud providers and does everything. Instead, I have a workflow that uses packer / terraform to manage my infra, and I'm using avalanche-cli to do the one-time setup tasks for registration.

I'm not proposing any changes here, just giving some feedback on my experiences, you can decide what to do with it.

My VM images have all the software they need baked in, including the the forked EVM I'm using for my CustomVM subnet.

Here's my saved output from the create command:

$ avalanche subnet create FerdyNet --custom-vm-path /home/ava/ferdynet/ferdynet_evm.bin --genesis /tmp/testnet.json
✔ Custom
creating custom VM subnet FerdyNet
Source code repository URL: https://github.com/tactical-retreat/subnet-evm
✔ Branch: master
Build script: scripts/build.sh
Successfully created subnet configuration

You'll notice that even though specified the --custom-vm-path it still forced me to provide a repo / branch / build script. This confused me. But anyway, the next confusing thing for me was this output:

$ avalanche subnet describe FerdyNet

_____       _        _ _
|  __ \     | |      (_) |
| |  | | ___| |_ __ _ _| |___
| |  | |/ _ \ __/ _  | | / __|
| |__| |  __/ || (_| | | \__ \
|_____/ \___|\__\__,_|_|_|___/
+-----------------+---------------------------------------------------+
|    PARAMETER    |                       VALUE                       |
+-----------------+---------------------------------------------------+
| Subnet Name     | FerdyNet                                          |
+-----------------+---------------------------------------------------+
| ChainID         | 668577                                            |
+-----------------+---------------------------------------------------+
| Mainnet ChainID | 0                                                 |
+-----------------+---------------------------------------------------+
| Token Name      | TEST                                              |
+-----------------+---------------------------------------------------+
| VM Version      |                                                   |
+-----------------+---------------------------------------------------+
| VM ID           | Y1BMATFymvjectqm11Zex8wVS88MwJHeh2nSx8q6ygNk8Br8P |
+-----------------+---------------------------------------------------+

VM Version I think only applies to vanilla subnet-evm. Not sure what Mainnet ChainID is or why it's 0. Token Name is hardcoded to TEST with no option to change it, but I'm not even sure what it's used for.

Moving on, I wanted to prep setting up my other nodes.

$ avalanche subnet export FerdyNet --output /tmp/ferdynet.json

$ cat /tmp/ferdynet.json | jq
{
 "Sidecar": {
   "Name": "FerdyNet",
   "VM": "Custom",
   "VMVersion": "",
   "RPCVersion": 30,
   "Subnet": "FerdyNet",
   "TokenName": "TEST",
   "ChainID": "",
   "Version": "1.4.0",
   "Networks": null,
   "ElasticSubnet": null,
   "ImportedFromAPM": false,
   "ImportedVMID": "",
   "CustomVMRepoURL": "https://github.com/tactical-retreat/subnet-evm",
   "CustomVMBranch": "master",
   "CustomVMBuildScript": "scripts/build.sh",
   "SubnetEVMMainnetChainID": 0
 },
 "Genesis": "<snip, long unnecessary base64 text>",
 "ChainConfig": null,
 "SubnetConfig": null,
 "NetworkUpgrades": null,
 "NodeConfig": null
}

ChainId is blank here, not sure why, SubnetEVMMainnetChainId of 0, etc. Given that this is a custom VM, might be a bit less confusing to not export some things and leave them as 'default' values, but I guess that depends on your philosophy on marshalling / unmarshalling. But this output was a bit confusing to me

Another thing that came up was during subnet joining (although probably not something you have to fix):

$ avalanche subnet join FerdyNet
✔ Fuji
✔ Automatic
Scanning your system for existing files...
Found a config file at /home/ava/.avalanchego/configs/node.json

My VM baking process uses the avalanche installer script, and the node.json ends up getting put in the user home dir. Prior to this setup, that wasn't an issue. My node data dir is on an attached persistent disk, so if I tear down the VM and use a newer baked VM, it can still pick up the data dir, and the node.json is fungible.

After doing avalanche subnet join it's no longer fungible because it's tracking the subnet. I don't really have a good suggestion for how to fix this. Even updating the avalanche installer script to let users specify the node config dir wouldn't fix this for me, since the baking process doesn't have access to write to the PD volume. I'll probably need to figure something out for that on my own.

Moving on, I went to go set up the subnet on my other validator nodes:

avalanche subnet import public
✔ Fuji
Provide the path to the genesis file: /tmp/testnet.json
✔ No
What is the ID of the blockchain?: e3k9c6RKgouPWtyQkr1ULC2PSFVu6BZdFefXZXJfwJ1zjWsmE
Getting information from the Fuji network...
Retrieved information. BlockchainID: e3k9c6RKgouPWtyQkr1ULC2PSFVu6BZdFefXZXJfwJ1zjWsmE, SubnetID: 2tjQ43HKjArCS5W5ckMRouHyvU8ez1yp81H4M6fZ9wMDMrn5rH, Name: FerdyNet, VMID: Y1BMATFymvjectqm11Zex8wVS88MwJHeh2nSx8q6ygNk8Br8P
✔ Custom
Error: importing custom VMs is not yet implemented, but will be available soon

OK, so I tried the other one:

$ avalanche subnet import file  
✔ File
✔ Select the file to import your subnet from: /tmp/ferdynet.json

Unfortunately, I can't import the CustomVM binary that I baked into my VM image. I had to install git/go on the machine and let it use the repo/build script to download and build it. But it did work.

I would prefer to be able to use a CustomVM binary from local disk, instead of going through the whole build process.

And finally, the join process:

avalanche subnet join FerdyNet
✔ Fuji
✔ Automatic
Scanning your system for existing files...
Found a config file at /home/ava/.avalanchego/configs/node.json
✔ Yes
Will use file at path /home/ava/.avalanchego/configs/node.json to update the configuration
Scanning your system for the plugin directory...
No plugin directory found on your system
Path to your avalanchego plugin dir (likely avalanchego/build/plugins): /mnt/avax-db/config/plugins

Note that it is unable to find my plugin directory. It's minor, but I think it should be able to do that though, by looking at my node config:

$ cat ~/.avalanchego/configs/node.json 
{
  "data-dir": "/mnt/avax-db/config",
  "db-dir": "/mnt/avax-db",
  "network-id": "fuji",
  "public-ip-resolution-service": "opendns",
  "track-subnets": "2tjQ43HKjArCS5W5ckMRouHyvU8ez1yp81H4M6fZ9wMDMrn5rH"
}

Although there were some hiccups and some confusing things, I did manage to get the whole thing done in a few hours, so these are mostly 'nice to have' type changes.

The most important thing for me would be to allow for joining the subnet using a CustomVM binary on local disk instead of having to go through the whole 'download and build and install' process.

[tactical-retreat]

To expand a bit about why it would be nice to have the VM loaded from disk instead of built on demand when joining, I just tried to add RPC nodes to my setup and they're failing to compile to VM, for some reason.

ava@rpc-fuji-1:/tmp$ avalanche subnet import file
✔ File
Select the file to import your subnet from: /tmp/ferdynet.json
Cloning into '/home/ava/.avalanche-cli/repos/FerdyNet'...
Using branch: master
Building Subnet EVM @ GitCommit: a10a6e67ca73f92e2691f7ca33887b9ea662f00d at /home/ava/.avalanche-cli/vms/FerdyNet
github.com/DataDog/zstd: build constraints exclude all Go files in /home/ava/go/pkg/mod/github.com/!data!dog/zstd@v1.5.2
# github.com/supranational/blst/bindings/go
../../../go/pkg/mod/github.com/supranational/blst@v0.3.11/bindings/go/rb_tree.go:130:18: undefined: Message
Error: error building custom vm binary using script scripts/build.sh on repo https://github.com/tactical-retreat/subnet-evm: exit status 1

(figured out the issue here, but it's still annoying)

[felipemadero]

Thanks for the detailed feedback. We will adding comments here once implemented the associated changes.