[Bug]: local-ic crashes with no transfer channel found between chains #1342
Description
What happened and what did you expect to happen?
Sometimes in a new environment local-ic crashes with an error that looks like this:
panic: no transfer channel found between chains: localcosmos-1 - localjuno-1
But in the exact same environment, it will succeed the next time its run.
Describe how to reproduce the bug
Start local-ic with the following chain config
{
"chains": [
{
"name": "gaia",
"chain_id": "localcosmos-1",
"denom": "uatom",
"binary": "gaiad",
"bech32_prefix": "cosmos",
"docker_image": {
"version": "v15.0.0-rc2"
},
"gas_prices": "0%DENOM%",
"chain_type": "cosmos",
"coin_type": 118,
"trusting_period": "336h",
"gas_adjustment": 1.3,
"number_vals": 1,
"number_node": 0,
"ibc_paths": ["gaia-juno"],
"debugging": true,
"block_time": "1s",
"genesis": {
"modify": [
{
"key": "app_state.gov.params.voting_period",
"value": "3s"
},
{
"key": "app_state.interchainaccounts.host_genesis_state.params.allow_messages",
"value": [
"/cosmos.bank.v1beta1.MsgSend",
"/cosmos.bank.v1beta1.MsgMultiSend",
"/cosmos.staking.v1beta1.MsgDelegate",
"/cosmos.staking.v1beta1.MsgUndelegate",
"/cosmos.staking.v1beta1.MsgBeginRedelegate",
"/cosmos.staking.v1beta1.MsgRedeemTokensforShares",
"/cosmos.staking.v1beta1.MsgTokenizeShares",
"/cosmos.distribution.v1beta1.MsgWithdrawDelegatorReward",
"/cosmos.distribution.v1beta1.MsgSetWithdrawAddress",
"/ibc.applications.transfer.v1.MsgTransfer"
]
}
],
"accounts": [
{
"name": "acc0",
"address": "cosmos1hj5fveer5cjtn4wd6wstzugjfdxzl0xpxvjjvr",
"amount": "20000000000000%DENOM%",
"mnemonic": "decorate bright ozone fork gallery riot bus exhaust worth way bone indoor calm squirrel merry zero scheme cotton until shop any excess stage laundry"
},
{
"name": "acc1",
"address": "cosmos1kljf09rj77uxeu5lye7muejx6ajsu55cc3re5h",
"amount": "20000000000000%DENOM%",
"mnemonic": "across army acoustic hurt help sad turkey switch popular fade purse obvious session tuition file asset cover agree number motor pupil slim hundred busy"
},
{
"name": "acc2",
"address": "cosmos17lp3n649rxt2jadn455frcj0q6anjndsw0xwrz",
"amount": "20000000000000%DENOM%",
"mnemonic": "demise erode feature decade dune uncle limb stock quit nation neck marriage pledge achieve tell cat baby wrist expect scrub welcome hole ribbon mirror"
},
{
"name": "acc3",
"address": "cosmos1p0var04vhr03r2j8zwv4jfrz73rxgjt5v29x49",
"amount": "20000000000000%DENOM%",
"mnemonic": "scheme force walk answer decide submit crowd flush slim raw type tackle lend follow multiply sting rule jealous coyote slight toddler skirt crawl decade"
}
]
}
},
{
"name": "neutron",
"chain_id": "localneutron-1",
"denom": "untrn",
"binary": "neutrond",
"bech32_prefix": "neutron",
"docker_image": {
"version": "v3.0.4",
"repository": "ghcr.io/strangelove-ventures/heighliner/neutron"
},
"gas_prices": "0.0untrn,0.0uatom",
"chain_type": "cosmos",
"coin_type": 118,
"trusting_period": "336h",
"gas_adjustment": 1.3,
"number_vals": 1,
"number_node": 0,
"ics_consumer_link": "localcosmos-1",
"ibc_paths": ["neutron-juno"],
"debugging": true,
"block_time": "1s",
"genesis": {
"modify": [
{
"key": "consensus_params.block.max_gas",
"value": "100000000"
},
{
"key": "app_state.ccvconsumer.params.soft_opt_out_threshold",
"value": "0.05"
},
{
"key": "app_state.ccvconsumer.params.reward_denoms",
"value": ["untrn"]
},
{
"key": "app_state.ccvconsumer.params.provider_reward_denoms",
"value": ["uatom"]
},
{
"key": "app_state.globalfee.params.minimum_gas_prices",
"value": [
{
"denom": "untrn",
"amount": "0"
}
]
},
{
"key": "app_state.feeburner.params.treasury_address",
"value": "neutron1hj5fveer5cjtn4wd6wstzugjfdxzl0xpznmsky"
},
{
"key": "app_state.tokenfactory.params.fee_collector_address",
"value": "neutron1hj5fveer5cjtn4wd6wstzugjfdxzl0xpznmsky"
},
{
"key": "app_state.interchainaccounts.host_genesis_state.params.allow_messages",
"value": [
"/cosmos.bank.v1beta1.MsgSend",
"/cosmos.bank.v1beta1.MsgMultiSend",
"/cosmos.staking.v1beta1.MsgDelegate",
"/cosmos.staking.v1beta1.MsgUndelegate",
"/cosmos.staking.v1beta1.MsgBeginRedelegate",
"/cosmos.staking.v1beta1.MsgRedeemTokensforShares",
"/cosmos.staking.v1beta1.MsgTokenizeShares",
"/cosmos.distribution.v1beta1.MsgWithdrawDelegatorReward",
"/cosmos.distribution.v1beta1.MsgSetWithdrawAddress",
"/ibc.applications.transfer.v1.MsgTransfer",
"/ibc.lightclients.localhost.v2.ClientState",
"/ibc.core.client.v1.MsgCreateClient",
"/ibc.core.client.v1.Query/ClientState",
"/ibc.core.client.v1.Query/ConsensusState",
"/ibc.core.connection.v1.Query/Connection"
]
}
],
"accounts": [
{
"name": "acc0",
"address": "neutron1hj5fveer5cjtn4wd6wstzugjfdxzl0xpznmsky",
"amount": "10000000000000%DENOM%",
"mnemonic": "decorate bright ozone fork gallery riot bus exhaust worth way bone indoor calm squirrel merry zero scheme cotton until shop any excess stage laundry"
},
{
"name": "acc1",
"address": "neutron1kljf09rj77uxeu5lye7muejx6ajsu55cuw2mws",
"amount": "10000000000000%DENOM%",
"mnemonic": "across army acoustic hurt help sad turkey switch popular fade purse obvious session tuition file asset cover agree number motor pupil slim hundred busy"
},
{
"name": "acc2",
"address": "neutron17lp3n649rxt2jadn455frcj0q6anjnds2s0ve9",
"amount": "10000000000000%DENOM%",
"mnemonic": "demise erode feature decade dune uncle limb stock quit nation neck marriage pledge achieve tell cat baby wrist expect scrub welcome hole ribbon mirror"
},
{
"name": "acc3",
"address": "neutron1p0var04vhr03r2j8zwv4jfrz73rxgjt5g4vy0z",
"amount": "10000000000000%DENOM%",
"mnemonic": "scheme force walk answer decide submit crowd flush slim raw type tackle lend follow multiply sting rule jealous coyote slight toddler skirt crawl decade"
}
]
}
},
{
"name": "juno",
"chain_id": "localjuno-1",
"denom": "ujuno",
"binary": "junod",
"bech32_prefix": "juno",
"docker_image": {
"repository": "ghcr.io/strangelove-ventures/heighliner/juno",
"version": "v24.0.0"
},
"gas_prices": "0%DENOM%",
"chain_type": "cosmos",
"coin_type": 118,
"trusting_period": "336h",
"gas_adjustment": 3.0,
"number_vals": 1,
"number_node": 0,
"ibc_paths": ["neutron-juno", "gaia-juno"],
"block_time": "1s",
"debugging": true,
"genesis": {
"modify": [
{
"key": "consensus_params.block.max_gas",
"value": "100000000"
},
{
"key": "app_state.gov.params.voting_period",
"value": "15s"
},
{
"key": "app_state.gov.params.max_deposit_period",
"value": "15s"
},
{
"key": "app_state.gov.params.min_deposit.0.denom",
"value": "ujuno"
}
],
"accounts": [
{
"name": "acc0",
"address": "juno1hj5fveer5cjtn4wd6wstzugjfdxzl0xps73ftl",
"amount": "10000000000000%DENOM%",
"mnemonic": "decorate bright ozone fork gallery riot bus exhaust worth way bone indoor calm squirrel merry zero scheme cotton until shop any excess stage laundry"
},
{
"name": "acc1",
"address": "juno1kljf09rj77uxeu5lye7muejx6ajsu55cwrqznt",
"amount": "10000000000000%DENOM%",
"mnemonic": "across army acoustic hurt help sad turkey switch popular fade purse obvious session tuition file asset cover agree number motor pupil slim hundred busy"
},
{
"name": "acc2",
"address": "juno17lp3n649rxt2jadn455frcj0q6anjndsca94y7",
"amount": "10000000000000%DENOM%",
"mnemonic": "demise erode feature decade dune uncle limb stock quit nation neck marriage pledge achieve tell cat baby wrist expect scrub welcome hole ribbon mirror"
},
{
"name": "acc3",
"address": "juno1p0var04vhr03r2j8zwv4jfrz73rxgjt56cxaje",
"amount": "10000000000000%DENOM%",
"mnemonic": "scheme force walk answer decide submit crowd flush slim raw type tackle lend follow multiply sting rule jealous coyote slight toddler skirt crawl decade"
}
]
}
}
]
}version
latest commit on main (4bd4690)
Relevant logs or stack trace
11:54:41 info Exec {"validator": true, "i": 0, "chain_id": "localcosmos-1", "test": "neutron_junoic", "image": "ghcr.io/strangelove-ventures/heighliner/gaia:v15.0.0-rc2", "test_name": "neutron_junoic", "command": "sh -c echo \"decorate bright ozone fork gallery riot bus exhaust worth way bone indoor calm squirrel merry zero scheme cotton until shop any excess stage laundry\" | gaiad keys add acc0 --recover --keyring-backend test --coin-type 118 --home /var/cosmos-chain/localcosmos-1 --output json", "hostname": "neutron_junoic-uscxwm", "container": "neutron_junoic-uscxwm"}
11:54:41 info Exec {"validator": true, "i": 0, "chain_id": "localcosmos-1", "test": "neutron_junoic", "image": "ghcr.io/strangelove-ventures/heighliner/gaia:v15.0.0-rc2", "test_name": "neutron_junoic", "command": "sh -c echo \"across army acoustic hurt help sad turkey switch popular fade purse obvious session tuition file asset cover agree number motor pupil slim hundred busy\" | gaiad keys add acc1 --recover --keyring-backend test --coin-type 118 --home /var/cosmos-chain/localcosmos-1 --output json", "hostname": "neutron_junoic-rfyudc", "container": "neutron_junoic-rfyudc"}
11:54:41 info Exec {"validator": true, "i": 0, "chain_id": "localcosmos-1", "test": "neutron_junoic", "image": "ghcr.io/strangelove-ventures/heighliner/gaia:v15.0.0-rc2", "test_name": "neutron_junoic", "command": "sh -c echo \"demise erode feature decade dune uncle limb stock quit nation neck marriage pledge achieve tell cat baby wrist expect scrub welcome hole ribbon mirror\" | gaiad keys add acc2 --recover --keyring-backend test --coin-type 118 --home /var/cosmos-chain/localcosmos-1 --output json", "hostname": "neutron_junoic-mffmot", "container": "neutron_junoic-mffmot"}
11:54:42 info Exec {"validator": true, "i": 0, "chain_id": "localcosmos-1", "test": "neutron_junoic", "image": "ghcr.io/strangelove-ventures/heighliner/gaia:v15.0.0-rc2", "test_name": "neutron_junoic", "command": "sh -c echo \"scheme force walk answer decide submit crowd flush slim raw type tackle lend follow multiply sting rule jealous coyote slight toddler skirt crawl decade\" | gaiad keys add acc3 --recover --keyring-backend test --coin-type 118 --home /var/cosmos-chain/localcosmos-1 --output json", "hostname": "neutron_junoic-esttfe", "container": "neutron_junoic-esttfe"}
11:54:42 info Exec {"validator": true, "i": 0, "chain_id": "localneutron-1", "test": "neutron_junoic", "image": "ghcr.io/strangelove-ventures/heighliner/neutron:v3.0.4", "test_name": "neutron_junoic", "command": "sh -c echo \"decorate bright ozone fork gallery riot bus exhaust worth way bone indoor calm squirrel merry zero scheme cotton until shop any excess stage laundry\" | neutrond keys add acc0 --recover --keyring-backend test --coin-type 118 --home /var/cosmos-chain/localneutron-1 --output json", "hostname": "neutron_junoic-rztnwo", "container": "neutron_junoic-rztnwo"}
11:54:42 info Exec {"validator": true, "i": 0, "chain_id": "localneutron-1", "test": "neutron_junoic", "image": "ghcr.io/strangelove-ventures/heighliner/neutron:v3.0.4", "test_name": "neutron_junoic", "command": "sh -c echo \"across army acoustic hurt help sad turkey switch popular fade purse obvious session tuition file asset cover agree number motor pupil slim hundred busy\" | neutrond keys add acc1 --recover --keyring-backend test --coin-type 118 --home /var/cosmos-chain/localneutron-1 --output json", "hostname": "neutron_junoic-daphxt", "container": "neutron_junoic-daphxt"}
11:54:43 info Exec {"validator": true, "i": 0, "chain_id": "localneutron-1", "test": "neutron_junoic", "image": "ghcr.io/strangelove-ventures/heighliner/neutron:v3.0.4", "test_name": "neutron_junoic", "command": "sh -c echo \"demise erode feature decade dune uncle limb stock quit nation neck marriage pledge achieve tell cat baby wrist expect scrub welcome hole ribbon mirror\" | neutrond keys add acc2 --recover --keyring-backend test --coin-type 118 --home /var/cosmos-chain/localneutron-1 --output json", "hostname": "neutron_junoic-kbobex", "container": "neutron_junoic-kbobex"}
11:54:43 info Exec {"validator": true, "i": 0, "chain_id": "localneutron-1", "test": "neutron_junoic", "image": "ghcr.io/strangelove-ventures/heighliner/neutron:v3.0.4", "test_name": "neutron_junoic", "command": "sh -c echo \"scheme force walk answer decide submit crowd flush slim raw type tackle lend follow multiply sting rule jealous coyote slight toddler skirt crawl decade\" | neutrond keys add acc3 --recover --keyring-backend test --coin-type 118 --home /var/cosmos-chain/localneutron-1 --output json", "hostname": "neutron_junoic-uvfcsn", "container": "neutron_junoic-uvfcsn"}
11:54:43 info Exec {"validator": true, "i": 0, "chain_id": "localjuno-1", "test": "neutron_junoic", "image": "ghcr.io/strangelove-ventures/heighliner/juno:v24.0.0", "test_name": "neutron_junoic", "command": "sh -c echo \"decorate bright ozone fork gallery riot bus exhaust worth way bone indoor calm squirrel merry zero scheme cotton until shop any excess stage laundry\" | junod keys add acc0 --recover --keyring-backend test --coin-type 118 --home /var/cosmos-chain/localjuno-1 --output json", "hostname": "neutron_junoic-gpqlbg", "container": "neutron_junoic-gpqlbg"}
11:54:44 info Exec {"validator": true, "i": 0, "chain_id": "localjuno-1", "test": "neutron_junoic", "image": "ghcr.io/strangelove-ventures/heighliner/juno:v24.0.0", "test_name": "neutron_junoic", "command": "sh -c echo \"across army acoustic hurt help sad turkey switch popular fade purse obvious session tuition file asset cover agree number motor pupil slim hundred busy\" | junod keys add acc1 --recover --keyring-backend test --coin-type 118 --home /var/cosmos-chain/localjuno-1 --output json", "hostname": "neutron_junoic-hqwrxo", "container": "neutron_junoic-hqwrxo"}
11:54:44 info Exec {"validator": true, "i": 0, "chain_id": "localjuno-1", "test": "neutron_junoic", "image": "ghcr.io/strangelove-ventures/heighliner/juno:v24.0.0", "test_name": "neutron_junoic", "command": "sh -c echo \"demise erode feature decade dune uncle limb stock quit nation neck marriage pledge achieve tell cat baby wrist expect scrub welcome hole ribbon mirror\" | junod keys add acc2 --recover --keyring-backend test --coin-type 118 --home /var/cosmos-chain/localjuno-1 --output json", "hostname": "neutron_junoic-asfmhz", "container": "neutron_junoic-asfmhz"}
11:54:44 info Exec {"validator": true, "i": 0, "chain_id": "localjuno-1", "test": "neutron_junoic", "image": "ghcr.io/strangelove-ventures/heighliner/juno:v24.0.0", "test_name": "neutron_junoic", "command": "sh -c echo \"scheme force walk answer decide submit crowd flush slim raw type tackle lend follow multiply sting rule jealous coyote slight toddler skirt crawl decade\" | junod keys add acc3 --recover --keyring-backend test --coin-type 118 --home /var/cosmos-chain/localjuno-1 --output json", "hostname": "neutron_junoic-cftwbc", "container": "neutron_junoic-cftwbc"}
11:54:45 info Exec {"image": "ghcr.io/cosmos/relayer:latest", "test_name": "neutron_junoic", "command": "rly q clients localcosmos-1 --home /home/relayer", "hostname": "neutron_junoic-fyydjs", "container": "neutron_junoic-fyydjs"}
11:54:45 info Exec {"image": "ghcr.io/cosmos/relayer:latest", "test_name": "neutron_junoic", "command": "rly q connections localcosmos-1 --home /home/relayer", "hostname": "neutron_junoic-qlytby", "container": "neutron_junoic-qlytby"}
11:54:45 info Exec {"image": "ghcr.io/cosmos/relayer:latest", "test_name": "neutron_junoic", "command": "rly q channels localcosmos-1 --home /home/relayer", "hostname": "neutron_junoic-jjolty", "container": "neutron_junoic-jjolty"}
panic: no transfer channel found between chains: localcosmos-1 - localjuno-1
goroutine 1 [running]:
github.com/strangelove-ventures/interchaintest/local-interchain/interchain.GetChannelConnections({0x39bc228, 0xc001387770}, 0x3?, {0xc001288510, 0x3, 0x1c?}, 0x0?, {0x39f7030, 0xc001a1c428}, {0x3982000, ...})
/home/runner/work/valence-protocol/valence-protocol/interchaintest/local-interchain/interchain/ibc.go:55 +0x659
github.com/strangelove-ventures/interchaintest/local-interchain/interchain.StartChain({0xc00007c054, 0x48}, {0xc000bb90c8, 0x11}, 0xc000fd4400)
/home/runner/work/valence-protocol/valence-protocol/interchaintest/local-interchain/interchain/start.go:308 +0x1f18
main.init.func7(0x56aca60, {0xc000de2630?, 0x4?, 0x31e3afb?})
/home/runner/work/valence-protocol/valence-protocol/interchaintest/local-interchain/cmd/local-ic/start_chain.go:87 +0x4cc
github.com/spf13/cobra.(*Command).execute(0x56aca60, {0xc000de25a0, 0x3, 0x3})
/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:989 +0xa91
github.com/spf13/cobra.(*Command).ExecuteC(0x56ac780)
/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
main.main()
/home/runner/work/valence-protocol/valence-protocol/interchaintest/local-interchain/cmd/local-ic/main.go:27 +0x1b2
panic: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.43/containers/json?filters=%7B%22label%22%3A%7B%22ibc-test%22%3Atrue%7D%7D": context canceled
goroutine 35 [running]:
github.com/strangelove-ventures/interchaintest/v8/dockerutil.KillAllInterchaintestContainers({0x39bc228, 0xc001387770})
/home/runner/work/valence-protocol/valence-protocol/interchaintest/dockerutil/kill.go:36 +0x50e
github.com/strangelove-ventures/interchaintest/local-interchain/interchain.killContainer({0x39bc228?, 0xc001387770?})
/home/runner/work/valence-protocol/valence-protocol/interchaintest/local-interchain/interchain/start.go:336 +0x25
github.com/strangelove-ventures/interchaintest/local-interchain/interchain.StartChain.func1()
/home/runner/work/valence-protocol/valence-protocol/interchaintest/local-interchain/interchain/start.go:53 +0xc7
created by github.com/strangelove-ventures/interchaintest/local-interchain/interchain.StartChain in goroutine 1
/home/runner/work/valence-protocol/valence-protocol/interchaintest/local-interchain/interchain/start.go:46 +0x3a5Code of Conduct
- I agree to follow this project's Code of Conduct