Prepare LN for large networks

[pinheadmz]

1. Move all the try/catch/retry logic from ln.py to ln_init.py. Meaning the API calls have no error handling on their own and propagate back up to the caller. This is to ensure consistency in the ln_init scenario which is opinionated about what is allowed to fail. Ideally the scenario completes successfully, terminates with an error, or runs endlessly because something is broken. In all those cases, ln_init needs to directly handle all API calls to LN nodes.
2. Refactor ln_init for large networks (specifically test/data/LN_100.json with >500 channels) ensuring LN nodes always have enough funds to open all their channels, that channel opens are always tx output 0 and channel open TXs are ordered as expected in their blocks.

[pinheadmz]

@macgyver13 I touched a lot of your LN API code, lemme know if you see anything smelly or frustrating in here,,,

[pinheadmz]

Tested with 10, 50, 100 node networks + simln: https://github.com/pinheadmz/ln100-warnet

On digital ocean cluster: 100 vCPUs / 200 GB Memory / 1.22 TB Disk

[macgyver13]

@macgyver13 I touched a lot of your LN API code, lemme know if you see anything smelly or frustrating in here,,,

Concept ACK - I like the separation of concerns, nice work - much needed clean up 👏
I am a little surprised the frequent use of reset_connection in CLN and LND - was that required for stability?

I will test later this week and even consider testing eclair with these changes to see if an underlying issue was causing instability with eclair nodes in warnet.

There are some nice improvements in ln.py in the eclair PR that may make sense to follow this PR if you see value - around the "use_rpc" concept.

[pinheadmz]

reset_connection

Well, the way it was before we only reset connection if there was a problem and my new idea is, ln.py doesn't deal with problems at all. I also had to deal with this already in scenarios based on the functional test framework:

warnet/resources/scenarios/commander.py

Lines 87 to 94 in e4ed4b6

	# Ensure that all RPC calls are made with brand new http connections
	def auth_proxy_request(self, method, path, postdata):
	self._set_conn() # creates new http client connection
	return self.oldrequest(method, path, postdata)
	AuthServiceProxy.oldrequest = AuthServiceProxy._request
	AuthServiceProxy._request = auth_proxy_request

Unlike the test framework, we can't expect an HTTP connection to survive the entire lifespan of a warnet, so we refresh the connection on every call.

[macgyver13]

I think you need to use {} for the default value
if default_file.get("lnd", {}).get("metricsExport"):

[macgyver13]

^^ same problem