Lotus CLI / services configuration and support for multiple services deployments

[nonsense]

Problem statement

Since Lotus v1.11, Lotus can be run in a variety of configurations, for example:

1. monolith lotus-miner (all subsystems) + embedded worker on a single host
2. monolith lotus-miner (all subsystems) + separate workers on multiple hosts
3. lotus-miner (mining subsystems) + embedded worker on a single host + lotus-miner (markets subsystem) on a different host
4. multiple hosts with lotus-miner (mining subsystems), lotus-miner (markets subsystem), multiple lotus-worker node on a number of hosts

It is getting increasingly difficult to manage various deployments with the existing configuration params and values, such as: LOTUS_MINER_PATH, MINER_API_INFO, MARKETS_API_INFO, etc.

lotus-miner binary, not only starts a given lotus-miner process (markets, or miner subsystems), but also acts as a CLI tool to inspect processes. Some lotus-miner commands target a specific process (i.e. a markets process), while others target multiple processes and aggregate the information from all of them (such as lotus-miner info).

Due to this complexity, miners/users often have a hard time configuring their deployments and getting results out of their interaction with their deployment, for example:

#7044
#7072
#6869

This is only going to get worse, once we introduce more components to Lotus, such as multiple markets nodes, or indexer services.

Currently Lotus can be configured at a few different places, which makes it difficult to report bugs accurately - we support config files, command line flags, as well as environment variables. Multiple times during the development/test process for Lotus v1.11 we’ve had to request additional information from bug reporters, just so that we understand their deployment and reproduce the issues they’ve had, which is not optimal.

Therefore I am starting a discussion on how to refactor the current codebase in order to lay the foundations for ease of management and operations of future Lotus deployments, where we have many more process types (mining/proving subsystems, markets subsystems, indexing subsystems, etc.)

Requirements

• continue to support monolith deployments, without introducing unnecessary complexity - it should still be easy and simple to start a Lotus miner on a single host, without having to manually juggle 10 different processes

Ideas / Areas for improvement

• lotusctl - build tooling to allow users/miners to easily report issues and include full information about their configuration and deployment setup
• make it impossible for users to query wrong subsystem on a wrong host
• have the same command return the same result, on matter the deployment, for example:
  • lotusctl get storage-deals - shows all storage-deals, by fetching them from the markets subsystem - should work on all deployments as well as monolith
• solve the environment variable mess in the CLI, potentially deprecating the many env vars + CLI flags, and moving to an explicit lotus-cli.yml config file (that could be auto-generated once from current env vars).
• decide whether we want or not a lotus-markets entrypoint (and similar future entrypoints to other upcoming services)

The idea for this discussion is to collect pain points / more requirements and ideas for Lotus improvements, and then propose a development plan.

[dd45e640b42e6da7da96faee3996ef7c]

maybe follow what we already have with miner/worker:

• a single user declared "head" miner process
• multiple miner-*** / worker processes on multiple (or not) hosts

all, let em call them, satellites take the head miners API info as input. the head miner takes care that every satellite in the setup gets the others satellites API info's it needs to operate.

[dd45e640b42e6da7da96faee3996ef7c]

the head miner houses the scheduler, obviously.

if an operator calls a command on any satellite miner process the call just gets delegated to the head miner as it is the only part in the system that knows any other part in the system.

if A knows B and A knows C then it shouldn't be that hard to let A make the introduction between B and C if needed. But B does not need to know where C is by env var or else. (i think this is already done with workers anyways, why not extend it to miner-*** processes too)

all it needs is a central coordinating miner process

[nonsense]

Here is an outline of the tasks that I think would help alleviate the problems mentioned above:


Design for how lotus-miner is going to host more services going forward.

There is functionality for this already in lotus-miner in the form of Subsystems. For now a single binary lotus-miner includes the full codebase, and depending on configuration it starts various subsystems and configures the different node types. In the future we might decide to split lotus-miner into multiple smaller codebases if they are completely independent, but for now there is no reason to do that.

We should still improve the configuration of various services (markets, or sealer (this is how I call the proving/sealing service) for example, and in the near future, the indexer). At the moment it is possible to configure a nonsensical service, such as:

[Subsystems]
  EnableMining = false
  EnableSealing = true
  EnableSectorStorage = false
  EnableMarkets = true

Therefore I suggest we hide the Subsystems config and just introduce a ServiceType config, with initial values of Markets and Sealer, which would be determining the necessary Subsystems to be instantiated.

Configuration of a service (ENV vars, config files, cmdline flags) - include all service configuration values in the config.toml

We already have the node type (in the form of enabled subsystems) in the config.toml , with the idea for this to be replaced with ServiceType. Therefore no need to have individual flags for different types of repos (and different services have different repos) -

We also communicate that information with various flags that I think we should deprecate:

• deprecate —miner-repo, —storagerepo, —markets-repo, etc. infer the service type from the config.toml
• deprecate LOTUS_MINER_PATH ; LOTUS_PATH ; LOTUS_MARKETS_PATH ; all repo/service env vars

Ideally we should be able to start a given lotus-miner service (worker, sealer, markets, etc.) by just providing one flag/env var - either the repo location or the config.toml (which again is in the repo directory).

CLI tools and interacting with a mining deployment (monolith, or multi-host environment)

At the moment we have to specify env var for every service that lotus-miner might interact with, for example:

FULLNODE_API_INFO=token:/ip4/127.0.0.1/tcp/1234/http
MARKETS_API_INFO=token:/ip4/127.0.0.1/tcp/2345/http
MINER_API_INFO=token:/ip4/127.0.0.1/tcp/8787/http

If lotus-miner is invoked locally on one of these service types, it could also get that information if a repo location is provided, but I think this should not be the source-of-truth, as local repos do not necessarily have information about the whole deployment, just about their part of the stack (for example lotus-workers do not know the location of markets nodes)

I suggest we explicitly define deployment config in a toml file, separate from individual services’ config/repos, which should be provided to lotusctl (or we keep the CLI interaction in lotus-miner). The important part is to be able to quickly get information about the environment from one place, and not based on precedence of env vars / config files / local directories / etc.

[neondragon]

Another potentially confusing point to consider -- some commands accept a path as an argument e.g.:

lotus-miner storage-deals import-data
lotus-miner backup

The path isn't read by the client, it's read by the server process. The help doesn't necessarily make that clear:

# lotus-miner storage-deals import-data --help
NAME:
   lotus-miner storage-deals import-data - Manually import data for a deal

USAGE:
   lotus-miner storage-deals import-data [command options] <proposal CID> <file>

OPTIONS:
   --help, -h  show help (default: false)

So if you have files you want to import, you could easily do this and get confused:

# ls
file.car

# MARKETS_API_INFO=token:/ip4/1.2.3.4/tcp/8787/http lotus-miner storage-deals import-data bafyreigx3k7ft65tuvair2vqkenmsnjvp5v7f5ygtnldcvkra5tbdvaow2 file.car
ERROR: failed to open given file: open file.car: no such file or directory

[dd45e640b42e6da7da96faee3996ef7c]

related #5199

[nonsense]

@neondragon this is indeed absolutely confusing. We are certainly considering renaming/refactoring commands so that they convey whether they are talking to a remote process or not - for example if we had lotusctl, instead of lotus, it'd be obvious that this command is control a remote daemon process, and avoid such footguns. Similarly daemon commands could probably be named lotusd or something like this.

In any case having lotus and lotus-miner act as both client-side utils and daemons is somewhat confusing to many users right now, and we are considering solutions.

[dd45e640b42e6da7da96faee3996ef7c]

to follow known patterns lotus daemon should be lotusd and lotus client should be lotusctl. same for lotus-miner/d and lotus-miner/ctl

[dd45e640b42e6da7da96faee3996ef7c]

it would be great if we then could build binaries that include exactly the functionality they need and not the whole codebase.

[raulk]

I think the binary separation between daemon processes and client commands is the way to go. That way the lotus client commands binary can be really light, as all it's doing is offering a CLI that ends up invoking JSON-RPC commands on the appropriate process. There is no reason that binary should include all of lotus.

To configure the processes it can talk to, we would have a lotusctl.cfg.yml file enumerating all known lotus/lotus-miner instances and their endpoints/repos.

In the future, we could add fancier "discovery" options like mDNS, Consul registries, and more.