Switch logging from using prefixes to the new package path format

[Inspector-Butters]

This PR sets the foundation for advanced log control: per package visibility and verbosity.

---

The goal of this big PR is the following:

1. Adding a log.go file to every package: commit

   • Writing a bash script that adds the log.go file to every package that imports logrus, except the excluded packages, configured at the top of the bash script.
   • the log.go file creates a log variable and sets a field called package to the full path of that package.
   • I have tried to fix every error/problem that came from mass generation of this file. (duplicate declarations, different prefix names, etc...)
   • some packages had the log.go file from before, and had some helper functions in there as well. I've moved all of them to a log_helpers.go file within each package.

2. Create a CI rule which verifies that: commit

   • every package which imports logrus, also has a log.go file, except the excluded packages.
   • the package field of each log.go variable, has the correct path. (to detect when we move a package or change it's name)
   • I pushed a commit with a manually changed log.go file to trigger the ci check failure: commit

3. Alter the logging system to read the prefix from this package field for every log while outputing: commit

   • some packages have/want/need a different log prefix than their package name (like kv). This can be solved by keeping a map of package paths to prefix names somewhere.

---

Some notes:

• Please review everything carefully.
• I created the prefixReplacement map and populated the data that I deemed necessary. Please check it and complain if something doesn't make sense or is missing. I attached at the bottom, the list of all the packages that used to use a different name than their package name as their prefix.
• I have chosen to mark some packages to be excluded from this whole process. They will either not log anything, or log without a prefix, or log using their previously defined prefix. See the list of exclusions in the bottom.
• I fixed all the tests that failed because of this change. These were failing because they were expecting the old prefix to be in the generated logs. I have changed those to expect the new package field instead. This might not be a great solution. Ideally we might want to remove this from the tests so they only test for relevant fields in the logs. but this is a problem for another day.
• Please run the node with this config, and mention if you see something weird in the logs. (use different verbosities)
• The CI workflow uses a script that basically runs the hack/gen-logs.sh and checks that the git diff is zero. that script is hack/check-logs.sh. This means that if one runs this script locally, it will not actually check anything, rather than just regenerate the log.go files and fix any mistake. This might be confusing. Please suggest solutions if you think it's a problem.

---

A list of packages that used a different prefix than their package names for their logs:

• beacon-chain/cache/depositsnapshot/ package depositsnapshot, prefix "cache"
• beacon-chain/core/transition/log.go — package transition, prefix "state"
• beacon-chain/db/kv/log.go — package kv, prefix "db"
• beacon-chain/db/slasherkv/log.go — package slasherkv, prefix "slasherdb"
• beacon-chain/db/pruner/pruner.go — package pruner, prefix "db-pruner"
• beacon-chain/light-client/log.go — package light_client, prefix "light-client"
• beacon-chain/operations/attestations/log.go — package attestations, prefix "pool/attestations"
• beacon-chain/operations/slashings/log.go — package slashings, prefix "pool/slashings"
• beacon-chain/rpc/core/log.go — package core, prefix "rpc/core"
• beacon-chain/rpc/eth/beacon/log.go — package beacon, prefix "rpc/beaconv1"
• beacon-chain/rpc/eth/validator/log.go — package validator, prefix "beacon-api"
• beacon-chain/rpc/prysm/v1alpha1/beacon/log.go — package beacon, prefix "rpc"
• beacon-chain/rpc/prysm/v1alpha1/validator/log.go — package validator, prefix "rpc/validator"
• beacon-chain/state/stategen/log.go — package stategen, prefix "state-gen"
• beacon-chain/sync/checkpoint/log.go — package checkpoint, prefix "checkpoint-sync"
• beacon-chain/sync/initial-sync/log.go — package initialsync, prefix "initial-sync"
• cmd/prysmctl/p2p/log.go — package p2p, prefix "prysmctl-p2p"
• config/features/log.go -- package features, prefix "flags"
• io/file/log.go — package file, prefix "fileutil"
• proto/prysm/v1alpha1/log.go — package eth, prefix "protobuf"
• validator/client/beacon-api/log.go — package beacon_api, prefix "beacon-api"
• validator/db/kv/log.go — package kv, prefix "db"
• validator/db/filesystem/db.go — package filesystem, prefix "db"
• validator/keymanager/derived/log.go — package derived, prefix "derived-keymanager"
• validator/keymanager/local/log.go — package local, prefix "local-keymanager"
• validator/keymanager/remote-web3signer/log.go — package remote_web3signer, prefix "remote-keymanager"
• validator/keymanager/remote-web3signer/internal/log.go — package internal, prefix "remote-web3signer-
  internal"
• beacon-chain/forkchoice/doubly... prefix is "forkchoice-doublylinkedtree"

List of excluded directories (their subdirectories are also excluded):

EXCLUDED_PATH_PREFIXES=(
    "testing"
    "validator/client/testutil"
    "beacon-chain/p2p/testing"
    "beacon-chain/rpc/eth/config"
    "beacon-chain/rpc/prysm/v1alpha1/debug"
    "tools"
    "runtime"
    "monitoring"
    "io"
    "cmd"
    ".well-known"
    "changelog"
    "hack"
    "specrefs"
    "third_party"
    "bazel-out"
    "bazel-bin"
    "bazel-prysm"
    "bazel-testlogs"
    "build"
    ".github"
    ".jj"
    ".idea"
    ".vscode"
)

[prestonvanloon]

You shouldn't need to do this every time. Can you make the file executable in the git repo?

[prestonvanloon]

If this could also exclude paths listed in gitignore, then it would be somewhat better positioned for future directories to ignore