Validators API needs filtering

[tdenisenko]

When calling /eth/v1/beacon/states/head/validators?status=active over 1 gigabyte of data is returned. We need to be able to get only the fields requested for various reasons. Such as only the indexes of the active validators which would reduce the size of the data returned significantly (~4 KB).

Proposal:

Allow filtering by fields, such as /eth/v1/beacon/states/head/validators?status=active&fields=index,public_key

[nflaig]

A new endpoint has been added recently to provide a minimal response /eth/v1/beacon/states/{state_id}/validator_identities (see postStateValidatorIdentities) which gives you just the index and public key. It's not widely supported yet though, Lodestar will have it in the next release (v1.23.0)

Allow filtering by fields

This is problematic as we want to support both JSON and SSZ responses, in case of SSZ it requires to predefine a container, a generic approach to select fields dynamically would be difficult to apply. We also try to return data structures that are defined by the CL spec.

Curious what others think.

[mcdee]

As @nflaig says, returning only a subset of fields would be a fair bit of pain from the server side, I suspect, as they are likely to map the validator object to it's spec struct for encoding. Field selection would likely be worse for this particular endpoint given that we have the validator struct embedded in the returned data rather than being a top-level object. (so fields would likely have to be fields=index,validator.pubkey and that is already looking ugly).

Note that running /eth/v1/beacon/states/head/validators?status=active with compression enabled is about 68MB of transferred data, so if bandwidth is a major issue this could be alleviated with a proxy.