refactor: split up internal pubkey serialization function into compressed/uncompressed variants

[theStack]

This PR splits up the pubkey serialization function secp256k1_eckey_pubkey_serialize into two variants for the compressed (33 bytes) and uncompressed (65 bytes) public key output format each, where only non-infinity group elements as input are allowed. The motivation is to simplify call-sites significantly, as they currently need to introduce two variables and a VERIFY_CHECKs on the return value and the in/out size parameter within a pre-processor block, typically leading to 8 lines of code. By using the new functions, the code is reduced to a single line of code that just calls the function (see #1773). This is helpful for already existing modules on master (ellswift, musig) and upcoming ones (silentpayments, see #1765).

One drawback is that the public API function secp256k1_ec_pubkey_serialize is now slightly more complex (we now call one of two functions instead of a single one, depending on whether the compressed flag is set or not), but that should hopefully not be a problem.

The commits are intentionally kept small to ease review, happy to squash them if that is preferred.

(Kudos to w0xlt for the initial idea (#1765 (review)) and to real-or-random for the suggestion to split the already existing function (#1773 (comment)).)

[w0xlt]

Approach ACK

[real-or-random]

utACK f5e815f

[w0xlt]

ACK f5e815f

[furszy]

Looks good, just left a minor nit. No need to tackle it.

[furszy]

In adb76f8:

nit: could remove ret if you write it as:

if (!secp256k1_pubkey_load(ctx, &Q, pubkey)) return 0;

if (flags & SECP256K1_FLAGS_BIT_COMPRESSION) {
    secp256k1_eckey_pubkey_serialize33(&Q, output);
    *outputlen = 33;
} else {
    secp256k1_eckey_pubkey_serialize65(&Q, output);
    *outputlen = 65;
}
return 1;

[theStack]

Makes sense yeah, will do if I have to retouch

[fanquake]

Note that our downstream fuzzing of secp256k1 is currently broken until we land changes to cryptofuzz to adapt to the API changes here. We are using https://github.com/fanquake/cryptofuzz, however there is some activity in https://github.com/MozillaSecurity/cryptofuzz, which could be(come?) the new upstream (MozillaSecurity/cryptofuzz#3).

[real-or-random]

Note that our downstream fuzzing of secp256k1 is currently broken until we land changes to cryptofuzz to adapt to the API changes here. We are using fanquake/cryptofuzz, however there is some activity in MozillaSecurity/cryptofuzz, which could be(come?) the new upstream (MozillaSecurity/cryptofuzz#3).

working on a PR to your repo

[real-or-random]

fanquake/cryptofuzz#1