Encrypt in a package and export method encrypt to be used in import "go.mozilla.org/sops/v3/encrypt"

[skrese]

In Go projects where you need encrypting with sops it'd be nice to have a direct access to the method func encrypt(opts encryptOpts) (encryptedFile []byte, err error) in cmd/sops/encrypt.go. In that manner you could do something like:

import (
...
    "go.mozilla.org/sops/v3/encrypt"
...
)
...
encryptedFile, _ := encrypt.Encrypt(encrypt.EncryptOpts{
                        InputStore:        &yaml.Store{},
                        OutputStore:       &yaml.Store{},
                        InputBytes:        buffer,
                        Cipher:            aes.NewCipher(),
                        KeyServices:       svcs,
                        UnencryptedSuffix: "_unencrypted",
                        EncryptedSuffix:   "",
                        UnencryptedRegex:  "",
                        EncryptedRegex:    "",
                        KeyGroups:         groups,
                        GroupThreshold:    0,
                    })

[hiddeco]

What justifies exposing this while the underlying API mechanics are publicly available and allow you to reconstruct the behavior with precision?

[skrese]

@hiddeco appreciate for reviewing the code. What I wanted is to include sops as a library in a Go project and then use it for encryption (also decryption) but couldn't achieve that with public API. Hence, exported encrypt.Encrypt and encrypt.EncryptOpts. Have I missed something and I could do it with API? My PR allows me to do:

encryptedFile, _ := encrypt.Encrypt(encrypt.EncryptOpts{
                        InputStore:        &yaml.Store{},
                        OutputStore:       &yaml.Store{},
                        InputBytes:        buffer,
                        Cipher:            aes.NewCipher(),
                        KeyServices:       svcs,
                        UnencryptedSuffix: "_unencrypted",
                        EncryptedSuffix:   "",
                        UnencryptedRegex:  "",
                        EncryptedRegex:    "",
                        KeyGroups:         groups,
                        GroupThreshold:    0,
                    })

[hiddeco]

You can absolutely achieve this using the public API by using the lower-level APIs as they are used within the (now private) encrypt function, the key service client you need to replicate SOPS is available in:

sops/keyservice/client.go

Lines 14 to 17 in 7138185

	// NewLocalClient creates a new local client
	func NewLocalClient() LocalClient {
	return LocalClient{Server{}}
	}

With this, it should be possible using something like:

	// One of https://github.com/getsops/sops/tree/main/stores,
	// for the file type you are working with. There are some utilities to figure
	// out what store to used based on a path, see e.g.
	// common.DefaultStoreForPathOrFormat()
	inputStore := &yaml.Store{}

	inputB := []byte(`<your bytes to encrypt>`)
	branches, err := inputStore.LoadPlainFile(inputB)
	if err != nil {
		return nil, fmt.Errorf("could not load plain file: %w", err)
	}

	tree := sops.Tree{
		Branches: branches,
		Metadata: sops.Metadata{
			// Your options from your own comment.
		},
		// In most scenarios, FilePath is not strictly required if you solely want to encrypt
		// bytes. Internally, they are used for auditing purposes which is not really actively
		// being utilized by 99% of the users.
		// FilePath: "",
	}

	// Local client which is basically how `sops` behaves. If you want to make custom
	// configurations for client secrets, etc. There is a `NewCustomLocalClient()`.
	ks := keyservice.NewLocalClient()
	dataKey, errs := tree.GenerateDataKeyWithKeyServices(ks)
	if len(errs) > 0 {
		return nil, fmt.Errorf("could not generate data key: %s", errs)
	}

	// Perform the encrypt operation.
	if err = common.EncryptTree(common.EncryptTreeOpts{
		DataKey: dataKey,
		Tree:    &tree,
		Cipher:  opts.Cipher,
	}); err != nil {
		return nil, err
	}

	// Output the encrypted bytes.
	encryptedB, err := inputStore.EmitEncryptedFile(tree)
	if err != nil {
		return nil, fmt.Errorf("could not marshal tree: %w", err)
	}

Which will break you free from all sorts of assumptions around returned error codes, etc. specifically designed for the CLI, which do currently exist in the encrypt package. While also providing fine grain control over any possible options you may want to configure now or later, and the encryption process as a whole.

[skrese]

@hiddeco I think you're pointing me to the right direction. Although still not sure how to set up groups (as I did in my original solution):
KeyGroups: groups,
The problem I had, I couldn't set up groups for my purposes, where I needed KMS key and multiple age keys. If I understand correctly you're suggesting I should use NewCustomLocalClient(), right?
This is the code I managed to set up groups:

func (m *MainWrapper) getGroups() ([]sops.KeyGroup, error) {
	var ageMasterKeys []keys.MasterKey
	var kmsKeys []keys.MasterKey

	// read age public keys from secrets manager
	agePublicKeys, err := m.SecretManagerWrapper.GetSecret(&m.agePublicKeysArn, false)
	if err != nil {
		m.logger.Error(fmt.Sprintf("Could not load secret value for %s: %v", m.agePublicKeysArn, err))
		mg.SafeExit(1)
	}
	var secret map[string]string
	err = json.Unmarshal([]byte(*agePublicKeys), &secret)
	if err != nil {
		m.logger.Error(fmt.Sprintf("Error unmarshalling secret: %v", err))
		mg.SafeExit(1)
	}

	// age master keys
	var values []string
	for key, value := range secret {
		if strings.HasPrefix(key, "age_key_") {
			values = append(values, value)
		}
	}
	ageKeysFromSecret := strings.Join(values, ",")
	ageKeys, _ := age.MasterKeysFromRecipients(ageKeysFromSecret)
	for _, k := range ageKeys {
		ageMasterKeys = append(ageMasterKeys, k)
	}

	// kms master keys
	key := &common.KmsKey{
		Arn:     m.kmsArn,
		Context: map[string]string{},
	}
	ctx := make(map[string]*string)
	for k, v := range key.Context {
		value := v // Allocate a new string to prevent the pointer below from referring to only the last iteration value
		ctx[k] = &value
	}
	for _, k := range kms.MasterKeysFromArnString(key.Arn, ctx, key.AwsProfile) {
		kmsKeys = append(kmsKeys, k)
	}

	var group sops.KeyGroup
	group = append(group, ageMasterKeys...)
	group = append(group, kmsKeys...)
	return []sops.KeyGroup{group}, nil
}

[hiddeco]

Key groups are for encryption instructions (and the loading looks pretty much like

sops/cmd/sops/main.go

Line 1088 in 7b3a032

func keyGroups(c *cli.Context, file string) ([]sops.KeyGroup, error) {

, which should be OK).

However, the further private parts are by default resolved from the runtime environment (via NewLocalClient()). If your goal is to have those work in an isolated way, this is now possible in the v3.8.0-rc.1 release using NewCustomLocalClient(), and I posted the ingredients in a Twitter thread (but they should eventually be documented in this repository): https://twitter.com/hiddeco/status/1695065074769997862