Access remote artifacts in private repository using secrets

[lbroudoux]

Reason/Context

As of today, it is not possible to access remote artifacts living in a private repository protected with a Secret.

The base microcks-testcontainers-java library allows this to be done easily because it provides withSecret() and withMainRemoteArtifacts() methods that can be used programmatically by the user. The logic of DevServices being different with everything done for you from a set of properties.

Description

We need to define additional properties to enable the definition of secrets and their attachment to remote artifacts.

Implementation ideas

I propose a new set of properties that enable the DevService to create secrets on startup for you. It could be something like:

# For a secret named 'my-secret'. We don't need an additional description.
quarkus.microcks.devservices.secrets.my-secret.type=basic                        // Secret of basic type 
quarkus.microcks.devservices.secrets.my-secret.username=my-username              // Basic secret needs a username and password
quarkus.microcks.devservices.secrets.my-secret.password=env:REPOSITORY_PASSWORD  // Password will be extracted from env variable

# For a secret named 'my-token'. We don't need an additional description.
quarkus.microcks.devservices.secrets.my-token.type=token                    // Secret of token type 
quarkus.microcks.devservices.secrets.my-token.token=env:REPOSITORY_TOKEN    // Token secret needs a token that will be extracted from env variable
quarkus.microcks.devservices.secrets.my-token.token-header=x-my-token       // Optional: if token is not using the standard 'Authorization: Bearer <token>' header form

Now, we also need to configure the association between the remote artifact and the secret to use when downloading it. I propose using an optional suffix just after the URL to specify the secret name to use.

As of today, we have this way to define remote artifacts with a comma-separated list:

quarkus.microcks.devservices.remote-artifacts.primaries=https://raw.githubusercontent.com/microcks/microcks/master/samples/films.graphql,https://myprivaterepo.com/samples/openapi.yaml

As the RF3986 defines the unreserved characters as follow:

unreserved  = ALPHA / DIGIT / "-" / "." / "_" / "~"

and the unreserved characters being:

reserved    = gen-delims / sub-delims
gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"
sub-delims  = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "="

we need to use other characters to split the URL from the secret name.

Thus, I propose using an optional "pipe" delimiter with the secret name. Our above properties definition would then become:

quarkus.microcks.devservices.remote-artifacts.primaries=https://raw.githubusercontent.com/microcks/microcks/master/samples/films.graphql,https://myprivaterepo.com/samples/openapi.yaml|my-secret,<other_remote_artifact_url>

Thus, we'd be able to know that 1st remote artifact is publicly accessible, 2nd one is private, and Microcks must use the my-secret secret, 3rd is public (or private we actually don't know 😉 ).

What do you think of this proposition? Does it make sense? Do you see some other - more easy - solution?

[Basuchi]

Hi @lbroudoux ,
Most of the description fits my requirements well.
However, I feel the implementation on the Microcks side is becoming a bit complex.

We can actually simplify it by authenticating using either a Classic PAT or a Fine-grained PAT, like this:

quarkus.microcks.devservices.remote-artifacts.primaries=https://raw.githubusercontent.com/<repo-path>/hello-1.0.0-openapi.yaml
quarkus.microcks.devservices.remote-artifacts.headers.Authorization=Bearer $MY_TOKEN
quarkus.microcks.devservices.remote-artifacts.headers.Accept=application/vnd.github.raw

end user can store this token in their own repository secrets.

That said, if you are aiming for a different scope or a broader architectural approach, I understand and support proceeding with your solution.

Thank you!.

[lbroudoux]

However, I feel the implementation on the Microcks side is becoming a bit complex.

I see your point, and I agree that it could seem a bit over-engineered for a simple use case. However, we already have users who need to access artifacts from different private repositories. At some point, we need to dissociate secrets from artifacts and allow to create bounds between the two concepts.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 30 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. Microcks is a Cloud Native Computing Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[lbroudoux]

We now need to bump to microcks-testcontainers-java:0.4.4 to get the new APIs! See https://github.com/microcks/microcks-testcontainers-java/releases/tag/0.4.4