[VC-43753] Update the dataupload package to be compatible with the inventory API #681

wallrj-cyberark · 2025-08-06T15:33:14Z

Closes: https://venafi.atlassian.net/browse/VC-43753

This PR updates the dataupload package for compatibility with the CyberArk Discovery and Context Inventory API, which is now available for testing.
You will find a link to the latest API design in the Jira issue.
The changes have been E2E tested in the PoC branch in WIP: Proof of concept / demo #678
We also need to modify structure of the uploaded data, but I'll do that in a followup PR.

Testing

I added a (temporary) integration test to demonstrate the dataupload client working with the real API

$ go test ./pkg/internal/cyberark/dataupload/... -v -run Real -count 1
=== RUN   TestPostDataReadingsWithOptionsWithRealAPI
    identity.go:328: I0807 09:43:44.914749] made successful request to StartAuthentication source="Identity.doStartAuthentication" summary="NewPackage"
    identity.go:444: I0807 09:43:45.478020] successfully completed AdvanceAuthentication request to CyberArk Identity; login complete username="[email protected]"
--- PASS: TestPostDataReadingsWithOptionsWithRealAPI (2.80s)
PASS
ok      github.com/jetstack/preflight/pkg/internal/cyberark/dataupload  2.818s

Here's the sequence of API calls:

Here's a response from the snapshot-links endpoint:

Here's the PUT request to the AWS presigned URL:

wallrj-cyberark · 2025-08-06T16:45:56Z

pkg/internal/cyberark/dataupload/dataupload.go

@@ -64,15 +72,15 @@ func (c *CyberArkClient) PostDataReadingsWithOptions(ctx context.Context, payloa

 	presignedUploadURL, err := c.retrievePresignedUploadURL(ctx, hex.EncodeToString(checksum.Sum(nil)), opts)
 	if err != nil {
-		return err
+		return fmt.Errorf("while retrieving snapshot upload URL: %s", err)


This extra context around the error helped me know which part of the post operation was failing, during testing. It will also be helpful for us to diagnose problems if customers encounter this error path in the field.
I updated the unit tests accordingly.

wallrj-cyberark · 2025-08-06T16:48:44Z

pkg/internal/cyberark/dataupload/dataupload.go

 	if err != nil {
 		return err
 	}

-	req.Header.Set("Content-Type", "application/json")


The AWS API is very picky about the request headers. With this content type header, the upload fails with the following error:

=== RUN TestPostDataReadingsWithOptionsWithRealAPI identity.go:328: I0806 17:48:18.208472] made successful request to StartAuthentication source="Identity.doStartAuthentication" summary="NewPackage" identity.go:444: I0806 17:48:18.599177] successfully completed AdvanceAuthentication request to CyberArk Identity; login complete username="[email protected]" dataupload_test.go:172: Error Trace: /home/richard/projects/jetstack/jetstack-secure/pkg/internal/cyberark/dataupload/dataupload_test.go:172 Error: Received unexpected error: received response with status code 403: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>ASIAR53UX6ITK7A63SVE</AWSAccessKeyId><StringToSign>PUT application/json 1754499204 x-amz-security-token:IQoJb3JpZ2luX2VjEEEaCXVzLWVhc3QtMSJGMEQCID5ReOMYdbV6+xXUg1buIf83rg/BzGYJJMuBXB+CRk7ZAiBNC/fNLgruAsTg1NUkcdJ+Na63tmasBmp96D/GA4V8virTAwh6EAAaDDEzMjg1MTk1NDIxNCIM6RQ9ZMY Test: TestPostDataReadingsWithOptionsWithRealAPI --- FAIL: TestPostDataReadingsWithOptionsWithRealAPI (8.32s) FAIL FAIL github.com/jetstack/preflight/pkg/internal/cyberark/dataupload 8.336s FAIL

wallrj-cyberark · 2025-08-06T16:49:38Z

pkg/internal/cyberark/dataupload/dataupload.go

-		Checksum           string `json:"checksum_sha256"`
+		ClusterID    string `json:"cluster_id"`
+		Checksum     string `json:"checksum_sha256"`
+		AgentVersion string `json:"agent_version"`


Removed ClusterDescription and added AgentVersion, as per the API design.

wallrj-cyberark · 2025-08-06T16:51:09Z

pkg/internal/cyberark/dataupload/dataupload.go

@@ -120,7 +128,7 @@ func (c *CyberArkClient) retrievePresignedUploadURL(ctx context.Context, checksu

 	req.Header.Set("Content-Type", "application/json")
 	if err := c.authenticateRequest(req); err != nil {
-		return "", fmt.Errorf("failed to authenticate request")
+		return "", fmt.Errorf("failed to authenticate request: %s", err)


I wanted to see the authentication error here to know what has failed.
The only error from the current implementation is no token cached, which means that you've forgotten to call Login on the identity client.

wallrj-cyberark · 2025-08-06T16:55:40Z

pkg/internal/cyberark/dataupload/dataupload_test.go

+		ClusterName: "bb068932-c80d-460d-88df-34bc7f3f3297",
+	})
+	require.NoError(t, err)
+}


A successful run of this test looks like this:

$ go test ./pkg/internal/cyberark/dataupload/... -run Real -v -test.v 6 warning: both GOPATH and GOROOT are the same directory (/home/richard/go); see https://go.dev/wiki/InstallTroubleshooting === RUN TestPostDataReadingsWithOptionsReal identity.go:328: I0806 17:14:42.059639] made successful request to StartAuthentication source="Identity.doStartAuthentication" summary="NewPackage" identity.go:444: I0806 17:14:42.438060] successfully completed AdvanceAuthentication request to CyberArk Identity; login complete username="********@cyberark.cloud.****" --- PASS: TestPostDataReadingsWithOptionsReal (2.67s) PASS ok github.com/jetstack/preflight/pkg/internal/cyberark/dataupload 2.681s

wallrj-cyberark · 2025-08-06T16:56:08Z

pkg/internal/cyberark/dataupload/dataupload_test.go

+	if platformDomain == "" || subdomain == "" || username == "" || secret == "" {
+		t.Skip("Skipping because one of the following environment variables is unset or empty: ARK_PLATFORM_DOMAIN, ARK_SUBDOMAIN, ARK_USERNAME, ARK_SECRET")
+		return
+	}


The skip message looks like this:

$ go test ./pkg/internal/cyberark/dataupload/... -run Real -v warning: both GOPATH and GOROOT are the same directory (/home/richard/go); see https://go.dev/wiki/InstallTroubleshooting === RUN TestPostDataReadingsWithOptionsWithRealAPI dataupload_test.go:137: Skipping because one of the following environment variables is unset or empty: ARK_PLATFORM_DOMAIN, ARK_SUBDOMAIN, ARK_USERNAME, ARK_SECRET --- SKIP: TestPostDataReadingsWithOptionsWithRealAPI (0.00s) PASS ok github.com/jetstack/preflight/pkg/internal/cyberark/dataupload 0.009s

wallrj-cyberark · 2025-08-06T16:56:54Z

pkg/version/version.go

 func SetUserAgent(req *http.Request) {
-	req.Header.Set("User-Agent", UserAgent())
+	req.Header.Set("User-Agent", "Mozilla/5.0 "+UserAgent())
 }


The API team are aware of this and I will pester them to fix it.

… inventory API Signed-off-by: Richard Wall <[email protected]>

wallrj-cyberark force-pushed the VC-43403-inventory-api branch 2 times, most recently from ed75fe8 to 07889c7 Compare August 6, 2025 16:27

wallrj-cyberark changed the title ~~[VC-43406] Update the dataupload package to be compatible inventory API~~ [VC-43406] Update the dataupload package to be compatible with the inventory API Aug 6, 2025

wallrj-cyberark changed the title ~~[VC-43406] Update the dataupload package to be compatible with the inventory API~~ [VC-43753] Update the dataupload package to be compatible with the inventory API Aug 6, 2025

wallrj-cyberark force-pushed the VC-43403-inventory-api branch from 07889c7 to f2ae463 Compare August 6, 2025 16:41

wallrj-cyberark commented Aug 6, 2025

View reviewed changes

Update the dataupload package to be compatible with the implementated…

fa4fb98

… inventory API Signed-off-by: Richard Wall <[email protected]>

wallrj-cyberark force-pushed the VC-43403-inventory-api branch from f2ae463 to fa4fb98 Compare August 7, 2025 08:52

wallrj-cyberark marked this pull request as ready for review August 7, 2025 08:55

wallrj-cyberark requested a review from inteon August 7, 2025 08:55

inteon approved these changes Aug 7, 2025

View reviewed changes

wallrj-cyberark merged commit 1c0ebaa into master Aug 7, 2025
2 checks passed

wallrj-cyberark deleted the VC-43403-inventory-api branch August 7, 2025 10:47

wallrj-cyberark mentioned this pull request Aug 7, 2025

[VC-43793] Add debug roundtripper to discovery and identity clients for easier debugging #683

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[VC-43753] Update the dataupload package to be compatible with the inventory API #681

[VC-43753] Update the dataupload package to be compatible with the inventory API #681

Uh oh!

wallrj-cyberark commented Aug 6, 2025 •

edited

Loading

Uh oh!

wallrj-cyberark Aug 6, 2025

Uh oh!

wallrj-cyberark Aug 6, 2025

Uh oh!

wallrj-cyberark Aug 6, 2025

Uh oh!

wallrj-cyberark Aug 6, 2025

Uh oh!

wallrj-cyberark Aug 6, 2025

Uh oh!

wallrj-cyberark Aug 6, 2025

Uh oh!

wallrj-cyberark Aug 6, 2025

Uh oh!

Uh oh!

Uh oh!

[VC-43753] Update the dataupload package to be compatible with the inventory API #681

[VC-43753] Update the dataupload package to be compatible with the inventory API #681

Uh oh!

Conversation

wallrj-cyberark commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

wallrj-cyberark Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

wallrj-cyberark Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

wallrj-cyberark Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

wallrj-cyberark Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

wallrj-cyberark Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

wallrj-cyberark Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

wallrj-cyberark Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wallrj-cyberark commented Aug 6, 2025 •

edited

Loading