Bugfix/s3 utils 188 support metrics above max safe by benzekrimaha · Pull Request #334 · scality/s3utils

benzekrimaha · 2025-01-17T14:58:44Z

No description provided.

bert-e · 2025-01-17T14:58:48Z

Hello benzekrimaha,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2025-01-17T14:58:52Z

Incorrect fix version

The Fix Version/s in issue S3UTILS-188 contains:

1.14.17

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

1.14.18
1.15.8

Please check the Fix Version/s of S3UTILS-188, or the target
branch of this pull request.

benzekrimaha · 2025-01-23T23:17:05Z

/approve

Issue: S3UTILS-188

- Jest version was too old to support BigInts - randomatic was used in some tests, yet never part of the project Issue: S3UTILS-188

Issue: S3UTILS-188

- We are backward compatible - But we now store the values as Long in MongoDB, explicitly, as it would be the case with large values anyway. Issue: S3UTILS-188

- Output always bigint - The conversion in place is to ensure backward compatibility: values might be numbers after an upgrade. It can be safely removed in an upcoming version once we confirm all customers are up to date. Issue: S3UTILS-188

- BigInts are serialized with a __bigint field and passed as string. - When the receiver gets the data, the value is automatically converted into. Issue: S3UTILS-188

ghost · 2025-01-24T12:52:39Z

The PR was updated to :

Compute all metrics with bigints
Store the storage metrics as Long in MongoDB
Store the global metrics (in the document with _id countitems) as numbers, to not change anything. The values here will be imprecise but much more work is required if we want to support it properly everywhere.

package.json

ghost · 2025-01-24T15:39:00Z

PR updated with a unit test to ensure we compute the values properly when we have bigints. Too hard to test as functional test: we won't be uploading large objects in the CI.

CountItems/CountWorker.js

DataReport/collectBucketMetricsAndUpdateBucketCapacityInfo.js

CountItems/utils/utils.js

tests/functional/utils/S3UtilsMongoClient.js

utils/S3UtilsMongoClient.js

tests/functional/utils/S3UtilsMongoClient.js

tests/functional/collectBucketMetricsAndUpdateBucketCapacityInfo.js

Issue: S3UTILS-188

- So we can follow the node naming convention for BigInt Issue: S3UTILS-188

Issue: S3UTILS-188

ghost · 2025-01-28T11:20:56Z

ping

bert-e · 2025-01-28T11:29:10Z

Build failed

The build for commit did not succeed in branch w/1.15/bugfix/S3UTILS-188-support-metrics-above-max-safe

The following options are set: approve

bert-e · 2025-01-28T11:40:27Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/1.14
✔️ development/1.15

The following branches have NOT changed:

development/1.13
development/1.4

Please check the status of the associated issue S3UTILS-188.

Goodbye benzekrimaha.

The following options are set: approve

benzekrimaha force-pushed the bugfix/S3UTILS-188-support-metrics-above-max-safe branch 2 times, most recently from 56798e7 to 9325b3f Compare January 23, 2025 08:25

Seragonia added 8 commits January 24, 2025 09:29

Update eslint config to support modern bigint notations

2dfe551

Issue: S3UTILS-188

Update and correct dev dependencies

d48a14f

- Jest version was too old to support BigInts - randomatic was used in some tests, yet never part of the project Issue: S3UTILS-188

Bump Arsenal

2004fd7

Issue: S3UTILS-188

[maintenance] keep CI up to date

d5d2b0a

Issue: S3UTILS-188

Update bucketVersionsStats to work with BigInts

b877a86

Issue: S3UTILS-188

Update VeeamSOSAPI consumption cronjob to support bigint

4b3da65

- We are backward compatible - But we now store the values as Long in MongoDB, explicitly, as it would be the case with large values anyway. Issue: S3UTILS-188

Handle bigint communications internally

b447ecd

- BigInts are serialized with a __bigint field and passed as string. - When the receiver gets the data, the value is automatically converted into. Issue: S3UTILS-188

ghost force-pushed the bugfix/S3UTILS-188-support-metrics-above-max-safe branch 2 times, most recently from bab3b6d to da1658f Compare January 24, 2025 12:50

ghost reviewed Jan 24, 2025

View reviewed changes

package.json Outdated Show resolved Hide resolved

ghost marked this pull request as ready for review January 24, 2025 14:53

ghost force-pushed the bugfix/S3UTILS-188-support-metrics-above-max-safe branch from da1658f to a8b1892 Compare January 24, 2025 15:37

ghost force-pushed the bugfix/S3UTILS-188-support-metrics-above-max-safe branch 3 times, most recently from e3cc316 to d396d9c Compare January 25, 2025 01:15

ghost reviewed Jan 27, 2025

View reviewed changes

CountItems/CountWorker.js Outdated Show resolved Hide resolved

ghost force-pushed the bugfix/S3UTILS-188-support-metrics-above-max-safe branch from c6ba83c to c034938 Compare January 27, 2025 08:12

ghost reviewed Jan 27, 2025

View reviewed changes

DataReport/collectBucketMetricsAndUpdateBucketCapacityInfo.js Show resolved Hide resolved