feat: new job to backup pins to s3 by joshghent · Pull Request #1844 · web3-storage/web3.storage

joshghent · 2022-09-06T10:10:48Z

⚠️ Requires a new migration on the psa_pins_request table to add the new backup_urls column. ⚠️

About this job

This new cron job which runs every 4 hours, grabs 10,000 pin requests, gets the car file and uploads it to s3.
It behaves in a similar way to nftstorage/backup.
It uses Dagula to grab the car file from the IPFS peer.

github-actions · 2022-09-06T10:14:43Z

Website preview 🔗✨

build log

flea89

This is getting together 🎉
I did a really quick review pass and left some high-level comments for you.

Also noticed some types still need updating across the board.

packages/cron/src/jobs/pins-backup.js

packages/cron/test/pins-backup.spec.js

.github/workflows/cron-backup-pins.yaml

packages/cron/src/bin/pins-backup.js

packages/cron/src/jobs/pins-backup.js

flea89 · 2022-09-09T13:07:08Z

packages/cron/src/jobs/pins-backup.js

+    this.MAX_DAG_SIZE = 1024 * 1024 * 1024 * 32 // don't try to transfer a DAG that's bigger than 32GB
+    this.log = debug('backup:pins')


What happens to this DAGs?
Is there a different approach we have in mind for those?

We don't... but it's a pattern that is found in other areas of the system. So might be worth flagging.

The comment is not pointing to the right line anymore, should be line above https://github.com/web3-storage/web3.storage/pull/1844/files#diff-4df8cd6f6a1194703527832b6e009d1515b1d8f708afe9a5af7c31f446f9aadeR25

@alanshaw you might know more about this.

My thinking was along the lines of that being the max in our ToS and wanting to have a cap at some point on the size of data we're willing to have uploaded/pinned. If you uploaded more than what we've said is the max allowed then we shouldn't be obliged to store that.

But we don't have that limit for pins do we? I can't find the ToS for pins, but I might just be missing it?

Assuming there isn't one, there's a chance content bigger than that is stored there and users should expect it to be migrated?

I've commented out the size check part, because, unless we have strong reasons not to, we should move all the files to eipfs.

Add logging to keep track of enormous dags

I added 2 pieces of logging:

every time for whatever reason dag export fails we log the number of bytes read to that point

If a successful export is greater than 32TiB, we log it.

32TiB or 32 GiB? i think we want to know about pins >= 32 GiB

yeah, I actually added 32 GiB in the code 👍 (see here)

packages/cron/src/jobs/pins-backup.js

joshghent · 2022-09-11T20:55:04Z

Ok, so before this is merged it needs a the migration running which adds a new column backup_urls to the psa_pins_request.
This cron job will need verifying on staging and make sure that it successfully backs up the pins.
After that, it can be safely merged to production.

github-actions · 2022-09-11T20:55:37Z

`package-lock.json` changes

Summary

Status	Count
	98
	6

Click to toggle table visibility

Name	Previous	Current
`@achingbrain/ip-address`	-	8.1.0
`@achingbrain/nat-port-mapper`	-	1.0.7
`@achingbrain/ssdp`	-	4.0.1
`@aws-sdk/lib-storage`	-	3.194.0
`@aws-sdk/middleware-endpoint`	-	3.193.0
`@aws-sdk/util-middleware`	-	3.193.0
`@chainsafe/libp2p-noise`	-	7.0.3
`@libp2p/connection`	-	2.0.4
`@libp2p/crypto`	-	1.0.7
`@libp2p/interface-connection-encrypter`	-	1.0.3
`@libp2p/interface-connection`	-	3.0.2
`@libp2p/interface-keys`	-	1.0.3
`@libp2p/interface-peer-id`	-	1.0.5
`@libp2p/interface-peer-info`	-	1.0.3
`@libp2p/interface-peer-store`	-	1.2.2
`@libp2p/interface-record`	-	2.0.1
`@libp2p/interface-transport`	-	1.0.4
`@libp2p/interfaces`	-	2.0.4
`@libp2p/logger`	-	2.0.2
`@libp2p/mplex`	-	1.2.2
`@libp2p/multistream-select`	-	1.0.6
`@libp2p/peer-collections`	-	2.2.0
`@libp2p/peer-id-factory`	-	1.0.19
`@libp2p/peer-id`	-	1.1.16
`@libp2p/peer-record`	-	1.0.12
`@libp2p/peer-store`	-	1.0.17
`@libp2p/tcp`	-	3.1.2
`@libp2p/tracked-map`	-	1.0.8
`@libp2p/utils`	-	3.0.2
`@libp2p/websockets`	-	3.0.4
`@multiformats/mafmt`	-	11.0.3
`@multiformats/multiaddr-to-uri`	-	9.0.2
`@multiformats/multiaddr`	-	10.5.0
`@noble/secp256k1`	1.4.0	1.7.0
`@stablelib/aead`	-	1.0.1
`@stablelib/binary`	-	1.0.1
`@stablelib/bytes`	-	1.0.1
`@stablelib/chacha`	-	1.0.1
`@stablelib/chacha20poly1305`	-	1.0.1
`@stablelib/constant-time`	-	1.0.1
`@stablelib/hash`	-	1.0.1
`@stablelib/hkdf`	-	1.0.1
`@stablelib/hmac`	-	1.0.1
`@stablelib/int`	-	1.0.1
`@stablelib/keyagreement`	-	1.0.1
`@stablelib/poly1305`	-	1.0.1
`@stablelib/random`	-	1.0.2
`@stablelib/sha256`	-	1.0.1
`@stablelib/wipe`	-	1.0.1
`@stablelib/x25519`	-	1.0.3
`@web3-storage/fast-unixfs-exporter`	-	0.2.1
`abortable-iterator`	-	4.0.2
`aws-sdk`	-	2.1239.0
`byte-access`	-	1.0.1
`clone-regexp`	-	3.0.0
`conf`	10.1.1	10.2.0
`convert-hrtime`	-	5.0.0
`dagula`	-	3.1.1
`datastore-core`	-	7.0.3
`default-gateway`	-	6.0.3
`event-iterator`	-	2.0.0
`format-number`	-	3.0.0
`freeport-promise`	-	2.0.0
`function-timeout`	-	0.1.1
`hashlru`	-	2.3.0
`interface-blockstore`	2.0.2	2.0.3
`interface-datastore`	6.0.3	6.1.1
`is-loopback-addr`	-	2.0.1
`is-regexp`	-	3.1.0
`it-foreach`	-	0.1.1
`it-handshake`	-	4.1.2
`it-length-prefixed`	-	7.0.1
`it-merge`	-	1.0.4
`it-pair`	-	2.0.3
`it-pb-stream`	-	2.0.2
`it-pushable`	-	2.0.2
`it-reader`	-	6.0.1
`it-sort`	-	1.0.1
`it-stream-types`	-	1.0.4
`it-ws`	-	5.0.3
`jmespath`	-	0.16.0
`jsbn`	-	1.1.0
`libp2p`	-	0.37.3
`longbits`	-	1.1.0
`mime-db`	1.51.0	1.52.0
`mime-types`	2.1.34	2.1.35
`mortice`	-	3.0.1
`mutable-proxy`	-	1.0.0
`netmask`	-	2.0.2
`observable-webworkers`	-	2.0.1
`p-queue`	-	7.3.0
`private-ip`	-	2.3.4
`protons-runtime`	-	2.0.2
`sanitize-filename`	-	1.6.3
`set-delayed-interval`	-	1.0.0
`super-regex`	-	0.2.0
`time-span`	-	5.1.0
`truncate-utf8-bytes`	-	1.0.2
`ts-mocha`	-	9.0.2
`uint8-varint`	-	1.0.4
`uint8arraylist`	-	2.3.3
`utf8-byte-length`	-	1.0.4
`wherearewe`	-	2.0.1
`xsalsa20`	-	1.2.0

flea89 · 2022-09-12T10:39:38Z

@alanshaw, while there are still a few tweaks required (ie. some types are missing/need fixing), I wonder if you could review this PR to see if the approach is what you expected it to be.
@joshghent is off for a few days, it'd be great to have your thoughts so that he can tidy everything up and action feedback (if any) from you.

.github/workflows/cron-backup-pins.yaml

alanshaw

You should check with @olizilla that the DB change is similar to what he's expecting to do.

alanshaw · 2022-09-12T12:19:42Z

.github/workflows/cron-backup-pins.yaml

+
+on:
+  schedule:
+    - cron: '*/30 * * * *'


Why not just schedule for the max amount of time a job can run for 6h?

If I understand this correctly the job has 2 goals:

move historical pins to EIPFS

keep moving new psa requests to EIPFS until we move to pickup.

For 2 I guess it's ideal to keep moving stuff as promptly as possible (30 min make sense, even less than that?) while we know the first runs of the job will be super slow (since they will have to go through all the historical data).

Isn't a solution to satisfy both words to keep the schedule as is and set concurrency on the job?

packages/cron/src/jobs/pins-backup.js

alanshaw · 2022-09-12T12:32:46Z

packages/db/postgres/tables.sql

  inserted_at     TIMESTAMP WITH TIME ZONE DEFAULT timezone('utc'::text, now()) NOT NULL,
-  updated_at      TIMESTAMP WITH TIME ZONE DEFAULT timezone('utc'::text, now()) NOT NULL
+  updated_at      TIMESTAMP WITH TIME ZONE DEFAULT timezone('utc'::text, now()) NOT NULL,
+  backup_urls     TEXT[]


Might be nice if this was NOT NULL DEFAULT [] so you don't have to distinguish between null and empty.

Changed :) FWIW, I used the same definition as backup_urls in uploads. Should I change that one too?

packages/cron/src/jobs/pins-backup.js

packages/cron/test/pins-backup.spec.js

flea89 · 2022-10-25T10:27:51Z

@olizilla can you please give this a thorough review 🙏
Let me know if you want to take it from here, or you will need more support from my end as well.

flea89 · 2022-10-28T11:08:38Z

packages/cron/src/jobs/pins-backup.js

+      let reportInterval
+      const libp2p = await getLibp2p()
+      try {
+        const dagula = await Dagula.fromNetwork(libp2p, { peer })


Cache instances for same peer location.

It should be fine to keep a single lilp2p instance

flea89 · 2022-10-28T11:20:22Z

packages/cron/src/jobs/pins-backup.js

+        throw (err)
+      } finally {
+        if (bytesReceived > this.MAX_UPLOAD_DAG_SIZE) {
+          this.log(`⚠️ CID: ${cid} dag is greater than ${this.fmt(this.MAX_UPLOAD_DAG_SIZE)}`)


Add a per batch summary:

failed cids (and their size)

Successful (bigger than MAX_UPLOAD_DAG_SIZE)

Remove all unnecessary per cids logging, log just errors by default. (leave it with higher debug)

Done!

Default logging (DEBUG=backupPins:log ) is quite succinct now, while the job can be run manually passing a more verbose DEBUG=backupPins:* through workflow inputs.

Example of default logging:

❯ DEBUG=backupPins:log npm test --workspace=packages/cron

❯ DEBUG=backupPins:log npm test --workspace=packages/cron

flea89 · 2022-10-28T11:21:23Z

packages/cron/src/jobs/pins-backup.js

+   * @returns {Promise<number | undefined>}
+   */
+
+  // Given for PIN requests we never limited files size we shouldn't check this. ie.


Delete stale code

flea89 · 2022-10-28T11:30:39Z

packages/cron/src/jobs/pins-backup.js

+            Bucket: bucketName,
+            Key: key,
+            Body: bak.content,
+            Metadata: { structure: 'Complete' }


Look into sending checksum for file, reject the upload if the bytes of car don't match the cid

I wonder if this is required.
Looking at the headers sent by the client

'content-type': 'application/xml', 'content-length': '11985', Expect: '100-continue', host: '127.0.0.1', 'x-amz-user-agent': 'aws-sdk-js/3.53.1', 'user-agent': 'aws-sdk-js/3.53.1 os/darwin/21.6.0 lang/js md/nodejs/16.14.0 md/crt-avail api/s3/3.53.1', 'amz-sdk-invocation-id': '25110079-acaf-425f-8933-3527fd8366c7', 'amz-sdk-request': 'attempt=1; max=3', 'x-amz-date': '20221031T122520Z', 'x-amz-content-sha256': 'ebd8a0f42b66a7756aaee73e6275d918143525f125137b584e6b079b364a6b5f', authorization: 'AWS4-HMAC-SHA256 Credential=minioadmin/20221031/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=90453c633c07234480f3319eb3c1b058d25b39a077eb3653063165c5bc137722' }

you can see it sends x-amz-content-sha256 which is the sha256 of the payload and implies the payload is signed (see docs.

If every chunk is hashed and verified I don't think we need the overall one? Or am I missing something?

cc @olizilla

…ually

mbommerez linked an issue Sep 6, 2022 that may be closed by this pull request

Existing PSA_Requests should be available to Elastic Provider #794

Open

flea89 suggested changes Sep 9, 2022

View reviewed changes

joshghent force-pushed the feat/794-copy-pins-to-eips branch from 2d4715a to 6017c94 Compare September 11, 2022 20:55

joshghent requested a review from flea89 September 11, 2022 20:57

flea89 reviewed Sep 12, 2022

View reviewed changes

.github/workflows/cron-backup-pins.yaml Show resolved Hide resolved

alanshaw requested changes Sep 12, 2022

View reviewed changes

joshghent force-pushed the feat/794-copy-pins-to-eips branch from 4edf2bf to b1fd1a9 Compare September 28, 2022 13:33

joshghent temporarily deployed to production September 28, 2022 13:50 Inactive

joshghent requested review from alanshaw and flea89 September 28, 2022 14:06

joshghent added 16 commits October 12, 2022 09:43

feat: added rough draft of the cron job to copy pins into s3

ca658d5

chore: add aws-sdk

cd7a420

feat: add cron

79219bb

feat: added downloading and uploading of file correctly

2cd96c7

chore: added required dependencies

48cb64d

feat: add new backup_urls column to the psa_pin_requests table

55367e7

feat: finished job to backup files to s3

fdb92d3

feat: added new test suite for pins backup

c910b65

feat: add github action to run cron job

bff06de

feat: added cron pins backup script

b34dd12

chore: removed todo

14cbd8c

feat: added test mocks

ef2600e

chore: add mocks for export car

9e6f035

chore: update package-lock, fixed fetch reference

ec58dd2

feat: swapped to a class tructure

0f0cbe5

chore: update the job to create a new backup class instance

d009be1

flea89 added 3 commits October 24, 2022 18:58

Improve logging and more tests

6e1f121

Improve testing and error handling

d573121

Remove only from test

fd02b6a

flea89 temporarily deployed to production October 24, 2022 21:12 Inactive

flea89 requested a review from olizilla October 25, 2022 07:59

flea89 added 2 commits October 25, 2022 09:10

Update logging

395dff7

Update node modules

c58990b

flea89 temporarily deployed to production October 25, 2022 08:17 Inactive

flea89 temporarily deployed to production October 25, 2022 08:39 Inactive

flea89 force-pushed the feat/794-copy-pins-to-eips branch from e6d131b to df32027 Compare October 25, 2022 08:48

Update setup node action

52984d6

flea89 force-pushed the feat/794-copy-pins-to-eips branch from df32027 to 52984d6 Compare October 25, 2022 08:51

flea89 temporarily deployed to production October 25, 2022 08:57 Inactive

Fix dependecies

694788e

flea89 temporarily deployed to production October 25, 2022 09:39 Inactive

Increase cron timeout

9dec63f

flea89 force-pushed the feat/794-copy-pins-to-eips branch from 2fe2914 to 9dec63f Compare October 25, 2022 10:19

flea89 temporarily deployed to production October 25, 2022 10:25 Inactive

flea89 reviewed Oct 28, 2022

View reviewed changes

flea89 added 5 commits October 31, 2022 11:39

Improve logging and remove stale code

61990b2

Cache instances of Dagula and use 1 libp2p

749e20d

Better instanciation of resources

be24923

Default to quiet logging, but allow for more verbose when running man…

7a48df2

…ually

Update cron help text

8eae8f6

joshJarr approved these changes Nov 23, 2022

View reviewed changes

flea89 approved these changes Nov 24, 2022

View reviewed changes

		this.MAX_DAG_SIZE = 1024 * 1024 * 1024 * 32 // don't try to transfer a DAG that's bigger than 32GB
		this.log = debug('backup:pins')

Conversation

joshghent commented Sep 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

About this job

Uh oh!

github-actions bot commented Sep 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Website preview 🔗✨

Uh oh!

flea89 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alanshaw Sep 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

joshghent commented Sep 11, 2022

Uh oh!

github-actions bot commented Sep 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

package-lock.json changes

Summary

Uh oh!

flea89 commented Sep 12, 2022

Uh oh!

Uh oh!

alanshaw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

flea89 commented Oct 25, 2022

Uh oh!

flea89 Oct 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flea89 Oct 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshghent commented Sep 6, 2022 •

edited

Loading

github-actions bot commented Sep 6, 2022 •

edited

Loading

alanshaw Sep 12, 2022 •

edited

Loading

github-actions bot commented Sep 11, 2022 •

edited

Loading

`package-lock.json` changes

flea89 Oct 28, 2022 •

edited

Loading

flea89 Oct 28, 2022 •

edited

Loading