feat: experimental support for bulk processing on arbitrum-sepolia-testnet (#463)

* feat: add bulk_cid in requestorder

* feat: raise orders volume limit to MAX_SAFE_INTEGER

* feat: add prepareDatasetBulk

* feat: upload bulk files to thegraph ipfs node

* feat: add bulkOnly filter on dataset orderbook

* chore: update test stack

Author: PierreJeanjacquot

docs/classes/IExecConfig.md

@@ -34,6 +34,7 @@ const wallet = IExecWalletModule.fromConfig(config);
 - [resolveEnsPublicResolverAddress](IExecConfig.md#resolveenspublicresolveraddress)
 - [resolveIexecGatewayURL](IExecConfig.md#resolveiexecgatewayurl)
 - [resolveIpfsGatewayURL](IExecConfig.md#resolveipfsgatewayurl)
+- [resolveIpfsNodeURL](IExecConfig.md#resolveipfsnodeurl)
 - [resolvePocoSubgraphURL](IExecConfig.md#resolvepocosubgraphurl)
 - [resolveResultProxyURL](IExecConfig.md#resolveresultproxyurl)
 - [resolveSmsURL](IExecConfig.md#resolvesmsurl)
@@ -184,6 +185,18 @@ resolve the current IPFS gateway URL
 
 ___
 
+### resolveIpfsNodeURL
+
+▸ **resolveIpfsNodeURL**(): `Promise`<`string`\>
+
+resolve the current IPFS node URL
+
+#### Returns
+
+`Promise`<`string`\>
+
+___
+
 ### resolvePocoSubgraphURL
 
 ▸ **resolvePocoSubgraphURL**(): `Promise`<`string`\>

docs/classes/IExecOrderModule.md

@@ -36,6 +36,7 @@ module exposing order methods
 - [hashRequestorder](IExecOrderModule.md#hashrequestorder)
 - [hashWorkerpoolorder](IExecOrderModule.md#hashworkerpoolorder)
 - [matchOrders](IExecOrderModule.md#matchorders)
+- [prepareDatasetBulk](IExecOrderModule.md#preparedatasetbulk)
 - [publishApporder](IExecOrderModule.md#publishapporder)
 - [publishDatasetorder](IExecOrderModule.md#publishdatasetorder)
 - [publishRequestorder](IExecOrderModule.md#publishrequestorder)
@@ -506,6 +507,43 @@ console.log(`created deal ${dealid} in tx ${txHash}`);
 
 ___
 
+### prepareDatasetBulk
+
+▸ **prepareDatasetBulk**(`datasetorders`, `options?`): `Promise`<{ `cid`: `string` ; `volume`: `number`  }\>
+
+Prepare a bulk from datasetorders to process multiple datasets with a single requestorder
+
+NB:
+- datasetorders used must authorize the requester to use the dataset in for free with an infinite volume (`utils.DATASET_INFINITE_VOLUME`)
+- depending on the number of datasetorders provided and the `maxDatasetPerTask` option, the bulk might require be splitted into multiple tasks to respect the max dataset per task limit, the returned `volume` is the number of tasks required to process the bulk
+
+example:
+```js
+const { bulkCid, volume } = await prepareDatasetBulk(datasetorders, { maxDatasetPerTask: 5 });
+console.log(`bulk_cid: ${bulkCid}, volume: ${volume}`);
+
+const requestorderTemplate = await createRequestorder({
+  app: appAddress,
+  category: 0,
+  volume: volume, // set the volume
+  params: { bulk_cid: bulkCid } // set the bulk cid in the requestorder params
+});
+```
+
+#### Parameters
+
+| Name | Type | Description |
+| :------ | :------ | :------ |
+| `datasetorders` | [`ConsumableDatasetorder`](../interfaces/internal_.ConsumableDatasetorder.md)[] | - |
+| `options?` | `Object` | - |
+| `options.maxDatasetPerTask?` | `number` | Maximum number of datasets to include in a single task **`Default`** ```ts 100 ``` |
+
+#### Returns
+
+`Promise`<{ `cid`: `string` ; `volume`: `number`  }\>
+
+___
+
 ### publishApporder
 
 ▸ **publishApporder**(`apporder`, `options?`): `Promise`<`string`\>

docs/classes/IExecOrderbookModule.md

@@ -152,9 +152,10 @@ console.log('total orders:', count);
 
 | Name | Type | Description |
 | :------ | :------ | :------ |
-| `datasetAddressOrOptions` | `string` \| { `app?`: `string` ; `dataset?`: `string` ; `datasetOwner?`: `string` ; `isAppStrict?`: `boolean` ; `isRequesterStrict?`: `boolean` ; `isWorkerpoolStrict?`: `boolean` ; `maxTag?`: [`Tag`](../modules.md#tag) \| `string`[] ; `minTag?`: [`Tag`](../modules.md#tag) \| `string`[] ; `minVolume?`: [`BNish`](../modules.md#bnish) ; `page?`: `number` ; `pageSize?`: `number` ; `requester?`: `string` ; `workerpool?`: `string`  } | - |
+| `datasetAddressOrOptions` | `string` \| { `app?`: `string` ; `bulkOnly?`: `boolean` ; `dataset?`: `string` ; `datasetOwner?`: `string` ; `isAppStrict?`: `boolean` ; `isRequesterStrict?`: `boolean` ; `isWorkerpoolStrict?`: `boolean` ; `maxTag?`: [`Tag`](../modules.md#tag) \| `string`[] ; `minTag?`: [`Tag`](../modules.md#tag) \| `string`[] ; `minVolume?`: [`BNish`](../modules.md#bnish) ; `page?`: `number` ; `pageSize?`: `number` ; `requester?`: `string` ; `workerpool?`: `string`  } | - |
 | `options?` | `Object` | **`Deprecated`** use first parameter instead migration: replace `fetchDatasetOrderbook(datasetAddress, options)` by `fetchDatasetOrderbook({ dataset: datasetAddress, ...options })` |
 | `options.app?` | `string` | include orders restricted to specified app (use `'any'` to include any app) |
+| `options.bulkOnly?` | `boolean` | filters out orders that don't allow bulk processing (default: `false`) |
 | `options.isAppStrict?` | `boolean` | filters out orders allowing “any” app (default: `false`) |
 | `options.isRequesterStrict?` | `boolean` | filters out orders allowing “any” requester (default: `false`) |
 | `options.isWorkerpoolStrict?` | `boolean` | filters out orders allowing “any” workerpool (default: `false`) |

docs/interfaces/IExecConfigOptions.md

@@ -16,6 +16,7 @@
 - [hubAddress](IExecConfigOptions.md#hubaddress)
 - [iexecGatewayURL](IExecConfigOptions.md#iexecgatewayurl)
 - [ipfsGatewayURL](IExecConfigOptions.md#ipfsgatewayurl)
+- [ipfsNodeURL](IExecConfigOptions.md#ipfsnodeurl)
 - [isNative](IExecConfigOptions.md#isnative)
 - [pocoSubgraphURL](IExecConfigOptions.md#pocosubgraphurl)
 - [providerOptions](IExecConfigOptions.md#provideroptions)
@@ -118,6 +119,14 @@ override the IPFS gateway URL to target a custom instance
 
 ___
 
+### ipfsNodeURL
+
+• `Optional` **ipfsNodeURL**: `string`
+
+override the IPFS node URL to target a custom instance
+
+___
+
 ### isNative
 
 • `Optional` **isNative**: `boolean`

docs/interfaces/internal_.PublishedDatasetorder.md

@@ -10,6 +10,7 @@ published sell order for a dataset
 
 ### Properties
 
+- [bulk](internal_.PublishedDatasetorder.md#bulk)
 - [chainId](internal_.PublishedDatasetorder.md#chainid)
 - [order](internal_.PublishedDatasetorder.md#order)
 - [orderHash](internal_.PublishedDatasetorder.md#orderhash)
@@ -20,6 +21,14 @@ published sell order for a dataset
 
 ## Properties
 
+### bulk
+
+• `Optional` **bulk**: `boolean`
+
+true if the order allows bulk processing
+
+___
+
 ### chainId
 
 • **chainId**: `number`

docs/interfaces/internal_.RequestorderParams.md

@@ -8,6 +8,7 @@
 
 ### Properties
 
+- [bulk\_cid](internal_.RequestorderParams.md#bulk_cid)
 - [iexec\_args](internal_.RequestorderParams.md#iexec_args)
 - [iexec\_input\_files](internal_.RequestorderParams.md#iexec_input_files)
 - [iexec\_result\_encryption](internal_.RequestorderParams.md#iexec_result_encryption)
@@ -17,6 +18,16 @@
 
 ## Properties
 
+### bulk\_cid
+
+• `Optional` **bulk\_cid**: `string`
+
+bulk CID for the request
+
+default none
+
+___
+
 ### iexec\_args
 
 • `Optional` **iexec\_args**: `string`

docs/modules/utils.md

@@ -10,6 +10,7 @@
 
 ### Variables
 
+- [DATASET\_INFINITE\_VOLUME](utils.md#dataset_infinite_volume)
 - [NULL\_ADDRESS](utils.md#null_address)
 - [NULL\_BYTES32](utils.md#null_bytes32)
 
@@ -27,6 +28,19 @@
 
 ## Variables
 
+### DATASET\_INFINITE\_VOLUME
+
+• `Const` **DATASET\_INFINITE\_VOLUME**: `number`
+
+infinite dataset volume
+
+A `Datasetorder` with this volume is considered as having an infinite volume and is eligible to be processed by any number of tasks without decrementing the remaining volume.
+The Dataset owner still can cancel the order at any time.
+
+NB: Infinite volume is represented by the maximum safe integer in JavaScript (`Number.MAX_SAFE_INTEGER`), which is `9007199254740991`.
+
+___
+
 ### NULL\_ADDRESS
 
 • `Const` **NULL\_ADDRESS**: `string`

eslint.config.js

@@ -56,9 +56,6 @@ export default [
       'sonarjs/no-nested-functions': 'warn', // todo refactor to enforce error on this rule
       'sonarjs/todo-tag': 'warn',
     },
-    settings: {
-      'import/ignore': ['ethers', 'graphql-request'],
-    },
   },
   {
     // cli specific files using nodejs globals