Skip to content

Commit ec61f9f

Browse files
feat(sdk): add bulk processing support for email sending (#235)
* chore(deps): add bulk processing support dependencies - Move @iexec/dataprotector to dependencies - Update @iexec/dataprotector to beta version with bulk processing support - Update iexec SDK to version with bulk processing support * feat(types): add bulk processing types and GrantedAccess - Add GrantedAccess type for detailed dataset order information - Update Contact type to include grantedAccess field - Update SendEmailParams to support grantedAccess array - Add ProcessBulkRequestResponse and SendEmailBulkResponse types - Add bulkOnly parameter to contact fetching params * feat(core): integrate IExecDataProtectorCore for bulk processing - Add dataProtector instance to IExecWeb3mail class - Update sendEmail return type to ProcessBulkRequestResponse | SendEmailSingleResponse - Initialize dataProtector with proper configuration matching web3telegram pattern - Update resolveConfig to create dataProtector instance with subgraphUrl, ipfsGateway, and ipfsNode * feat(sendEmail): add bulk processing support with grantedAccess - Refactor sendEmail to support both single and bulk processing modes - Add bulk processing logic using dataProtector.prepareBulkRequest and processBulkRequest - Update single processing to use dataProtector.processProtectedData - Add validation for grantedAccess parameter - Support grantedAccess array for bulk email sending * feat(contacts): add bulkOnly parameter and grantedAccess support - Add bulkOnly parameter to fetchMyContacts and fetchUserContacts - Update fetchUserContacts to populate grantedAccess field with full dataset order details - Pass bulkOnly as allowBulk to fetchDatasetOrderbook - Update subgraphQuery.getValidContact to include grantedAccess in returned contacts - Filter null/undefined contacts matching web3telegram pattern * test(unit): add bulk processing unit tests - Add unit tests for grantedAccess parameter in sendEmail - Add unit tests for bulkOnly parameter in fetchMyContacts - Add test for grantedAccess property in returned contacts - Update sendEmail unit tests to match web3telegram pattern - Add comprehensive bulk processing test coverage * test(e2e): add bulkOnly parameter e2e tests - Add bulkOnly parameter tests to fetchMyContacts e2e suite - Add bulkOnly parameter tests to fetchUserContacts e2e suite - Test bulk access filtering and default behavior - Match web3telegram e2e test patterns * test(e2e): add bulk email sending e2e tests - Fix sendEmail e2e test type guards for ProcessBulkRequestResponse | SendEmailSingleResponse - Add sendEmailBulk e2e test file for bulk processing - Update error message expectations for processProtectedData errors - Test bulk email sending with grantedAccess array * test: fix constructor test and update test utilities - Fix constructor test error message expectation (remove dappAddress from missing config) - Update test utilities for bulk processing support * test(e2e): fix sendEmail e2e test error handling - Fix voucher error test to check error message in cause chain - Update error message expectations to handle wrapped errors from processProtectedData - Fix timeout issue in beforeEach hook for workerpool order test * feat(sendEmail): add missing parameters to processProtectedData - Add defaultWorkerpool parameter - Add workerpoolMaxPrice, dataMaxPrice, appMaxPrice parameters - Add useVoucher parameter - Add dataMaxPrice validation * style: format code with prettier - Format fetchMyContacts validation to single line - Format beforeAll hook in fetchUserContacts test * fix(types): add grantedAccess property to test contact objects - Add grantedAccess mock data to all test contact objects - Update test expectations to include grantedAccess property - Fix TypeScript errors in subgraphQuery.test.ts * feat: add prepareEmailCampaign and sendEmailCampaign methods for bulk processing - Add prepareEmailCampaign to prepare bulk email campaigns with encryption and IPFS upload - Add sendEmailCampaign to process prepared bulk campaigns - Support for bulk email processing using @iexec/dataprotector SDK * feat: add types for bulk email campaign processing - Add PrepareEmailCampaignParams and PrepareEmailCampaignResponse types - Add SendEmailCampaignParams and ProcessBulkRequestResponse types - Add GrantedAccess type for detailed access grant information - Update Contact type to include grantedAccess field * feat: integrate bulk email campaign methods into IExecWeb3mail - Add prepareEmailCampaign method to IExecWeb3mail class - Add sendEmailCampaign method to IExecWeb3mail class - Initialize IExecDataProtectorCore instance for bulk processing support * test: add e2e tests for bulk email campaign methods - Add prepareEmailCampaign e2e tests - Add sendEmailCampaign e2e tests with comprehensive coverage - Test error handling, validation, and integration scenarios * test: add prepare-iexec script to disable checkImplementedOnChain - Add script to disable checkImplementedOnChain in iexec config module - Required for testing on experimental networks * fix(test): update constructor test to use isUserStrict for accurate contact filtering - Set isUserStrict: true in fetchUserContacts test to only get user-specific orders - Fix test expecting empty array for random wallet with no grants * chore: update dataprotector and iexec deps to latest stable version * chore: remove obsolete sendEmailBulk test file - Remove sendEmailBulk.test.ts as bulk processing is now handled by prepareEmailCampaign and sendEmailCampaign * refactor: restore sendEmail to original single email sending implementation - Remove bulk processing logic from sendEmail - Restore orderbook-based single email sending - Bulk processing now handled by prepareEmailCampaign and sendEmailCampaign methods * fix(test): handle transaction already imported error in ensureSufficientStake - Add error handling for parallel test execution in CI - Handle 'transaction already imported' error (-32003) gracefully - Wait and re-check balance after transaction submission - Make deposit operation idempotent for parallel test execution - Prevents nonce conflicts when multiple tests run simultaneously * fix: update types * fix: update types * chore: stop using deprecated function * chore: update unit tests * fix: update subgraphQuery Co-authored-by: pjt <[email protected]> * fix: remove as any IExecWeb3mail Co-authored-by: pjt <[email protected]> * fix: remove as any IExecWeb3mail Co-authored-by: pjt <[email protected]> * fix: avoid wrapping ValidationError into WorkflowError Co-authored-by: pjt <[email protected]> * fix: add missing imports * chore: remove deprecate function call * fix: e2e tests * fix(test): update e2e tests to match SDK implementation and fix timeouts - Update sendEmailCampaign tests to accept ValidationError and TypeError where appropriate - Fix workerpool matching in tests by passing correct workerpoolAddressOrEns - Increase prepareEmailCampaign beforeEach timeout * fix: no waiting for response in a campaign sending * docs: improve documentation in types * chore: clean package.json * fix: use grantedAccess plural form for GrantedAccess list * chore: remove useless params from sendEmailCampaign * feat: add campaignRequest validation * fix: add dealId in sendEmail response * fix(validators): improve campaignRequest schema validation error message - Add strict mode to prevent undefined coercion to empty object - Add custom test to check for undefined/null/empty object before field validation - Fix error message to show 'campaignRequest is required' instead of 'sign is a required field' when campaignRequest is undefined --------- Co-authored-by: Pierre Jeanjacquot <[email protected]>
1 parent 0fd827f commit ec61f9f

23 files changed

+2147
-169
lines changed

package-lock.json

Lines changed: 283 additions & 64 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

package.json

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,7 @@
2020
"scripts": {
2121
"build": "rm -rf dist && tsc --project tsconfig.build.json",
2222
"check-types": "tsc --noEmit",
23-
"test:prepare": "node tests/scripts/prepare-bellecour-fork-for-tests.js",
24-
"test": "NODE_OPTIONS=--experimental-vm-modules jest --testMatch \"**/tests/**/*.test.ts\" --forceExit -b",
23+
"test:prepare": "node tests/scripts/prepare-bellecour-fork-for-tests.js && node tests/scripts/prepare-iexec.js",
2524
"test:coverage": "NODE_OPTIONS=--experimental-vm-modules jest --testMatch \"**/tests/**/*.test.ts\" --forceExit --coverage",
2625
"test:unit": "NODE_OPTIONS=--experimental-vm-modules jest --testMatch \"**/tests/unit/**/*.test.ts\" -b",
2726
"test:unit:coverage": "NODE_OPTIONS=--experimental-vm-modules jest --testMatch \"**/tests/unit/**/*.unit.ts\" --coverage",
@@ -31,7 +30,7 @@
3130
"format": "prettier --write \"(src|tests)/**/*.ts\"",
3231
"check-format": "prettier --check \"(src|tests)/**/*.ts\"",
3332
"stop-test-stack": "cd tests && docker compose down --volumes --remove-orphans",
34-
"start-test-stack": "cd tests && npm run stop-test-stack && node scripts/prepare-test-env.js && docker compose build && docker compose up -d && node scripts/prepare-bellecour-fork-for-tests.js"
33+
"start-test-stack": "cd tests && npm run stop-test-stack && node scripts/prepare-test-env.js && docker compose build && docker compose up -d && npm run test:prepare"
3534
},
3635
"repository": {
3736
"type": "git",
@@ -49,15 +48,15 @@
4948
"dependencies": {
5049
"@ethersproject/bytes": "^5.7.0",
5150
"@ethersproject/random": "^5.7.0",
51+
"@iexec/dataprotector": "^2.0.0-beta.21",
5252
"buffer": "^6.0.3",
5353
"ethers": "^6.13.2",
5454
"graphql-request": "^6.1.0",
55-
"iexec": "^8.20.0",
55+
"iexec": "^8.22.0",
5656
"kubo-rpc-client": "^4.1.1",
5757
"yup": "^1.1.1"
5858
},
5959
"devDependencies": {
60-
"@iexec/dataprotector": "^2.0.0-beta.19",
6160
"@jest/globals": "^29.7.0",
6261
"@swc/core": "^1.3.96",
6362
"@swc/jest": "^0.2.29",

src/utils/subgraphQuery.ts

Lines changed: 17 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -71,20 +71,23 @@ export const getValidContact = async (
7171
);
7272

7373
// Convert protectedData[] into Contact[] using the map for constant time lookups
74-
return protectedDataList.map(({ id, name }) => {
75-
const contact = contactsMap.get(id);
76-
if (contact) {
77-
return {
78-
address: id,
79-
name: name,
80-
remainingAccess: contact.remainingAccess,
81-
accessPrice: contact.accessPrice,
82-
owner: contact.owner,
83-
accessGrantTimestamp: contact.accessGrantTimestamp,
84-
isUserStrict: contact.isUserStrict,
85-
};
86-
}
87-
});
74+
return protectedDataList
75+
.map(({ id, name }) => {
76+
const contact = contactsMap.get(id);
77+
if (contact) {
78+
return {
79+
address: id,
80+
name: name,
81+
remainingAccess: contact.remainingAccess,
82+
accessPrice: contact.accessPrice,
83+
owner: contact.owner,
84+
accessGrantTimestamp: contact.accessGrantTimestamp,
85+
isUserStrict: contact.isUserStrict,
86+
grantedAccess: contact.grantedAccess,
87+
};
88+
}
89+
})
90+
.filter((contact) => !!contact);
8891
} catch (error) {
8992
throw new WorkflowError({
9093
message: 'Failed to fetch subgraph',

src/utils/validators.ts

Lines changed: 68 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import { isAddress } from 'ethers';
22
import { IExec } from 'iexec';
3-
import { ValidationError, boolean, number, string } from 'yup';
3+
import { NULL_ADDRESS } from 'iexec/utils';
4+
import { ValidationError, boolean, number, object, string } from 'yup';
45

56
export const isValidProvider = async (iexec: IExec) => {
67
const client = await iexec.config.resolveContractsClient();
@@ -61,3 +62,69 @@ export const positiveNumberSchema = () =>
6162

6263
export const booleanSchema = () =>
6364
boolean().strict().typeError('${path} should be a boolean');
65+
66+
const isPositiveIntegerStringTest = (value: string) => /^\d+$/.test(value);
67+
68+
const stringSchema = () =>
69+
string().strict().typeError('${path} should be a string');
70+
71+
const positiveIntegerStringSchema = () =>
72+
string().test(
73+
'is-positive-int',
74+
'${path} should be a positive integer',
75+
(value) => isUndefined(value) || isPositiveIntegerStringTest(value)
76+
);
77+
78+
const positiveStrictIntegerStringSchema = () =>
79+
string().test(
80+
'is-positive-strict-int',
81+
'${path} should be a strictly positive integer',
82+
(value) =>
83+
isUndefined(value) ||
84+
(value !== '0' && isPositiveIntegerStringTest(value))
85+
);
86+
87+
export const campaignRequestSchema = () =>
88+
object({
89+
app: addressSchema().required(),
90+
appmaxprice: positiveIntegerStringSchema().required(),
91+
workerpool: addressSchema().required(),
92+
workerpoolmaxprice: positiveIntegerStringSchema().required(),
93+
dataset: addressSchema().oneOf([NULL_ADDRESS]).required(),
94+
datasetmaxprice: positiveIntegerStringSchema().oneOf(['0']).required(),
95+
params: stringSchema()
96+
.test(
97+
'is-valid-bulk-params',
98+
'${path} should be a valid JSON string with bulk_cid field',
99+
(value) => {
100+
try {
101+
// eslint-disable-next-line @typescript-eslint/naming-convention
102+
const { bulk_cid } = JSON.parse(value);
103+
if (typeof bulk_cid === 'string') {
104+
return true;
105+
}
106+
} catch {}
107+
return false;
108+
}
109+
)
110+
.required(),
111+
requester: addressSchema().required(),
112+
beneficiary: addressSchema().required(),
113+
callback: addressSchema().required(),
114+
category: positiveIntegerStringSchema().required(),
115+
volume: positiveStrictIntegerStringSchema().required(),
116+
tag: stringSchema().required(),
117+
trust: positiveIntegerStringSchema().required(),
118+
salt: stringSchema().required(),
119+
sign: stringSchema().required(),
120+
})
121+
.strict()
122+
.typeError('${path} should be a BulkRequest object')
123+
.test('is-defined', '${path} is required', (value) => {
124+
// Check if value is undefined, null, or an empty object (which would be coerced from undefined)
125+
return (
126+
value !== undefined &&
127+
value !== null &&
128+
!(typeof value === 'object' && Object.keys(value).length === 0)
129+
);
130+
});

src/web3mail/IExecWeb3mail.ts

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,12 @@
11
import { AbstractProvider, AbstractSigner, Eip1193Provider } from 'ethers';
22
import { IExec } from 'iexec';
3+
import { IExecDataProtectorCore } from '@iexec/dataprotector';
34
import { GraphQLClient } from 'graphql-request';
45
import { fetchUserContacts } from './fetchUserContacts.js';
56
import { fetchMyContacts } from './fetchMyContacts.js';
67
import { sendEmail } from './sendEmail.js';
8+
import { prepareEmailCampaign } from './prepareEmailCampaign.js';
9+
import { sendEmailCampaign } from './sendEmailCampaign.js';
710
import {
811
Contact,
912
FetchUserContactsParams,
@@ -13,6 +16,10 @@ import {
1316
SendEmailResponse,
1417
Web3SignerProvider,
1518
FetchMyContactsParams,
19+
PrepareEmailCampaignParams,
20+
PrepareEmailCampaignResponse,
21+
SendEmailCampaignParams,
22+
SendEmailCampaignResponse,
1623
} from './types.js';
1724
import { isValidProvider } from '../utils/validators.js';
1825
import { getChainIdFromProvider } from '../utils/getChainId.js';
@@ -34,6 +41,7 @@ interface Web3mailResolvedConfig {
3441
ipfsGateway: string;
3542
defaultWorkerpool: string;
3643
iexec: IExec;
44+
dataProtector: IExecDataProtectorCore;
3745
}
3846

3947
export class IExecWeb3mail {
@@ -51,6 +59,8 @@ export class IExecWeb3mail {
5159

5260
private iexec!: IExec;
5361

62+
private dataProtector!: IExecDataProtectorCore;
63+
5464
private initPromise: Promise<void> | null = null;
5565

5666
private ethProvider: EthersCompatibleProvider;
@@ -75,6 +85,7 @@ export class IExecWeb3mail {
7585
this.ipfsGateway = config.ipfsGateway;
7686
this.defaultWorkerpool = config.defaultWorkerpool;
7787
this.iexec = config.iexec;
88+
this.dataProtector = config.dataProtector;
7889
});
7990
}
8091
return this.initPromise;
@@ -121,6 +132,36 @@ export class IExecWeb3mail {
121132
});
122133
}
123134

135+
async prepareEmailCampaign(
136+
args: PrepareEmailCampaignParams
137+
): Promise<PrepareEmailCampaignResponse> {
138+
await this.init();
139+
await isValidProvider(this.iexec);
140+
return prepareEmailCampaign({
141+
...args,
142+
workerpoolAddressOrEns:
143+
args.workerpoolAddressOrEns || this.defaultWorkerpool,
144+
iexec: this.iexec,
145+
dataProtector: this.dataProtector,
146+
ipfsNode: this.ipfsNode,
147+
ipfsGateway: this.ipfsGateway,
148+
dappAddressOrENS: this.dappAddressOrENS,
149+
});
150+
}
151+
152+
async sendEmailCampaign(
153+
args: SendEmailCampaignParams
154+
): Promise<SendEmailCampaignResponse> {
155+
await this.init();
156+
await isValidProvider(this.iexec);
157+
return sendEmailCampaign({
158+
...args,
159+
workerpoolAddressOrEns:
160+
args.workerpoolAddressOrEns || this.defaultWorkerpool,
161+
dataProtector: this.dataProtector,
162+
});
163+
}
164+
124165
private async resolveConfig(): Promise<Web3mailResolvedConfig> {
125166
const chainId = await getChainIdFromProvider(this.ethProvider);
126167
const chainDefaultConfig = getChainDefaultConfig(chainId, {
@@ -184,6 +225,17 @@ export class IExecWeb3mail {
184225
throw new Error(`Failed to create GraphQLClient: ${error.message}`);
185226
}
186227

228+
const dataProtector = new IExecDataProtectorCore(this.ethProvider, {
229+
iexecOptions: {
230+
ipfsGatewayURL: ipfsGateway,
231+
...this.options?.iexecOptions,
232+
allowExperimentalNetworks: this.options.allowExperimentalNetworks,
233+
},
234+
ipfsGateway,
235+
ipfsNode,
236+
subgraphUrl,
237+
});
238+
187239
return {
188240
dappAddressOrENS,
189241
dappWhitelistAddress: dappWhitelistAddress.toLowerCase(),
@@ -192,6 +244,7 @@ export class IExecWeb3mail {
192244
ipfsNode,
193245
ipfsGateway,
194246
iexec,
247+
dataProtector,
195248
};
196249
}
197250
}

src/web3mail/fetchMyContacts.ts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ export const fetchMyContacts = async ({
1616
dappAddressOrENS = throwIfMissing(),
1717
dappWhitelistAddress = throwIfMissing(),
1818
isUserStrict = false,
19+
bulkOnly = false,
1920
}: IExecConsumer &
2021
SubgraphConsumer &
2122
DappAddressConsumer &
@@ -24,6 +25,7 @@ export const fetchMyContacts = async ({
2425
const vIsUserStrict = booleanSchema()
2526
.label('isUserStrict')
2627
.validateSync(isUserStrict);
28+
const vBulkOnly = booleanSchema().label('bulkOnly').validateSync(bulkOnly);
2729

2830
const userAddress = await iexec.wallet.getAddress();
2931
return fetchUserContacts({
@@ -33,5 +35,6 @@ export const fetchMyContacts = async ({
3335
dappWhitelistAddress,
3436
userAddress,
3537
isUserStrict: vIsUserStrict,
38+
bulkOnly: vBulkOnly,
3639
});
3740
};

src/web3mail/fetchUserContacts.ts

Lines changed: 28 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ export const fetchUserContacts = async ({
2727
dappWhitelistAddress = throwIfMissing(),
2828
userAddress,
2929
isUserStrict = false,
30+
bulkOnly = false,
3031
}: IExecConsumer &
3132
SubgraphConsumer &
3233
DappAddressConsumer &
@@ -47,6 +48,7 @@ export const fetchUserContacts = async ({
4748
const vIsUserStrict = booleanSchema()
4849
.label('isUserStrict')
4950
.validateSync(isUserStrict);
51+
const vBulkOnly = booleanSchema().label('bulkOnly').validateSync(bulkOnly);
5052

5153
try {
5254
const [dappOrders, whitelistOrders] = await Promise.all([
@@ -55,12 +57,14 @@ export const fetchUserContacts = async ({
5557
userAddress: vUserAddress,
5658
appAddress: vDappAddressOrENS,
5759
isUserStrict: vIsUserStrict,
60+
bulkOnly: vBulkOnly,
5861
}),
5962
fetchAllOrdersByApp({
6063
iexec,
6164
userAddress: vUserAddress,
6265
appAddress: vDappWhitelistAddress,
6366
isUserStrict: vIsUserStrict,
67+
bulkOnly: vBulkOnly,
6468
}),
6569
]);
6670

@@ -84,6 +88,18 @@ export const fetchUserContacts = async ({
8488
accessPrice: order.order.datasetprice,
8589
accessGrantTimestamp: order.publicationTimestamp,
8690
isUserStrict: order.order.requesterrestrict !== ZeroAddress,
91+
grantedAccess: {
92+
dataset: order.order.dataset,
93+
datasetprice: order.order.datasetprice.toString(),
94+
volume: order.order.volume.toString(),
95+
tag: order.order.tag.toString(),
96+
apprestrict: order.order.apprestrict,
97+
workerpoolrestrict: order.order.workerpoolrestrict,
98+
requesterrestrict: order.order.requesterrestrict,
99+
salt: order.order.salt,
100+
sign: order.order.sign,
101+
remainingAccess: order.remaining,
102+
},
87103
};
88104
myContacts.push(contact);
89105
}
@@ -107,23 +123,24 @@ async function fetchAllOrdersByApp({
107123
userAddress,
108124
appAddress,
109125
isUserStrict,
126+
bulkOnly,
110127
}: {
111128
iexec: IExec;
112129
userAddress: string;
113130
appAddress: string;
114131
isUserStrict: boolean;
132+
bulkOnly: boolean;
115133
}): Promise<PublishedDatasetorder[]> {
116-
const ordersFirstPage = iexec.orderbook.fetchDatasetOrderbook(
117-
ANY_DATASET_ADDRESS,
118-
{
119-
app: appAddress,
120-
requester: userAddress,
121-
isAppStrict: true,
122-
isRequesterStrict: isUserStrict,
123-
// Use maxPageSize here to avoid too many round-trips (we want everything anyway)
124-
pageSize: 1000,
125-
}
126-
);
134+
const ordersFirstPage = iexec.orderbook.fetchDatasetOrderbook({
135+
dataset: ANY_DATASET_ADDRESS,
136+
app: appAddress,
137+
requester: userAddress,
138+
isAppStrict: true,
139+
isRequesterStrict: isUserStrict,
140+
bulkOnly,
141+
// Use maxPageSize here to avoid too many round-trips (we want everything anyway)
142+
pageSize: 1000,
143+
});
127144
const { orders: allOrders } = await autoPaginateRequest({
128145
request: ordersFirstPage,
129146
});

src/web3mail/internalTypes.ts

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import { IExec } from 'iexec';
2+
import { IExecDataProtectorCore } from '@iexec/dataprotector';
23
import { AddressOrENS } from './types.js';
34
import { GraphQLClient } from 'graphql-request';
45

@@ -34,3 +35,7 @@ export type IExecConsumer = {
3435
export type SubgraphConsumer = {
3536
graphQLClient: GraphQLClient;
3637
};
38+
39+
export type DataProtectorConsumer = {
40+
dataProtector: IExecDataProtectorCore;
41+
};

0 commit comments

Comments
 (0)