Skip to content

Commit 15c34e2

Browse files
nagrahamnagraham
authored
feat: Update R2 Data Catalog compaction commands to work on catalog level (remove table/namespace) (#10622)
* [R2 Data Catalog] Update r2 catalog compaction commands to enable/disable catalog For our intial open-beta, we want customers to have an easier interface for enabling compaction on their catalogs. This change allows them to enable once, so any table they create will have compaction enabled. We will add the ability to control specific tables at a later time. * [R2 Data Catalog] Enable compaction requires a CLOUDFLARE_API_TOKEN For compaction to work, we need customers to grant our system permission to access their R2 Bucket and R2 Data Catalog on their behalf. This permission grant is made via a token they provide when they enable compaciton. We encrypt the token and store it, and use it only via our backend system. Because the token is required for compaction to work, we make it required. * Add change set file * Change the token in 'compaction enable' to be cli arg instead of CLOUDFALRE_API_TOKEN
1 parent ebf2297 commit 15c34e2

File tree

4 files changed

+136
-116
lines changed

4 files changed

+136
-116
lines changed

‎.changeset/thin-webs-wait.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"wrangler": minor
3+
---
4+
5+
Modify R2 Data Catalog compaction commands to enable/disable for Catalog (remove table/namespace args), and require Cloudflare API token on enable.

‎packages/wrangler/src/__tests__/r2.test.ts‎

Lines changed: 48 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -969,7 +969,7 @@ describe("r2", () => {
969969
wrangler r2 bucket catalog enable <bucket> Enable the data catalog on an R2 bucket [open-beta]
970970
wrangler r2 bucket catalog disable <bucket> Disable the data catalog for an R2 bucket [open-beta]
971971
wrangler r2 bucket catalog get <bucket> Get the status of the data catalog for an R2 bucket [open-beta]
972-
wrangler r2 bucket catalog compaction Manage compaction maintenance for tables in your R2 data catalog [private-beta]
972+
wrangler r2 bucket catalog compaction Control settings for automatic file compaction maintenance jobs for your R2 data catalog [open-beta]
973973
974974
GLOBAL FLAGS
975975
-c, --config Path to Wrangler configuration file [string]
@@ -1249,11 +1249,11 @@ For more details, refer to: https://developers.cloudflare.com/r2/api/s3/tokens/"
12491249
"
12501250
wrangler r2 bucket catalog compaction
12511251
1252-
Manage compaction maintenance for tables in your R2 data catalog [private-beta]
1252+
Control settings for automatic file compaction maintenance jobs for your R2 data catalog [open-beta]
12531253
12541254
COMMANDS
1255-
wrangler r2 bucket catalog compaction enable <bucket> Enable compaction maintenance for a table in the R2 data catalog [private-beta]
1256-
wrangler r2 bucket catalog compaction disable <bucket> Disable compaction maintenance for a table in the R2 data catalog [private-beta]
1255+
wrangler r2 bucket catalog compaction enable <bucket> Enable automatic file compaction for your R2 data catalog [open-beta]
1256+
wrangler r2 bucket catalog compaction disable <bucket> Disable automatic file compaction for your R2 data catalog [open-beta]
12571257
12581258
GLOBAL FLAGS
12591259
-c, --config Path to Wrangler configuration file [string]
@@ -1266,16 +1266,32 @@ For more details, refer to: https://developers.cloudflare.com/r2/api/s3/tokens/"
12661266
});
12671267

12681268
describe("enable", () => {
1269-
it("should enable compaction for the given table", async () => {
1269+
it("should enable compaction for the catalog", async () => {
12701270
msw.use(
12711271
http.post(
1272-
"*/accounts/some-account-id/r2-catalog/testBucket/namespaces/testNamespace/tables/testTable/maintenance-configs",
1272+
"*/accounts/some-account-id/r2-catalog/testBucket/credential",
12731273
async ({ request }) => {
12741274
const body = await request.json();
1275+
expect(request.method).toEqual("POST");
12751276
expect(body).toEqual({
1276-
configuration_type: "compaction",
1277-
configuration: {},
1278-
state: "enabled",
1277+
token: "fakecloudflaretoken",
1278+
});
1279+
return HttpResponse.json(
1280+
createFetchResult({ success: true }, true)
1281+
);
1282+
},
1283+
{ once: true }
1284+
),
1285+
http.post(
1286+
"*/accounts/some-account-id/r2-catalog/testBucket/maintenance-configs",
1287+
async ({ request }) => {
1288+
const body = await request.json();
1289+
expect(request.method).toEqual("POST");
1290+
expect(body).toEqual({
1291+
compaction: {
1292+
state: "enabled",
1293+
targetSizeMb: 512,
1294+
},
12791295
});
12801296
return HttpResponse.json(
12811297
createFetchResult({ success: true }, true)
@@ -1285,10 +1301,10 @@ For more details, refer to: https://developers.cloudflare.com/r2/api/s3/tokens/"
12851301
)
12861302
);
12871303
await runWrangler(
1288-
"r2 bucket catalog compaction enable testBucket --table testTable --namespace testNamespace"
1304+
"r2 bucket catalog compaction enable testBucket --token fakecloudflaretoken --targetSizeMb 512"
12891305
);
12901306
expect(std.out).toMatchInlineSnapshot(
1291-
`"✨ Successfully enabled compaction maintenance for table 'testTable' in namespace 'testNamespace' of bucket 'testBucket'."`
1307+
`"✨ Successfully enabled file compaction for the data catalog for bucket 'testBucket'."`
12921308
);
12931309
});
12941310

@@ -1302,10 +1318,10 @@ For more details, refer to: https://developers.cloudflare.com/r2/api/s3/tokens/"
13021318
"
13031319
wrangler r2 bucket catalog compaction enable <bucket>
13041320
1305-
Enable compaction maintenance for a table in the R2 data catalog [private-beta]
1321+
Enable automatic file compaction for your R2 data catalog [open-beta]
13061322
13071323
POSITIONALS
1308-
bucket The name of the bucket [string] [required]
1324+
bucket The name of the bucket which contains the catalog [string] [required]
13091325
13101326
GLOBAL FLAGS
13111327
-c, --config Path to Wrangler configuration file [string]
@@ -1316,8 +1332,8 @@ For more details, refer to: https://developers.cloudflare.com/r2/api/s3/tokens/"
13161332
-v, --version Show version number [boolean]
13171333
13181334
OPTIONS
1319-
--table The name of the table to enable compaction for [string] [required]
1320-
--namespace The namespace containing the table [string] [required]"
1335+
--targetSizeMb The target size for compacted files (allowed values: 64, 128, 256, 512) [number] [default: 128]
1336+
--token A cloudflare api token with access to R2 and R2 Data Catalog which will be used to read/write files for compaction. [string] [required]"
13211337
`);
13221338
expect(std.err).toMatchInlineSnapshot(`
13231339
"X [ERROR] Not enough non-option arguments: got 0, need at least 1
@@ -1326,23 +1342,13 @@ For more details, refer to: https://developers.cloudflare.com/r2/api/s3/tokens/"
13261342
`);
13271343
});
13281344

1329-
it("should error if --table is not provided", async () => {
1345+
it("should error if --token is not provided", async () => {
13301346
await expect(
13311347
runWrangler(
1332-
"r2 bucket catalog compaction enable testBucket --namespace testNamespace"
1348+
"r2 bucket catalog compaction enable testBucket --targetSizeMb 512"
13331349
)
13341350
).rejects.toThrowErrorMatchingInlineSnapshot(
1335-
`[Error: Missing required argument: table]`
1336-
);
1337-
});
1338-
1339-
it("should error if --namespace is not provided", async () => {
1340-
await expect(
1341-
runWrangler(
1342-
"r2 bucket catalog compaction enable testBucket --table testTable"
1343-
)
1344-
).rejects.toThrowErrorMatchingInlineSnapshot(
1345-
`[Error: Missing required argument: namespace]`
1351+
`[Error: Missing required argument: token]`
13461352
);
13471353
});
13481354
});
@@ -1360,22 +1366,18 @@ For more details, refer to: https://developers.cloudflare.com/r2/api/s3/tokens/"
13601366
"
13611367
wrangler r2 bucket catalog compaction disable <bucket>
13621368
1363-
Disable compaction maintenance for a table in the R2 data catalog [private-beta]
1369+
Disable automatic file compaction for your R2 data catalog [open-beta]
13641370
13651371
POSITIONALS
1366-
bucket The name of the bucket [string] [required]
1372+
bucket The name of the bucket which contains the catalog [string] [required]
13671373
13681374
GLOBAL FLAGS
13691375
-c, --config Path to Wrangler configuration file [string]
13701376
--cwd Run as if Wrangler was started in the specified directory instead of the current working directory [string]
13711377
-e, --env Environment to use for operations, and for selecting .env and .dev.vars files [string]
13721378
--env-file Path to an .env file to load - can be specified multiple times - values from earlier files are overridden by values in later files [array]
13731379
-h, --help Show help [boolean]
1374-
-v, --version Show version number [boolean]
1375-
1376-
OPTIONS
1377-
--table The name of the table to disable compaction for [string] [required]
1378-
--namespace The namespace containing the table [string] [required]"
1380+
-v, --version Show version number [boolean]"
13791381
`);
13801382
expect(std.err).toMatchInlineSnapshot(`
13811383
"X [ERROR] Not enough non-option arguments: got 0, need at least 1
@@ -1384,19 +1386,21 @@ For more details, refer to: https://developers.cloudflare.com/r2/api/s3/tokens/"
13841386
`);
13851387
});
13861388

1387-
it("should disable compaction for the given table with confirmation", async () => {
1389+
it("should disable compaction with confirmation", async () => {
13881390
setIsTTY(true);
13891391
mockConfirm({
1390-
text: "Are you sure you want to disable compaction maintenance for table 'testTable' in namespace 'testNamespace' of bucket 'testBucket'?",
1392+
text: "Are you sure you want to disable file compaction for the data catalog for bucket 'testBucket'?",
13911393
result: true,
13921394
});
13931395
msw.use(
1394-
http.put(
1395-
"*/accounts/some-account-id/r2-catalog/testBucket/namespaces/testNamespace/tables/testTable/maintenance-configs/compaction",
1396+
http.post(
1397+
"*/accounts/some-account-id/r2-catalog/testBucket/maintenance-configs",
13961398
async ({ request }) => {
13971399
const body = await request.json();
13981400
expect(body).toEqual({
1399-
state: "disabled",
1401+
compaction: {
1402+
state: "disabled",
1403+
},
14001404
});
14011405
return HttpResponse.json(
14021406
createFetchResult({ success: true }, true)
@@ -1406,44 +1410,24 @@ For more details, refer to: https://developers.cloudflare.com/r2/api/s3/tokens/"
14061410
)
14071411
);
14081412
await runWrangler(
1409-
"r2 bucket catalog compaction disable testBucket --table testTable --namespace testNamespace"
1413+
"r2 bucket catalog compaction disable testBucket"
14101414
);
14111415
expect(std.out).toMatchInlineSnapshot(
1412-
`"Successfully disabled compaction maintenance for table 'testTable' in namespace 'testNamespace' of bucket 'testBucket'."`
1416+
`"Successfully disabled file compaction for the data catalog for bucket 'testBucket'."`
14131417
);
14141418
});
14151419

14161420
it("should cancel disable when confirmation is rejected", async () => {
14171421
setIsTTY(true);
14181422
mockConfirm({
1419-
text: "Are you sure you want to disable compaction maintenance for table 'testTable' in namespace 'testNamespace' of bucket 'testBucket'?",
1423+
text: "Are you sure you want to disable file compaction for the data catalog for bucket 'testBucket'?",
14201424
result: false,
14211425
});
14221426
await runWrangler(
1423-
"r2 bucket catalog compaction disable testBucket --table testTable --namespace testNamespace"
1427+
"r2 bucket catalog compaction disable testBucket"
14241428
);
14251429
expect(std.out).toMatchInlineSnapshot(`"Disable cancelled."`);
14261430
});
1427-
1428-
it("should error if --table is not provided", async () => {
1429-
await expect(
1430-
runWrangler(
1431-
"r2 bucket catalog compaction disable testBucket --namespace testNamespace"
1432-
)
1433-
).rejects.toThrowErrorMatchingInlineSnapshot(
1434-
`[Error: Missing required argument: table]`
1435-
);
1436-
});
1437-
1438-
it("should error if --namespace is not provided", async () => {
1439-
await expect(
1440-
runWrangler(
1441-
"r2 bucket catalog compaction disable testBucket --table testTable"
1442-
)
1443-
).rejects.toThrowErrorMatchingInlineSnapshot(
1444-
`[Error: Missing required argument: namespace]`
1445-
);
1446-
});
14471431
});
14481432
});
14491433
});

‎packages/wrangler/src/r2/catalog.ts‎

Lines changed: 31 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import {
1111
enableR2Catalog,
1212
enableR2CatalogCompaction,
1313
getR2Catalog,
14+
upsertR2DataCatalogCredential,
1415
} from "./helpers";
1516

1617
export const r2BucketCatalogNamespace = createNamespace({
@@ -38,7 +39,6 @@ export const r2BucketCatalogEnableCommand = createCommand({
3839
},
3940
async handler(args, { config }) {
4041
const accountId = await requireAuth(config);
41-
4242
const response = await enableR2Catalog(config, accountId, args.bucket);
4343

4444
let catalogHost: string;
@@ -162,75 +162,72 @@ export const r2BucketCatalogGetCommand = createCommand({
162162
export const r2BucketCatalogCompactionNamespace = createNamespace({
163163
metadata: {
164164
description:
165-
"Manage compaction maintenance for tables in your R2 data catalog",
166-
status: "private-beta",
165+
"Control settings for automatic file compaction maintenance jobs for your R2 data catalog",
166+
status: "open-beta",
167167
owner: "Product: R2 Data Catalog",
168168
},
169169
});
170170

171171
export const r2BucketCatalogCompactionEnableCommand = createCommand({
172172
metadata: {
173-
description:
174-
"Enable compaction maintenance for a table in the R2 data catalog",
175-
status: "private-beta",
173+
description: "Enable automatic file compaction for your R2 data catalog",
174+
status: "open-beta",
176175
owner: "Product: R2 Data Catalog",
177176
},
178177
positionalArgs: ["bucket"],
179178
args: {
180179
bucket: {
181-
describe: "The name of the bucket",
180+
describe: "The name of the bucket which contains the catalog",
182181
type: "string",
183182
demandOption: true,
184183
},
185-
table: {
186-
describe: "The name of the table to enable compaction for",
187-
type: "string",
188-
demandOption: true,
184+
targetSizeMb: {
185+
describe:
186+
"The target size for compacted files (allowed values: 64, 128, 256, 512)",
187+
type: "number",
188+
demandOption: false,
189+
default: 128,
189190
},
190-
namespace: {
191-
describe: "The namespace containing the table",
192-
type: "string",
191+
token: {
192+
describe:
193+
"A cloudflare api token with access to R2 and R2 Data Catalog which will be used to read/write files for compaction.",
193194
demandOption: true,
195+
type: "string",
194196
},
195197
},
196198
async handler(args, { config }) {
197199
const accountId = await requireAuth(config);
198200

201+
await upsertR2DataCatalogCredential(
202+
config,
203+
accountId,
204+
args.bucket,
205+
args.token
206+
);
207+
199208
await enableR2CatalogCompaction(
200209
config,
201210
accountId,
202211
args.bucket,
203-
args.namespace,
204-
args.table
212+
args.targetSizeMb
205213
);
206214

207215
logger.log(
208-
`✨ Successfully enabled compaction maintenance for table '${args.table}' in namespace '${args.namespace}' of bucket '${args.bucket}'.`
216+
`✨ Successfully enabled file compaction for the data catalog for bucket '${args.bucket}'.`
209217
);
210218
},
211219
});
212220

213221
export const r2BucketCatalogCompactionDisableCommand = createCommand({
214222
metadata: {
215-
description:
216-
"Disable compaction maintenance for a table in the R2 data catalog",
217-
status: "private-beta",
223+
description: "Disable automatic file compaction for your R2 data catalog",
224+
status: "open-beta",
218225
owner: "Product: R2 Data Catalog",
219226
},
220227
positionalArgs: ["bucket"],
221228
args: {
222229
bucket: {
223-
describe: "The name of the bucket",
224-
type: "string",
225-
demandOption: true,
226-
},
227-
table: {
228-
describe: "The name of the table to disable compaction for",
229-
type: "string",
230-
demandOption: true,
231-
},
232-
namespace: {
233-
describe: "The namespace containing the table",
230+
describe: "The name of the bucket which contains the catalog",
234231
type: "string",
235232
demandOption: true,
236233
},
@@ -239,23 +236,17 @@ export const r2BucketCatalogCompactionDisableCommand = createCommand({
239236
const accountId = await requireAuth(config);
240237

241238
const confirmedDisable = await confirm(
242-
`Are you sure you want to disable compaction maintenance for table '${args.table}' in namespace '${args.namespace}' of bucket '${args.bucket}'?`
239+
`Are you sure you want to disable file compaction for the data catalog for bucket '${args.bucket}'?`
243240
);
244241
if (!confirmedDisable) {
245242
logger.log("Disable cancelled.");
246243
return;
247244
}
248245

249-
await disableR2CatalogCompaction(
250-
config,
251-
accountId,
252-
args.bucket,
253-
args.namespace,
254-
args.table
255-
);
246+
await disableR2CatalogCompaction(config, accountId, args.bucket);
256247

257248
logger.log(
258-
`Successfully disabled compaction maintenance for table '${args.table}' in namespace '${args.namespace}' of bucket '${args.bucket}'.`
249+
`Successfully disabled file compaction for the data catalog for bucket '${args.bucket}'.`
259250
);
260251
},
261252
});

0 commit comments

Comments
 (0)