refactor: improve partitioning code (#2447)

* refactor: improve partitioning code

* fix(firestore-bigquery-change-tracker): update imports

* fix(firestore-bigquery-export): make sure tests are running correctly with the correct project ID

Author: cabljac

firestore-bigquery-export/CONTRIBUTING.md

@@ -1,13 +1,15 @@
 # Development Setup
 
-The firestore-bigquery-export extension is split out into 4 node.js packages, three of
-which are hosted on [npm](https://www.npmjs.com/search?q=firebaseextensions).
+The firestore-bigquery-export extension is split out into 4 node.js packages,
+three of which are hosted on
+[npm](https://www.npmjs.com/search?q=firebaseextensions).
 
 **[firestore-bigquery-change-tracker](https://www.npmjs.com/package/@firebaseextensions/firestore-bigquery-change-tracker)**:
-Contains the core interface defintions for document changes. Also exposes an
-API for uploading changes to BigQuery. The business-logic associated with
-creating the raw changelog and the latest snapshot of live documents in the
-changelog also lives in this package. This package is a dependency for all 3 other packages.
+Contains the core interface defintions for document changes. Also exposes an API
+for uploading changes to BigQuery. The business-logic associated with creating
+the raw changelog and the latest snapshot of live documents in the changelog
+also lives in this package. This package is a dependency for all 3 other
+packages.
 
 **[fs-bq-import-collection](https://www.npmjs.com/package/@firebaseextensions/fs-bq-import-collection)**:
 Contains a script for resumably importing a firestore collection into BigQuery
@@ -17,10 +19,10 @@ using the interface definitions in `firestore-bigquery-change-tracker`.
 Contains a script for generating BigQuery views that provide typed-checked
 access to the changelog created in `firestore-bigquery-change-tracker`.
 
-**firestore-bigquery-export-functions (a part of the extension)**: Contains the definition
-for a Google Cloud function that is called on each write to some collection.
-The function constructs the relevant change event and calls the API in
-`firestore-bigquery-change-tracker` to upload the change to BigQuery.
+**firestore-bigquery-export-functions (a part of the extension)**: Contains the
+definition for a Google Cloud function that is called on each write to some
+collection. The function constructs the relevant change event and calls the API
+in `firestore-bigquery-change-tracker` to upload the change to BigQuery.
 
 ## Building Locally
 
@@ -40,10 +42,10 @@ done;
 
 ### Local package.json file pointers
 
-Npm supports using [local paths as
-dependencies](https://docs.npmjs.com/files/package.json#local-paths) in package.json.
-You'll need to update the following package.json files with local pointers to
-the firestore-bigquery-change-tracker package:
+Npm supports using
+[local paths as dependencies](https://docs.npmjs.com/files/package.json#local-paths)
+in package.json. You'll need to update the following package.json files with
+local pointers to the firestore-bigquery-change-tracker package:
 
 1. firestore-bigquery-export/package.json
 1. firestore-bigquery-export/scripts/import/package.json
@@ -73,6 +75,45 @@ do
 done;
 ```
 
+## Running Integration Tests
+
+Some tests (particularly in `firestore-bigquery-change-tracker`) are integration
+tests that make real calls to BigQuery. These require GCP authentication to be
+configured correctly.
+
+### Prerequisites
+
+1. **Set your GCP project:**
+
+   ```bash
+   gcloud config set project <your-gcp-project-id>
+   ```
+
+2. **Set the quota project for Application Default Credentials:**
+
+   ```bash
+   gcloud auth application-default set-quota-project <your-gcp-project-id>
+   ```
+
+3. **Set the `PROJECT_ID` environment variable** (some tests read this):
+   ```bash
+   export PROJECT_ID=<your-gcp-project-id>
+   ```
+
+### Running Tests
+
+From the `firestore-bigquery-change-tracker` directory:
+
+```bash
+npm test
+```
+
+Or to run a specific test file:
+
+```bash
+npx jest src/__tests__/bigquery/partitioning.test.ts --no-coverage
+```
+
 ## Publishing
 
 _The following instructions are for Firebase team members only._
@@ -87,9 +128,15 @@ npm pack
 npm publish
 ```
 
-In general, you should publish `firestore-bigquery-change-tracker` first, since it doesn't depend on anything else, and all other packages depend on it. Follow these steps:
+In general, you should publish `firestore-bigquery-change-tracker` first, since
+it doesn't depend on anything else, and all other packages depend on it. Follow
+these steps:
 
-1. Publish the package to NPM once the PR has been approved, by checking out the branch.
-1. If a critical bug fix has been made to `firestore-bigquery-change-tracker`, update the dependency version in all 3 other packages. Publish `fs-bq-import-collection` and `fs-bq-schema-views` to NPM.
+1. Publish the package to NPM once the PR has been approved, by checking out the
+   branch.
+1. If a critical bug fix has been made to `firestore-bigquery-change-tracker`,
+   update the dependency version in all 3 other packages. Publish
+   `fs-bq-import-collection` and `fs-bq-schema-views` to NPM.
 1. Merge the PR to `next` branch
-1. If `firestore-bigquery-export-functions` had any changes in code or `package.json`, an extension release is required.
+1. If `firestore-bigquery-export-functions` had any changes in code or
+   `package.json`, an extension release is required.

firestore-bigquery-export/firestore-bigquery-change-tracker/package-lock.json

@@ -5,7 +5,7 @@
     "url": "github.com/firebase/extensions.git",
     "directory": "firestore-bigquery-export/firestore-bigquery-change-tracker"
   },
-  "version": "1.1.42",
+  "version": "2.0.0",
   "description": "Core change-tracker library for Cloud Firestore Collection BigQuery Exports",
   "main": "./lib/index.js",
   "scripts": {
@@ -50,6 +50,7 @@
     "nyc": "^17.1.0",
     "rimraf": "^2.6.3",
     "ts-jest": "29.1.2",
-    "typescript": "^4.9.4"
+    "typescript": "^4.9.4",
+    "wait-for-expect": "^3.0.2"
   }
 }