|
| 1 | +# Usage |
| 2 | + |
| 3 | +## Using the extension |
| 4 | + |
| 5 | +The extension uses multiple mechanisms to find data to be exported. |
| 6 | + |
| 7 | +## By path |
| 8 | + |
| 9 | +When configuring the Cloud Firestore, Realtime Database & Cloud Storage paths in the configuration, it’s possible to define a `UID` variable in the paths which will be replaced with the authenticated users UID. When the `exportUserData` callable function is called from an authenticated client, the extension will start to export data from the given paths, for example: |
| 10 | + |
| 11 | +- Cloud Firestore path(s): `users/{UID},movies` |
| 12 | +- Realtime Database path(s): `likes/{UID}` |
| 13 | +- Cloud Storage path(s): `{DEFAULT}/uploads/{UID},{DEFAULT}/avatars/{UID}.jpeg` |
| 14 | + |
| 15 | +Be aware of the following behavioral differences between each service: |
| 16 | + |
| 17 | +- Firestore: if a document path is specified, a single CSV file for that document will be created. If a collection path is specified, a single CSV file containing all document data for that collection will be created. |
| 18 | +- Realtime Database: all data at the specified node will be exported to a single CSV per specified path. |
| 19 | +- Storage: if a file path is specified, the single file will be exported (copied). If a directory path is specified, all files within that directory will be exported. |
| 20 | + |
| 21 | +## Custom function hook |
| 22 | + |
| 23 | +For cases where data discovery requires complex queries to identify, it is possible to define a custom function hook URL. The function that accepts a UID should return an object containing Firestore, Realtime Database and/or Storage paths to export. For example: |
| 24 | + |
| 25 | +```js |
| 26 | +export const getCustomExportPaths = functions.https.onRequest( |
| 27 | + async (req, res) => { |
| 28 | + const uid = req.body.uid; |
| 29 | + |
| 30 | + // Perform custom query |
| 31 | + |
| 32 | + return { |
| 33 | + firestorePaths: [‘/document/path’], |
| 34 | + databasePaths: [‘path/to/node’], |
| 35 | + storagePaths: [‘/storage/path’], |
| 36 | + }; |
| 37 | + }); |
| 38 | +``` |
| 39 | + |
| 40 | +## Archiving exported data |
| 41 | + |
| 42 | +By default the extension will generate a list of CSV files. If you wish to additionally export a single zip archive containing all files, set the `Zip’ configuration option to true. |
| 43 | + |
| 44 | +**Note: Archiving is resource intensive and you may run into resource limitations with large volumes of exported data.** |
| 45 | + |
| 46 | +To manually export the data into an archive, the following code snippet can be used on a NodeJS environment using the archiver package: |
| 47 | + |
| 48 | +```js |
| 49 | +const admin = require('firebase-admin'); |
| 50 | +const archiver = require('archiver'); |
| 51 | +const fs = require('fs'); |
| 52 | + |
| 53 | +function zipDirectory(storagePath, bucketName, outPath) { |
| 54 | + const archive = archiver('zip', { zlib: { level: 9 } }); |
| 55 | + const stream = fs.createWriteStream(outPath); |
| 56 | + archive.pipe(stream); |
| 57 | + |
| 58 | + return new Promise((resolve, reject) => { |
| 59 | + admin |
| 60 | + .storage(bucketName) |
| 61 | + .getFiles({ |
| 62 | + prefix: storagePath, |
| 63 | + }) |
| 64 | + .then(files => { |
| 65 | + files.forEach(file => { |
| 66 | + archive.append(file.createReadStream(), { name: file.name }); |
| 67 | + }); |
| 68 | + archive.finalize(); |
| 69 | + }); |
| 70 | + |
| 71 | + stream.on('close', () => resolve(false)); |
| 72 | + }); |
| 73 | +} |
| 74 | +``` |
| 75 | + |
| 76 | +## Firestore Security rules |
| 77 | + |
| 78 | +To enable your clients to read the export document, add the following security rules to match the user’s UID with the document uid: |
| 79 | + |
| 80 | +``` |
| 81 | +rules_version = '2'; |
| 82 | +service cloud.firestore { |
| 83 | + match /databases/{database}/documents { |
| 84 | + match /exports/{exportId} { |
| 85 | + allow read: if request.auth != null && request.auth.uid == resource.data.uid; |
| 86 | + } |
| 87 | + } |
| 88 | +} |
| 89 | +``` |
| 90 | + |
| 91 | +If you initialized Firestore in production mode, your rules will be configured by default such that no clients can read the export document. If you created Firestore in test mode, your rules will be configured by default such that all clients can read the export document. |
| 92 | + |
| 93 | +## Storage Security rules |
| 94 | + |
| 95 | +By default, security rules will prevent any client reads to exported files. To allow such reads, security rules can be set up so that access is only granted when the user’s UID matches the UID field of the export record document, for example: |
| 96 | + |
| 97 | +``` |
| 98 | +rules_version = '2'; |
| 99 | +service firebase.storage { |
| 100 | +match /b/{bucket}/o { |
| 101 | +match /${param:FIRESTORE_EXPORTS_COLLECTION}/{exportId}/{allPaths=**} { |
| 102 | + allow read: if firestore.exists(/databases/(default)/documents/${param:FIRESTORE_EXPORTS_COLLECTION}/$(exportId)) |
| 103 | + && (request.auth != null && firestore.get(/databases/(default)/documents/${param:FIRESTORE_EXPORTS_COLLECTION}/$(exportId)).data.uid == request.auth.uid) |
| 104 | +} |
| 105 | +} |
| 106 | +``` |
| 107 | + |
| 108 | +## Client SDK Usage |
| 109 | + |
| 110 | +The extension requires an authenticated user to call the `exportUserData` callable function. The function returns a unique export ID which can be used to subscribe to a Firestore document within the `${param:FIRESTORE_EXPORTS_COLLECTION}` collection. The document contains the real time status of an export, containing a `status` property. See the reference API for full details. |
| 111 | + |
| 112 | +Note: The following code requires that security rules have been added to the project. |
| 113 | + |
| 114 | +The following code snippet could be used to trigger an export and list out all files that have been exported once complete: |
| 115 | + |
| 116 | +```js |
| 117 | +import { getFirestore(), doc, onSnapshot } from 'firebase/firestore'; |
| 118 | +import { getStorage, listAll, ref } from 'firebase/storage'; |
| 119 | + |
| 120 | + |
| 121 | +// Trigger the export. |
| 122 | +const result = await httpsCallable(functions, `ext-${param:EXT_INSTANCE_ID}-exportUserData`)(); |
| 123 | + |
| 124 | +// Get the returned export id. |
| 125 | +const exportId = result.data.exportId; |
| 126 | + |
| 127 | + // Make a Firestore reference to the export. |
| 128 | + const documentRef = doc(getFirestore(), '${param:FIRESTORE_EXPORTS_COLLECTION}', exportId); |
| 129 | + |
| 130 | +// Listen for changes to the export - when complete returned the storage path of the export items. |
| 131 | +const { storagePath, zipPath } = await new Promise<string>((resolve, reject) => { |
| 132 | + const unsubscribe = onSnapshot(documentRef, snapshot => { |
| 133 | + if (!snapshot.exists) { |
| 134 | + unsubscribe(); |
| 135 | + return reject(new Error("Export document not found")); |
| 136 | + } |
| 137 | + |
| 138 | + const data = snapshot.data()!; |
| 139 | + |
| 140 | + if (data.status === 'complete') { |
| 141 | + unsubscribe(); |
| 142 | + return resolve({ |
| 143 | + storagePath: data.storagePath, |
| 144 | + zipPath: data.zipPath, |
| 145 | + }); |
| 146 | + } |
| 147 | + }); |
| 148 | +}); |
| 149 | + |
| 150 | +// Get a list of all the files in the export. |
| 151 | +const listResult = await listAll(ref(getStorage(), storagePath)); |
| 152 | + |
| 153 | +// Log out all exported file paths |
| 154 | +listResults.forEach((result) => { |
| 155 | + console.log(result.fullPath); |
| 156 | +}); |
| 157 | +``` |
| 158 | + |
| 159 | +To generate download links for the client, use the `getDownloadURL` function using a export files path: |
| 160 | + |
| 161 | +```js |
| 162 | +import { listAll, ref, getDownloadURL } from 'firebase/storage'; |
| 163 | + |
| 164 | +const url = await getDownloadURL(ref(getStorage(), fullPath)); |
| 165 | +``` |
| 166 | + |
| 167 | +## Sending an export email |
| 168 | + |
| 169 | +To send an email to your users once an export has completed with the export, you can deploy a [Custom Event Trigger](https://firebase.google.com/docs/functions/beta/custom-events). Listen to the completed export event, and send an email with the export as an attachment, for example using [nodemailer](https://nodemailer.com/). |
| 170 | + |
| 171 | +Monitoring |
| 172 | +As a best practice, you can [monitor the activity](https://firebase.google.com/docs/extensions/manage-installed-extensions#monitor) of your installed extension, including checks on its health, usage, and logs. |
0 commit comments