Fdc brownfield setup (#8150)

* Helper functions and basic setup for dataconnect:sql:setup

* Refactor setupIamUsers for better composability

* FDC MVP brownfield and greenfield to brownfield schema setup

* Add required logic inside schemaMigration for handling brownfield

* Cleanup and fix bugs in brownfield setup

* Use firebasesuperuser instead of cloudsqlsuper user for brownfield migration success

* Add default permissions for brownfield

* Fix lint/format

* Refactor to allow setup reruns

* Fix small things and address comments

* Fix bug in role grants

* Add logging that database setup completed

* Make grant command not go through setup if roles can be granted in brownfield

* bug fix from changing the getting schema owner command

* Simplify getSchemaMetaData in permissions.ts

* Fix log statement

* Split permissions.ts into front facing permissions_setup.ts and keep backend permissions there

* No need to ask user if they want to rerun greenfield setup

* Make setupSQLPermissions return a setup status instead of a boolean

* Change an if statment to switch statement

* Keep upserting new user in grant command

* Bump FDC local toolkit to v1.8.0. (#8210)

* Bump FDC local toolkit to v1.8.0.

* Update changelog.

* First pass at auto generating sdk configs (#7833)

* First pass at auto generating sdk configs

* Fixed formatting issues

* Removed extra command

* Deleted unnecessary files

* Fixed more linting'

* Removed test assertion

* Fixed formatting

* Updated erros

* Misc

* Updated platforms list

* Undid last changes

* Addressed comments

* Fixed client test

* Driveby type fixing

* missed a spot

* Fixed test

* Fix issue where if a user passes in an empty 'out' parameter, the CLI crashes

* Added intelligent sensing where app should be

* Fixed formatting

* Fixed lint

* Fixed app dir

* Misc

* Wrote tests

* Reverted apps sdkconfig changes

* Fixed formatting

* Small changes

* Revert shrinkwrap changes

* Updated test:management script

* Fixed apps-sdkconfig boolean check

* Fixed more boolean

* Fixed formatting

* Added changelog

* Added new options

* Removed unused var

* Added experimental flag

* Moved apps:init behind a flag

* Added apps:init command

* Removed unnecessary experiments

* Addressed comments

---------

Co-authored-by: Joe Hanley <[email protected]>

* 13.31.0

* [firebase-release] Removed change log and reset repo after 13.31.0 release

* FDC Emulator Update v1.8.1(#8216)

* 13.31.1

* [firebase-release] Removed change log and reset repo after 13.31.1 release

* Update formatting of connector evolution and insecure operation issues. (#8204)

* Format INTERACTIVE_ACK issues as table as well and add extra "type" column to table.

* Update warning and prompt wording to reflect insecure operations as well as connector evolution issues.

* Wording.

* Sort issues in table by category and some formatting fixes.

* Use correct import path for data connect emulator (#8220)

* Use correct import path for data connect emulator

* Actually fix it this time

* fix the thing i broke and format

* Update src/emulator/dataconnect/pgliteServer.ts

Co-authored-by: Maneesh Tewani <[email protected]>

---------

Co-authored-by: Maneesh Tewani <[email protected]>

* Don't surface insecure operations errors in VSCode. (#8215)

* Don't surface connector evolution or insecure operation issues in VSCode.'

* Fix

* Filter by "INSECURE" substring rather than warningLevel.

* Add path information to formatted GraphqlError. (#8228)

* App Hosting Emulator bug - apphosting emulator info is not complete when env vars for emulators are set (#8231)

* fix bug where apphosting emulator info is not complete when env vars for other emulators are set

* add proper fix and test

* fix

* remove async from non-async test func

* address comments

* Bump FDC local toolkit to v1.8.2. (#8232)

* Bump FDC local toolkit to v1.8.2.

* Update changelog.

* 13.31.2

* [firebase-release] Removed change log and reset repo after 13.31.2 release

* fix: #8168 - enforce webframeworks only when needed (#8169)

* fix: #8168 - enforce webframeworks only when needed

In deployments where `--only hosting:boo` is used, enforce webframeworks
enablement only when the target actually uses webframeworks.

* remove console

* add changelog, add tests

* Add matchesHostingTarget to improve readability

---------

Co-authored-by: Chalo Salvador <[email protected]>

* Added env var to magically import data connect service from console (#8237)

* Added env var to magically import data connect service from console

* actually check location too

* formats

* Formats

* Add initial delay when loading python functions (#8239)

* Improve robustness of function discovery for python

Anecdotally, python function discovery is flakey. We propose 2 change in
this PR:

1. For python discovery, add a small initial delay for python's admin
server to boot.

2. Add a request timeout to retry call to retrieve trigger information.
Previously, the default timeout would've been set to OS-level TCP
timeout, which in my laptop was between 20~30s.

* Add changelog.

* Remove per-req timeout to accomodate loading large/slow main.py.

* Update changelog.

* Revert timeout bump.

* Update vscode to 0.13.1 (#8236)

* update vscode to 0.13.1

* remove changelog line

* Propagate overrides (#8253)

* Apply ajv and ajv-formats overrides to shrinkwrap

* Apply whatwg-url override to shrinkwrap

* npm i to stabilize shrinkwrap

---------

Co-authored-by: Joe Hanley <[email protected]>

* Print warning about --location removal from apphosting commands. (#8229)

`--location` will be removed from apphosting commands in the next major CLI release. Before then, the CLI will print a warning about this removal whenever `--location` is used.

* Fix issue where apps:init breaks on app creation (#8258)

* Rename MetaData to Metadata

* Change setup to set up in firebase error

* Improve logger message

* Fix bugs in brownfield setup status checks

* fix lint issues

---------

Co-authored-by: Rosalyn Tan <[email protected]>
Co-authored-by: Maneesh Tewani <[email protected]>
Co-authored-by: Joe Hanley <[email protected]>
Co-authored-by: Google Open Source Bot <[email protected]>
Co-authored-by: Mathusan Selvarajah <[email protected]>
Co-authored-by: Philip Su <[email protected]>
Co-authored-by: Chalo Salvador <[email protected]>
Co-authored-by: Daniel Lee <[email protected]>
Co-authored-by: Harold Shen <[email protected]>
Co-authored-by: Sarah Clark <[email protected]>
Co-authored-by: annajowang <[email protected]>

Author: 12 people

CHANGELOG.md

@@ -1 +1,2 @@
 - Fix webframeworks deployments when using `site` in `firebase.json`. (#8295)
+- Add support for brownfield project onboard `dataconnect:sql:setup` (#8150)

src/commands/dataconnect-sql-grant.ts

@@ -7,7 +7,7 @@ import { pickService } from "../dataconnect/fileUtils";
 import { grantRoleToUserInSchema } from "../dataconnect/schemaMigration";
 import { requireAuth } from "../requireAuth";
 import { FirebaseError } from "../error";
-import { fdcSqlRoleMap } from "../gcp/cloudsql/permissions";
+import { fdcSqlRoleMap } from "../gcp/cloudsql/permissions_setup";
 
 const allowedRoles = Object.keys(fdcSqlRoleMap);
 

src/commands/dataconnect-sql-setup.ts

@@ -0,0 +1,38 @@
+import { Command } from "../command";
+import { Options } from "../options";
+import { needProjectId } from "../projectUtils";
+import { pickService } from "../dataconnect/fileUtils";
+import { FirebaseError } from "../error";
+import { requireAuth } from "../requireAuth";
+import { requirePermissions } from "../requirePermissions";
+import { ensureApis } from "../dataconnect/ensureApis";
+import { setupSQLPermissions, getSchemaMetadata } from "../gcp/cloudsql/permissions_setup";
+import { DEFAULT_SCHEMA } from "../gcp/cloudsql/permissions";
+import { getIdentifiers } from "../dataconnect/schemaMigration";
+
+export const command = new Command("dataconnect:sql:setup [serviceId]")
+  .description("Setup your CloudSQL database")
+  .before(requirePermissions, [
+    "firebasedataconnect.services.list",
+    "firebasedataconnect.schemas.list",
+    "firebasedataconnect.schemas.update",
+    "cloudsql.instances.connect",
+  ])
+  .before(requireAuth)
+  .action(async (serviceId: string, options: Options) => {
+    const projectId = needProjectId(options);
+    await ensureApis(projectId);
+    const serviceInfo = await pickService(projectId, options.config, serviceId);
+    const instanceId =
+      serviceInfo.dataConnectYaml.schema.datasource.postgresql?.cloudSql.instanceId;
+    if (!instanceId) {
+      throw new FirebaseError(
+        "dataconnect.yaml is missing field schema.datasource.postgresql.cloudsql.instanceId",
+      );
+    }
+
+    const { databaseId } = getIdentifiers(serviceInfo.schema);
+
+    const schemaInfo = await getSchemaMetadata(instanceId, databaseId, DEFAULT_SCHEMA, options);
+    await setupSQLPermissions(instanceId, databaseId, schemaInfo, options);
+  });

src/commands/index.ts

@@ -221,6 +221,7 @@ export function load(client: any): any {
   client.dataconnect.services.list = loadCommand("dataconnect-services-list");
   client.dataconnect.sql = {};
   client.dataconnect.sql.diff = loadCommand("dataconnect-sql-diff");
+  client.dataconnect.sql.setup = loadCommand("dataconnect-sql-setup");
   client.dataconnect.sql.migrate = loadCommand("dataconnect-sql-migrate");
   client.dataconnect.sql.grant = loadCommand("dataconnect-sql-grant");
   client.dataconnect.sql.shell = loadCommand("dataconnect-sql-shell");

src/dataconnect/schemaMigration.ts

@@ -4,26 +4,29 @@ import { format } from "sql-formatter";
 import { IncompatibleSqlSchemaError, Diff, SCHEMA_ID, SchemaValidation } from "./types";
 import { getSchema, upsertSchema, deleteConnector } from "./client";
 import {
-  setupIAMUsers,
   getIAMUser,
   executeSqlCmdsAsIamUser,
   executeSqlCmdsAsSuperUser,
   toDatabaseUser,
 } from "../gcp/cloudsql/connect";
+import { needProjectId } from "../projectUtils";
 import {
-  firebaseowner,
-  iamUserIsCSQLAdmin,
   checkSQLRoleIsGranted,
   fdcSqlRoleMap,
-} from "../gcp/cloudsql/permissions";
-import * as cloudSqlAdminClient from "../gcp/cloudsql/cloudsqladmin";
-import { needProjectId } from "../projectUtils";
+  setupSQLPermissions,
+  getSchemaMetadata,
+  SchemaSetupStatus,
+} from "../gcp/cloudsql/permissions_setup";
+import { DEFAULT_SCHEMA, firebaseowner } from "../gcp/cloudsql/permissions";
 import { promptOnce, confirm } from "../prompt";
 import { logger } from "../logger";
 import { Schema } from "./types";
 import { Options } from "../options";
 import { FirebaseError } from "../error";
 import { logLabeledBullet, logLabeledWarning, logLabeledSuccess } from "../utils";
+import { iamUserIsCSQLAdmin } from "../gcp/cloudsql/cloudsqladmin";
+import * as cloudSqlAdminClient from "../gcp/cloudsql/cloudsqladmin";
+
 import * as errors from "./errors";
 
 export async function diffSchema(
@@ -236,10 +239,34 @@ export async function grantRoleToUserInSchema(options: Options, schema: Schema)
     );
   }
 
-  // Run the database roles setup. This should be idempotent.
-  await setupIAMUsers(instanceId, databaseId, options);
+  // Make sure we have the right setup for the requested role grant.
+  const schemaInfo = await getSchemaMetadata(instanceId, databaseId, DEFAULT_SCHEMA, options);
+  let isGreenfieldSetup = schemaInfo.setupStatus === SchemaSetupStatus.GreenField;
+  switch (schemaInfo.setupStatus) {
+    case SchemaSetupStatus.NotSetup:
+    case SchemaSetupStatus.NotFound:
+      const newSetupStatus = await setupSQLPermissions(instanceId, databaseId, schemaInfo, options);
+      isGreenfieldSetup = newSetupStatus === SchemaSetupStatus.GreenField;
+      break;
+    default:
+      logger.info(
+        `Detected schema "${schemaInfo.name}" is setup in ${schemaInfo.setupStatus} mode. Skipping Setup.`,
+      );
+      break;
+  }
+
+  // Edge case: we can't grant firebase owner unless database is greenfield.
+  if (!isGreenfieldSetup && fdcSqlRole === firebaseowner(databaseId, DEFAULT_SCHEMA)) {
+    const newSetupStatus = await setupSQLPermissions(instanceId, databaseId, schemaInfo, options);
 
-  // Upsert user account into the database.
+    if (newSetupStatus !== SchemaSetupStatus.GreenField) {
+      throw new FirebaseError(
+        `Can't grant owner rule for brownfield databases. Consider fully migrating your database to FDC using 'firebase dataconnect:sql:setup'`,
+      );
+    }
+  }
+
+  // Upsert new user account into the database.
   await cloudSqlAdminClient.createUser(projectId, instanceId, mode, user);
 
   // Grant the role to the user.
@@ -351,10 +378,21 @@ async function handleIncompatibleSchemaError(args: {
         ${commandsToExecuteBySuperUser.join("\n")}`);
     }
 
-    // TODO (tammam-g): at some point we would want to only run this after notifying the admin but
-    // until we confirm stability it's ok to run it on every migration by admin user.
-    if (userIsCSQLAdmin) {
-      await setupIAMUsers(instanceId, databaseId, options);
+    const schemaInfo = await getSchemaMetadata(instanceId, databaseId, DEFAULT_SCHEMA, options);
+    if (schemaInfo.setupStatus !== SchemaSetupStatus.GreenField) {
+      const newSetupStatus = await setupSQLPermissions(
+        instanceId,
+        databaseId,
+        schemaInfo,
+        options,
+        /* silent=*/ true,
+      );
+
+      if (newSetupStatus !== SchemaSetupStatus.GreenField) {
+        throw new FirebaseError(
+          `Can't migrate brownfield databases. Consider fully migrating your database to FDC using 'firebase dataconnect:sql:setup'`,
+        );
+      }
     }
 
     // Test if iam user has access to the roles required for this migration

src/gcp/cloudsql/cloudsqladmin.ts

@@ -2,6 +2,10 @@ import { Client, ClientResponse } from "../../apiv2";
 import { cloudSQLAdminOrigin } from "../../api";
 import * as operationPoller from "../../operation-poller";
 import { Instance, Database, User, UserType, DatabaseFlag } from "./types";
+import { needProjectId } from "../../projectUtils";
+import { Options } from "../../options";
+import { logger } from "../../logger";
+import { testIamPermissions } from "../iam";
 import { FirebaseError } from "../../error";
 const API_VERSION = "v1";
 
@@ -16,6 +20,24 @@ interface Operation {
   name: string;
 }
 
+export async function iamUserIsCSQLAdmin(options: Options): Promise<boolean> {
+  const projectId = needProjectId(options);
+  const requiredPermissions = [
+    "cloudsql.instances.connect",
+    "cloudsql.instances.get",
+    "cloudsql.users.create",
+    "cloudsql.users.update",
+  ];
+
+  try {
+    const iamResult = await testIamPermissions(projectId, requiredPermissions);
+    return iamResult.passed;
+  } catch (err: any) {
+    logger.debug(`[iam] error while checking permissions, command may fail: ${err}`);
+    return false;
+  }
+}
+
 export async function listInstances(projectId: string): Promise<Instance[]> {
   const res = await client.get<{ items: Instance[] }>(`projects/${projectId}/instances`);
   return res.body.items ?? [];

src/gcp/cloudsql/connect.ts

@@ -11,7 +11,6 @@ import { logger } from "../../logger";
 import { FirebaseError } from "../../error";
 import { Options } from "../../options";
 import { FBToolsAuthClient } from "./fbToolsAuthClient";
-import { setupSQLPermissions, firebaseowner, firebasewriter } from "./permissions";
 
 export async function execute(
   sqlStatements: string[],
@@ -23,7 +22,7 @@ export async function execute(
     password?: string;
     silent?: boolean;
   },
-) {
+): Promise<pg.QueryResult[]> {
   const logFn = opts.silent ? logger.debug : logger.info;
   const instance = await cloudSqlAdminClient.getInstance(opts.projectId, opts.instanceId);
   const user = await cloudSqlAdminClient.getUser(opts.projectId, opts.instanceId, opts.username);
@@ -92,11 +91,12 @@ export async function execute(
   }
 
   const conn = await pool.connect();
+  const results: pg.QueryResult[] = [];
   logFn(`Logged in as ${opts.username}`);
   for (const s of sqlStatements) {
     logFn(`Executing: '${s}'`);
     try {
-      await conn.query(s);
+      results.push(await conn.query(s));
     } catch (err) {
       throw new FirebaseError(`Error executing ${err}`);
     }
@@ -105,6 +105,7 @@ export async function execute(
   conn.release();
   await pool.end();
   connector.close();
+  return results;
 }
 
 export async function executeSqlCmdsAsIamUser(
@@ -113,7 +114,7 @@ export async function executeSqlCmdsAsIamUser(
   databaseId: string,
   cmds: string[],
   silent = false,
-): Promise<void> {
+): Promise<pg.QueryResult[]> {
   const projectId = needProjectId(options);
   const { user: iamUser } = await getIAMUser(options);
 
@@ -135,7 +136,7 @@ export async function executeSqlCmdsAsSuperUser(
   databaseId: string,
   cmds: string[],
   silent = false,
-) {
+): Promise<pg.QueryResult[]> {
   const projectId = needProjectId(options);
   // 1. Create a temporary builtin user
   const superuser = "firebasesuperuser";
@@ -148,7 +149,7 @@ export async function executeSqlCmdsAsSuperUser(
     temporaryPassword,
   );
 
-  return await execute([`SET ROLE = cloudsqlsuperuser`, ...cmds], {
+  return await execute([`SET ROLE = '${superuser}'`, ...cmds], {
     projectId,
     instanceId,
     databaseId,
@@ -177,8 +178,6 @@ export async function getIAMUser(options: Options): Promise<{ user: string; mode
 // Steps:
 // 1. Create an IAM user for the current identity
 // 2. Create an IAM user for FDC P4SA
-// 3. Run setupSQLPermissions to setup the SQL database roles and permissions.
-// 4. Connect to the DB as the temporary user and run the necessary grants
 export async function setupIAMUsers(
   instanceId: string,
   databaseId: string,
@@ -200,18 +199,6 @@ export async function setupIAMUsers(
   );
   await cloudSqlAdminClient.createUser(projectId, instanceId, fdcP4SAmode, fdcP4SAUser);
 
-  // 3. Setup FDC required SQL roles and permissions.
-  await setupSQLPermissions(instanceId, databaseId, options, true);
-
-  // 4. Apply necessary grants.
-  const grants = [
-    // Grant firebaseowner role to the current IAM user.
-    `GRANT "${firebaseowner(databaseId)}" TO "${user}"`,
-    // Grant firebaswriter to the FDC P4SA user
-    `GRANT "${firebasewriter(databaseId)}" TO "${fdcP4SAUser}"`,
-  ];
-
-  await executeSqlCmdsAsSuperUser(options, instanceId, databaseId, grants, /** silent=*/ true);
   return user;
 }