Fdc brownfield setup (#8150)

tammam-g · rosalyntan · maneesht · web-flow · commit 1d450356ead8 · 2025-03-11T01:17:28.000Z
* Helper functions and basic setup for dataconnect:sql:setup * Refactor setupIamUsers for better composability * FDC MVP brownfield and greenfield to brownfield schema setup * Add required logic inside schemaMigration for handling brownfield * Cleanup and fix bugs in brownfield setup * Use firebasesuperuser instead of cloudsqlsuper user for brownfield migration success * Add default permissions for brownfield * Fix lint/format * Refactor to allow setup reruns * Fix small things and address comments * Fix bug in role grants * Add logging that database setup completed * Make grant command not go through setup if roles can be granted in brownfield * bug fix from changing the getting schema owner command * Simplify getSchemaMetaData in permissions.ts * Fix log statement * Split permissions.ts into front facing permissions_setup.ts and keep backend permissions there * No need to ask user if they want to rerun greenfield setup * Make setupSQLPermissions return a setup status instead of a boolean * Change an if statment to switch statement * Keep upserting new user in grant command * Bump FDC local toolkit to v1.8.0. (#8210) * Bump FDC local toolkit to v1.8.0. * Update changelog. * First pass at auto generating sdk configs (#7833) * First pass at auto generating sdk configs * Fixed formatting issues * Removed extra command * Deleted unnecessary files * Fixed more linting' * Removed test assertion * Fixed formatting * Updated erros * Misc * Updated platforms list * Undid last changes * Addressed comments * Fixed client test * Driveby type fixing * missed a spot * Fixed test * Fix issue where if a user passes in an empty 'out' parameter, the CLI crashes * Added intelligent sensing where app should be * Fixed formatting * Fixed lint * Fixed app dir * Misc * Wrote tests * Reverted apps sdkconfig changes * Fixed formatting * Small changes * Revert shrinkwrap changes * Updated test:management script * Fixed apps-sdkconfig boolean check * Fixed more boolean * Fixed formatting * Added changelog * Added new options * Removed unused var * Added experimental flag * Moved apps:init behind a flag * Added apps:init command * Removed unnecessary experiments * Addressed comments --------- Co-authored-by: Joe Hanley <joehanley@google.com> * 13.31.0 * [firebase-release] Removed change log and reset repo after 13.31.0 release * FDC Emulator Update v1.8.1(#8216) * 13.31.1 * [firebase-release] Removed change log and reset repo after 13.31.1 release * Update formatting of connector evolution and insecure operation issues. (#8204) * Format INTERACTIVE_ACK issues as table as well and add extra "type" column to table. * Update warning and prompt wording to reflect insecure operations as well as connector evolution issues. * Wording. * Sort issues in table by category and some formatting fixes. * Use correct import path for data connect emulator (#8220) * Use correct import path for data connect emulator * Actually fix it this time * fix the thing i broke and format * Update src/emulator/dataconnect/pgliteServer.ts Co-authored-by: Maneesh Tewani <maneesht@users.noreply.github.com> --------- Co-authored-by: Maneesh Tewani <maneesht@users.noreply.github.com> * Don't surface insecure operations errors in VSCode. (#8215) * Don't surface connector evolution or insecure operation issues in VSCode.' * Fix * Filter by "INSECURE" substring rather than warningLevel. * Add path information to formatted GraphqlError. (#8228) * App Hosting Emulator bug - apphosting emulator info is not complete when env vars for emulators are set (#8231) * fix bug where apphosting emulator info is not complete when env vars for other emulators are set * add proper fix and test * fix * remove async from non-async test func * address comments * Bump FDC local toolkit to v1.8.2. (#8232) * Bump FDC local toolkit to v1.8.2. * Update changelog. * 13.31.2 * [firebase-release] Removed change log and reset repo after 13.31.2 release * fix: #8168 - enforce webframeworks only when needed (#8169) * fix: #8168 - enforce webframeworks only when needed In deployments where `--only hosting:boo` is used, enforce webframeworks enablement only when the target actually uses webframeworks. * remove console * add changelog, add tests * Add matchesHostingTarget to improve readability --------- Co-authored-by: Chalo Salvador <chalo@monogram.io> * Added env var to magically import data connect service from console (#8237) * Added env var to magically import data connect service from console * actually check location too * formats * Formats * Add initial delay when loading python functions (#8239) * Improve robustness of function discovery for python Anecdotally, python function discovery is flakey. We propose 2 change in this PR: 1. For python discovery, add a small initial delay for python's admin server to boot. 2. Add a request timeout to retry call to retrieve trigger information. Previously, the default timeout would've been set to OS-level TCP timeout, which in my laptop was between 20~30s. * Add changelog. * Remove per-req timeout to accomodate loading large/slow main.py. * Update changelog. * Revert timeout bump. * Update vscode to 0.13.1 (#8236) * update vscode to 0.13.1 * remove changelog line * Propagate overrides (#8253) * Apply ajv and ajv-formats overrides to shrinkwrap * Apply whatwg-url override to shrinkwrap * npm i to stabilize shrinkwrap --------- Co-authored-by: Joe Hanley <joehanley@google.com> * Print warning about --location removal from apphosting commands. (#8229) `--location` will be removed from apphosting commands in the next major CLI release. Before then, the CLI will print a warning about this removal whenever `--location` is used. * Fix issue where apps:init breaks on app creation (#8258) * Rename MetaData to Metadata * Change setup to set up in firebase error * Improve logger message * Fix bugs in brownfield setup status checks * fix lint issues --------- Co-authored-by: Rosalyn Tan <rosalyntan@google.com> Co-authored-by: Maneesh Tewani <maneesht@users.noreply.github.com> Co-authored-by: Joe Hanley <joehanley@google.com> Co-authored-by: Google Open Source Bot <firebase-oss-bot@google.com> Co-authored-by: Mathusan Selvarajah <mathusans52@gmail.com> Co-authored-by: Philip Su <39933441+fivecar@users.noreply.github.com> Co-authored-by: Chalo Salvador <chalo@monogram.io> Co-authored-by: Daniel Lee <danielylee@google.com> Co-authored-by: Harold Shen <hlshen@google.com> Co-authored-by: Sarah Clark <seclark@nextquestion.net> Co-authored-by: annajowang <31288696+annajowang@users.noreply.github.com>
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1 +1,2 @@
 - Fix webframeworks deployments when using `site` in `firebase.json`. (#8295)
+- Add support for brownfield project onboard `dataconnect:sql:setup` (#8150)
diff --git a/src/commands/dataconnect-sql-grant.ts b/src/commands/dataconnect-sql-grant.ts
@@ -7,7 +7,7 @@ import { pickService } from "../dataconnect/fileUtils";
 import { grantRoleToUserInSchema } from "../dataconnect/schemaMigration";
 import { requireAuth } from "../requireAuth";
 import { FirebaseError } from "../error";
-import { fdcSqlRoleMap } from "../gcp/cloudsql/permissions";
+import { fdcSqlRoleMap } from "../gcp/cloudsql/permissions_setup";
 
 const allowedRoles = Object.keys(fdcSqlRoleMap);
 
diff --git a/src/commands/dataconnect-sql-setup.ts b/src/commands/dataconnect-sql-setup.ts
@@ -0,0 +1,38 @@
+import { Command } from "../command";
+import { Options } from "../options";
+import { needProjectId } from "../projectUtils";
+import { pickService } from "../dataconnect/fileUtils";
+import { FirebaseError } from "../error";
+import { requireAuth } from "../requireAuth";
+import { requirePermissions } from "../requirePermissions";
+import { ensureApis } from "../dataconnect/ensureApis";
+import { setupSQLPermissions, getSchemaMetadata } from "../gcp/cloudsql/permissions_setup";
+import { DEFAULT_SCHEMA } from "../gcp/cloudsql/permissions";
+import { getIdentifiers } from "../dataconnect/schemaMigration";
+
+export const command = new Command("dataconnect:sql:setup [serviceId]")
+  .description("Setup your CloudSQL database")
+  .before(requirePermissions, [
+    "firebasedataconnect.services.list",
+    "firebasedataconnect.schemas.list",
+    "firebasedataconnect.schemas.update",
+    "cloudsql.instances.connect",
+  ])
+  .before(requireAuth)
+  .action(async (serviceId: string, options: Options) => {
+    const projectId = needProjectId(options);
+    await ensureApis(projectId);
+    const serviceInfo = await pickService(projectId, options.config, serviceId);
+    const instanceId =
+      serviceInfo.dataConnectYaml.schema.datasource.postgresql?.cloudSql.instanceId;
+    if (!instanceId) {
+      throw new FirebaseError(
+        "dataconnect.yaml is missing field schema.datasource.postgresql.cloudsql.instanceId",
+      );
+    }
+
+    const { databaseId } = getIdentifiers(serviceInfo.schema);
+
+    const schemaInfo = await getSchemaMetadata(instanceId, databaseId, DEFAULT_SCHEMA, options);
+    await setupSQLPermissions(instanceId, databaseId, schemaInfo, options);
+  });
diff --git a/src/commands/index.ts b/src/commands/index.ts
@@ -221,6 +221,7 @@ export function load(client: any): any {
   client.dataconnect.services.list = loadCommand("dataconnect-services-list");
   client.dataconnect.sql = {};
   client.dataconnect.sql.diff = loadCommand("dataconnect-sql-diff");
+  client.dataconnect.sql.setup = loadCommand("dataconnect-sql-setup");
   client.dataconnect.sql.migrate = loadCommand("dataconnect-sql-migrate");
   client.dataconnect.sql.grant = loadCommand("dataconnect-sql-grant");
   client.dataconnect.sql.shell = loadCommand("dataconnect-sql-shell");
diff --git a/src/dataconnect/schemaMigration.ts b/src/dataconnect/schemaMigration.ts
@@ -4,26 +4,29 @@ import { format } from "sql-formatter";
 import { IncompatibleSqlSchemaError, Diff, SCHEMA_ID, SchemaValidation } from "./types";
 import { getSchema, upsertSchema, deleteConnector } from "./client";
 import {
-  setupIAMUsers,
   getIAMUser,
   executeSqlCmdsAsIamUser,
   executeSqlCmdsAsSuperUser,
   toDatabaseUser,
 } from "../gcp/cloudsql/connect";
+import { needProjectId } from "../projectUtils";
 import {
-  firebaseowner,
-  iamUserIsCSQLAdmin,
   checkSQLRoleIsGranted,
   fdcSqlRoleMap,
-} from "../gcp/cloudsql/permissions";
-import * as cloudSqlAdminClient from "../gcp/cloudsql/cloudsqladmin";
-import { needProjectId } from "../projectUtils";
+  setupSQLPermissions,
+  getSchemaMetadata,
+  SchemaSetupStatus,
+} from "../gcp/cloudsql/permissions_setup";
+import { DEFAULT_SCHEMA, firebaseowner } from "../gcp/cloudsql/permissions";
 import { promptOnce, confirm } from "../prompt";
 import { logger } from "../logger";
 import { Schema } from "./types";
 import { Options } from "../options";
 import { FirebaseError } from "../error";
 import { logLabeledBullet, logLabeledWarning, logLabeledSuccess } from "../utils";
+import { iamUserIsCSQLAdmin } from "../gcp/cloudsql/cloudsqladmin";
+import * as cloudSqlAdminClient from "../gcp/cloudsql/cloudsqladmin";
+
 import * as errors from "./errors";
 
 export async function diffSchema(
@@ -236,10 +239,34 @@ export async function grantRoleToUserInSchema(options: Options, schema: Schema)
     );
   }
 
-  // Run the database roles setup. This should be idempotent.
-  await setupIAMUsers(instanceId, databaseId, options);
+  // Make sure we have the right setup for the requested role grant.
+  const schemaInfo = await getSchemaMetadata(instanceId, databaseId, DEFAULT_SCHEMA, options);
+  let isGreenfieldSetup = schemaInfo.setupStatus === SchemaSetupStatus.GreenField;
+  switch (schemaInfo.setupStatus) {
+    case SchemaSetupStatus.NotSetup:
+    case SchemaSetupStatus.NotFound:
+      const newSetupStatus = await setupSQLPermissions(instanceId, databaseId, schemaInfo, options);
+      isGreenfieldSetup = newSetupStatus === SchemaSetupStatus.GreenField;
+      break;
+    default:
+      logger.info(
+        `Detected schema "${schemaInfo.name}" is setup in ${schemaInfo.setupStatus} mode. Skipping Setup.`,
+      );
+      break;
+  }
+
+  // Edge case: we can't grant firebase owner unless database is greenfield.
+  if (!isGreenfieldSetup && fdcSqlRole === firebaseowner(databaseId, DEFAULT_SCHEMA)) {
+    const newSetupStatus = await setupSQLPermissions(instanceId, databaseId, schemaInfo, options);
 
-  // Upsert user account into the database.
+    if (newSetupStatus !== SchemaSetupStatus.GreenField) {
+      throw new FirebaseError(
+        `Can't grant owner rule for brownfield databases. Consider fully migrating your database to FDC using 'firebase dataconnect:sql:setup'`,
+      );
+    }
+  }
+
+  // Upsert new user account into the database.
   await cloudSqlAdminClient.createUser(projectId, instanceId, mode, user);
 
   // Grant the role to the user.
@@ -351,10 +378,21 @@ async function handleIncompatibleSchemaError(args: {
         ${commandsToExecuteBySuperUser.join("\n")}`);
     }
 
-    // TODO (tammam-g): at some point we would want to only run this after notifying the admin but
-    // until we confirm stability it's ok to run it on every migration by admin user.
-    if (userIsCSQLAdmin) {
-      await setupIAMUsers(instanceId, databaseId, options);
+    const schemaInfo = await getSchemaMetadata(instanceId, databaseId, DEFAULT_SCHEMA, options);
+    if (schemaInfo.setupStatus !== SchemaSetupStatus.GreenField) {
+      const newSetupStatus = await setupSQLPermissions(
+        instanceId,
+        databaseId,
+        schemaInfo,
+        options,
+        /* silent=*/ true,
+      );
+
+      if (newSetupStatus !== SchemaSetupStatus.GreenField) {
+        throw new FirebaseError(
+          `Can't migrate brownfield databases. Consider fully migrating your database to FDC using 'firebase dataconnect:sql:setup'`,
+        );
+      }
     }
 
     // Test if iam user has access to the roles required for this migration
diff --git a/src/gcp/cloudsql/cloudsqladmin.ts b/src/gcp/cloudsql/cloudsqladmin.ts
@@ -2,6 +2,10 @@ import { Client, ClientResponse } from "../../apiv2";
 import { cloudSQLAdminOrigin } from "../../api";
 import * as operationPoller from "../../operation-poller";
 import { Instance, Database, User, UserType, DatabaseFlag } from "./types";
+import { needProjectId } from "../../projectUtils";
+import { Options } from "../../options";
+import { logger } from "../../logger";
+import { testIamPermissions } from "../iam";
 import { FirebaseError } from "../../error";
 const API_VERSION = "v1";
 
@@ -16,6 +20,24 @@ interface Operation {
   name: string;
 }
 
+export async function iamUserIsCSQLAdmin(options: Options): Promise<boolean> {
+  const projectId = needProjectId(options);
+  const requiredPermissions = [
+    "cloudsql.instances.connect",
+    "cloudsql.instances.get",
+    "cloudsql.users.create",
+    "cloudsql.users.update",
+  ];
+
+  try {
+    const iamResult = await testIamPermissions(projectId, requiredPermissions);
+    return iamResult.passed;
+  } catch (err: any) {
+    logger.debug(`[iam] error while checking permissions, command may fail: ${err}`);
+    return false;
+  }
+}
+
 export async function listInstances(projectId: string): Promise<Instance[]> {
   const res = await client.get<{ items: Instance[] }>(`projects/${projectId}/instances`);
   return res.body.items ?? [];
diff --git a/src/gcp/cloudsql/connect.ts b/src/gcp/cloudsql/connect.ts
@@ -11,7 +11,6 @@ import { logger } from "../../logger";
 import { FirebaseError } from "../../error";
 import { Options } from "../../options";
 import { FBToolsAuthClient } from "./fbToolsAuthClient";
-import { setupSQLPermissions, firebaseowner, firebasewriter } from "./permissions";
 
 export async function execute(
   sqlStatements: string[],
@@ -23,7 +22,7 @@ export async function execute(
     password?: string;
     silent?: boolean;
   },
-) {
+): Promise<pg.QueryResult[]> {
   const logFn = opts.silent ? logger.debug : logger.info;
   const instance = await cloudSqlAdminClient.getInstance(opts.projectId, opts.instanceId);
   const user = await cloudSqlAdminClient.getUser(opts.projectId, opts.instanceId, opts.username);
@@ -92,11 +91,12 @@ export async function execute(
   }
 
   const conn = await pool.connect();
+  const results: pg.QueryResult[] = [];
   logFn(`Logged in as ${opts.username}`);
   for (const s of sqlStatements) {
     logFn(`Executing: '${s}'`);
     try {
-      await conn.query(s);
+      results.push(await conn.query(s));
     } catch (err) {
       throw new FirebaseError(`Error executing ${err}`);
     }
@@ -105,6 +105,7 @@ export async function execute(
   conn.release();
   await pool.end();
   connector.close();
+  return results;
 }
 
 export async function executeSqlCmdsAsIamUser(
@@ -113,7 +114,7 @@ export async function executeSqlCmdsAsIamUser(
   databaseId: string,
   cmds: string[],
   silent = false,
-): Promise<void> {
+): Promise<pg.QueryResult[]> {
   const projectId = needProjectId(options);
   const { user: iamUser } = await getIAMUser(options);
 
@@ -135,7 +136,7 @@ export async function executeSqlCmdsAsSuperUser(
   databaseId: string,
   cmds: string[],
   silent = false,
-) {
+): Promise<pg.QueryResult[]> {
   const projectId = needProjectId(options);
   // 1. Create a temporary builtin user
   const superuser = "firebasesuperuser";
@@ -148,7 +149,7 @@ export async function executeSqlCmdsAsSuperUser(
     temporaryPassword,
   );
 
-  return await execute([`SET ROLE = cloudsqlsuperuser`, ...cmds], {
+  return await execute([`SET ROLE = '${superuser}'`, ...cmds], {
     projectId,
     instanceId,
     databaseId,
@@ -177,8 +178,6 @@ export async function getIAMUser(options: Options): Promise<{ user: string; mode
 // Steps:
 // 1. Create an IAM user for the current identity
 // 2. Create an IAM user for FDC P4SA
-// 3. Run setupSQLPermissions to setup the SQL database roles and permissions.
-// 4. Connect to the DB as the temporary user and run the necessary grants
 export async function setupIAMUsers(
   instanceId: string,
   databaseId: string,
@@ -200,18 +199,6 @@ export async function setupIAMUsers(
   );
   await cloudSqlAdminClient.createUser(projectId, instanceId, fdcP4SAmode, fdcP4SAUser);
 
-  // 3. Setup FDC required SQL roles and permissions.
-  await setupSQLPermissions(instanceId, databaseId, options, true);
-
-  // 4. Apply necessary grants.
-  const grants = [
-    // Grant firebaseowner role to the current IAM user.
-    `GRANT "${firebaseowner(databaseId)}" TO "${user}"`,
-    // Grant firebaswriter to the FDC P4SA user
-    `GRANT "${firebasewriter(databaseId)}" TO "${fdcP4SAUser}"`,
-  ];
-
-  await executeSqlCmdsAsSuperUser(options, instanceId, databaseId, grants, /** silent=*/ true);
   return user;
 }
 
diff --git a/src/gcp/cloudsql/permissions.ts b/src/gcp/cloudsql/permissions.ts
diff --git a/src/gcp/cloudsql/permissions_setup.ts b/src/gcp/cloudsql/permissions_setup.ts

Original file line number	Diff line number	Diff line change
`@@ -1 +1,2 @@`
`1`	`1`	- Fix webframeworks deployments when using `site` in `firebase.json`. (#8295)
	`2`	+- Add support for brownfield project onboard `dataconnect:sql:setup` (#8150)