DO-2075 Added fenix and desktop baseline city seen tables #7974

wwyc · 2025-08-19T23:01:37Z

Description

Initialize the *baseline_city_seen tables by deriving each client’s first-seen and last-seen city, subdivision and country fields from the stable tables.
Note: This one-time initialization logic will no longer apply once city/subdivision/country fields are nulled in the stable tables.
Ongoing updates: After initialization, the tables will be updated daily via ETL using live tables (appending new clients and advancing last-seen values).

Related Tickets & Documents

DO-2075

Reviewer, please follow this checklist

sql/moz-fx-data-shared-prod/fenix_derived/clients_city_seen_v1/schema.yaml

sql/moz-fx-data-shared-prod/fenix_derived/clients_city_seen_v1/metadata.yaml

sql/moz-fx-data-shared-prod/fenix_derived/clients_city_seen_v1/schema.yaml

sql/moz-fx-data-shared-prod/firefox_desktop_derived/clients_city_seen_v1/query.sql

sql_generators/clients_city_seen_v1/templates/metadata.yaml

sql/moz-fx-data-shared-prod/fenix_derived/clients_city_seen_v1/metadata.yaml

sql_generators/clients_city_seen_v1/templates/metadata.yaml

sql_generators/clients_city_seen_v1/__init__.py

sql_generators/clients_city_seen_v1/templates/query.sql

sql/moz-fx-data-shared-prod/fenix_derived/clients_city_seen_v1/schema.yaml

BenWu

You should remove all the generated files in sql/ before merging because generated sql doesn't need to be checked in.

The is_init query with a 100% sample might be too expensive run in a single query and timeout. This is fine for the POC but something to prepare for later

sql_generators/baseline_clients_city_seen_v1/templates/query.sql

sql_generators/baseline_clients_city_seen_v1/templates/metadata.yaml

sql_generators/baseline_clients_city_seen_v1/__init__.py

BenWu · 2025-09-25T19:23:00Z

sql_generators/baseline_clients_city_seen_v1/templates/query.sql

+    {% endraw %}
+  WHERE
+    client_info.client_id IS NOT NULL
+    AND sample_id = 0


You should pay attention to how long this takes to init with a 1% sample since it blocks artifact deployment during the init. It should be fine for now but we might need to do the full initial backfill another way.

I just did an init with a 2% sample and it took:

firefox_desktop: 319s

fenix: 111s
However, if I were to add @sample_id then the table would be initialized in parallel per sample_id:

bigquery-etl/bigquery_etl/cli/query.py

Line 1460 in 59139fd

To run in parallel per sample_id, include a @sample_id parameter in the query.

Would that be sufficient or is there another way to do the full initial backfill that you would recommend? Thanks!

sql_generators/baseline_clients_city_seen_v1/templates/metadata.yaml

Co-authored-by: Ben Wu <[email protected]>

dataops-ci-bot · 2025-10-03T20:54:31Z

Integration report for "Merge branch 'main' into do-2075-initialize-client-city-seen"

`sql.diff`

Click to expand!

Only in /tmp/workspace/generated-sql/dags/: bqetl_baseline_clients_city_seen.py
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_baseline_clients_city_seen.py /tmp/workspace/generated-sql/dags/bqetl_baseline_clients_city_seen.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_baseline_clients_city_seen.py	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_baseline_clients_city_seen.py	2025-10-03 20:53:31.000000000 +0000
@@ -0,0 +1,77 @@
+# Generated via https://github.com/mozilla/bigquery-etl/blob/main/bigquery_etl/query_scheduling/generate_airflow_dags.py
+
+from airflow import DAG
+from airflow.sensors.external_task import ExternalTaskMarker
+from airflow.sensors.external_task import ExternalTaskSensor
+from airflow.utils.task_group import TaskGroup
+import datetime
+from operators.gcp_container_operator import GKEPodOperator
+from utils.constants import ALLOWED_STATES, FAILED_STATES
+from utils.gcp import bigquery_etl_query, bigquery_dq_check, bigquery_bigeye_check
+
+docs = """
+### bqetl_baseline_clients_city_seen
+
+Built from bigquery-etl repo, [`dags/bqetl_baseline_clients_city_seen.py`](https://github.com/mozilla/bigquery-etl/blob/generated-sql/dags/bqetl_baseline_clients_city_seen.py)
+
+#### Description
+
+Scheduled queries for client city seen tables
+#### Owner
+
+[email protected]
+
+#### Tags
+
+* impact/tier_2
+* repo/bigquery-etl
+"""
+
+
+default_args = {
+    "owner": "[email protected]",
+    "start_date": datetime.datetime(2025, 9, 20, 0, 0),
+    "end_date": None,
+    "email": ["[email protected]", "[email protected]"],
+    "depends_on_past": False,
+    "retry_delay": datetime.timedelta(seconds=1800),
+    "email_on_failure": True,
+    "email_on_retry": True,
+    "retries": 2,
+    "max_active_tis_per_dag": None,
+}
+
+tags = ["impact/tier_2", "repo/bigquery-etl"]
+
+with DAG(
+    "bqetl_baseline_clients_city_seen",
+    default_args=default_args,
+    schedule_interval="0 4 * * *",
+    doc_md=docs,
+    tags=tags,
+    catchup=False,
+) as dag:
+
+    fenix_baseline_clients_city_seen_v1 = bigquery_etl_query(
+        task_id="fenix_baseline_clients_city_seen_v1",
+        destination_table="baseline_clients_city_seen_v1",
+        dataset_id="fenix_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=True,
+        parameters=["submission_date:DATE:{{ds}}"],
+    )
+
+    firefox_desktop_baseline_clients_city_seen_v1 = bigquery_etl_query(
+        task_id="firefox_desktop_baseline_clients_city_seen_v1",
+        destination_table="baseline_clients_city_seen_v1",
+        dataset_id="firefox_desktop_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="[email protected]",
+        email=["[email protected]", "[email protected]"],
+        date_partition_parameter=None,
+        depends_on_past=True,
+        parameters=["submission_date:DATE:{{ds}}"],
+    )
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix: baseline_clients_city_seen
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived: baseline_clients_city_seen_v1
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop: baseline_clients_city_seen
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived: baseline_clients_city_seen_v1
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_daily_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_daily_v1/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_daily_v1/metadata.yaml	2025-10-03 20:47:19.000000000 +0000
@@ -39,7 +39,7 @@
   enabled: false
   collection: null
   partition_column: null
-  partition_column_set: true
+  partition_column_set: false
   freshness:
     blocking: false
   volume:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_first_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_first_seen_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_first_seen_v1/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_first_seen_v1/metadata.yaml	2025-10-03 20:47:19.000000000 +0000
@@ -49,7 +49,7 @@
   enabled: false
   collection: null
   partition_column: null
-  partition_column_set: true
+  partition_column_set: false
   freshness:
     blocking: false
   volume:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_last_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_last_seen_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_last_seen_v1/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/baseline_clients_last_seen_v1/metadata.yaml	2025-10-03 20:47:19.000000000 +0000
@@ -48,7 +48,7 @@
   enabled: false
   collection: null
   partition_column: null
-  partition_column_set: true
+  partition_column_set: false
   freshness:
     blocking: false
   volume:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/metrics_clients_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/metrics_clients_daily_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/metrics_clients_daily_v1/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_cirrus_derived/metrics_clients_daily_v1/metadata.yaml	2025-10-03 20:47:19.000000000 +0000
@@ -34,7 +34,7 @@
   enabled: false
   collection: null
   partition_column: null
-  partition_column_set: true
+  partition_column_set: false
   freshness:
     blocking: false
   volume:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_daily_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_daily_v1/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_daily_v1/metadata.yaml	2025-10-03 20:47:18.000000000 +0000
@@ -39,7 +39,7 @@
   enabled: false
   collection: null
   partition_column: null
-  partition_column_set: true
+  partition_column_set: false
   freshness:
     blocking: false
   volume:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_first_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_first_seen_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_first_seen_v1/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_first_seen_v1/metadata.yaml	2025-10-03 20:47:18.000000000 +0000
@@ -49,7 +49,7 @@
   enabled: false
   collection: null
   partition_column: null
-  partition_column_set: true
+  partition_column_set: false
   freshness:
     blocking: false
   volume:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_last_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_last_seen_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_last_seen_v1/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/baseline_clients_last_seen_v1/metadata.yaml	2025-10-03 20:47:18.000000000 +0000
@@ -48,7 +48,7 @@
   enabled: false
   collection: null
   partition_column: null
-  partition_column_set: true
+  partition_column_set: false
   freshness:
     blocking: false
   volume:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/metrics_clients_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/metrics_clients_daily_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/metrics_clients_daily_v1/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/metrics_clients_daily_v1/metadata.yaml	2025-10-03 20:47:18.000000000 +0000
@@ -34,7 +34,7 @@
   enabled: false
   collection: null
   partition_column: null
-  partition_column_set: true
+  partition_column_set: false
   freshness:
     blocking: false
   volume:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/metrics_clients_last_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/metrics_clients_last_seen_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/metrics_clients_last_seen_v1/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/burnham_derived/metrics_clients_last_seen_v1/metadata.yaml	2025-10-03 20:47:18.000000000 +0000
@@ -48,7 +48,7 @@
   enabled: false
   collection: null
   partition_column: null
-  partition_column_set: true
+  partition_column_set: false
   freshness:
     blocking: false
   volume:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml	2025-10-03 20:37:24.000000000 +0000
@@ -1,49 +1,49 @@
 fields:
-- mode: NULLABLE
-  name: submission_date
+- name: submission_date
   type: DATE
-- mode: NULLABLE
-  name: source
+  mode: NULLABLE
+- name: source
   type: STRING
-- mode: NULLABLE
-  name: event_type
+  mode: NULLABLE
+- name: event_type
   type: STRING
-- mode: NULLABLE
-  name: form_factor
+  mode: NULLABLE
+- name: form_factor
   type: STRING
-- mode: NULLABLE
-  name: country
+  mode: NULLABLE
+- name: country
   type: STRING
-- mode: NULLABLE
-  name: subdivision1
+  mode: NULLABLE
+- name: subdivision1
   type: STRING
-- mode: NULLABLE
-  name: advertiser
+  mode: NULLABLE
+- name: advertiser
   type: STRING
-- mode: NULLABLE
-  name: release_channel
+  mode: NULLABLE
+- name: release_channel
   type: STRING
-- mode: NULLABLE
-  name: position
+  mode: NULLABLE
+- name: position
   type: INTEGER
-- mode: NULLABLE
-  name: provider
+  mode: NULLABLE
+- name: provider
   type: STRING
-- mode: NULLABLE
-  name: match_type
+  mode: NULLABLE
+- name: match_type
   type: STRING
-- mode: NULLABLE
-  name: normalized_os
+  mode: NULLABLE
+- name: normalized_os
   type: STRING
-- mode: NULLABLE
-  name: suggest_data_sharing_enabled
+  mode: NULLABLE
+- name: suggest_data_sharing_enabled
   type: BOOLEAN
-- mode: NULLABLE
-  name: event_count
+  mode: NULLABLE
+- name: event_count
   type: INTEGER
-- mode: NULLABLE
-  name: user_count
+  mode: NULLABLE
+- name: user_count
   type: INTEGER
-- mode: NULLABLE
-  name: query_type
+  mode: NULLABLE
+- name: query_type
   type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml	2025-10-03 20:37:55.000000000 +0000
@@ -1,40 +1,40 @@
 fields:
-- mode: NULLABLE
-  name: submission_date
+- name: submission_date
   type: DATE
-- mode: NULLABLE
-  name: form_factor
+  mode: NULLABLE
+- name: form_factor
   type: STRING
-- mode: NULLABLE
-  name: country
+  mode: NULLABLE
+- name: country
   type: STRING
-- mode: NULLABLE
-  name: advertiser
+  mode: NULLABLE
+- name: advertiser
   type: STRING
-- mode: NULLABLE
-  name: normalized_os
+  mode: NULLABLE
+- name: normalized_os
   type: STRING
-- mode: NULLABLE
-  name: release_channel
+  mode: NULLABLE
+- name: release_channel
   type: STRING
-- mode: NULLABLE
-  name: position
+  mode: NULLABLE
+- name: position
   type: INTEGER
-- mode: NULLABLE
-  name: provider
+  mode: NULLABLE
+- name: provider
   type: STRING
-- mode: NULLABLE
-  name: match_type
+  mode: NULLABLE
+- name: match_type
   type: STRING
-- mode: NULLABLE
-  name: suggest_data_sharing_enabled
+  mode: NULLABLE
+- name: suggest_data_sharing_enabled
   type: BOOLEAN
-- mode: NULLABLE
-  name: impression_count
+  mode: NULLABLE
+- name: impression_count
   type: INTEGER
-- mode: NULLABLE
-  name: click_count
+  mode: NULLABLE
+- name: click_count
   type: INTEGER
-- mode: NULLABLE
-  name: query_type
+  mode: NULLABLE
+- name: query_type
   type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/baseline_clients_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/baseline_clients_daily_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/baseline_clients_daily_v1/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/baseline_clients_daily_v1/metadata.yaml	2025-10-03 20:47:18.000000000 +0000
@@ -39,7 +39,7 @@
   enabled: true
   collection: null
   partition_column: null
-  partition_column_set: true
+  partition_column_set: false
   freshness:
     blocking: false
   volume:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/baseline_clients_first_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/baseline_clients_first_seen_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/baseline_clients_first_seen_v1/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/baseline_clients_first_seen_v1/metadata.yaml	2025-10-03 20:47:18.000000000 +0000
@@ -49,7 +49,7 @@
   enabled: true
   collection: null
   partition_column: null
-  partition_column_set: true
+  partition_column_set: false
   freshness:
     blocking: false
   volume:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/baseline_clients_last_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/baseline_clients_last_seen_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/baseline_clients_last_seen_v1/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/baseline_clients_last_seen_v1/metadata.yaml	2025-10-03 20:47:18.000000000 +0000
@@ -51,7 +51,7 @@
   enabled: true
   collection: null
   partition_column: null
-  partition_column_set: true
+  partition_column_set: false
   freshness:
     blocking: false
   volume:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/metrics_clients_daily_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/metrics_clients_daily_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/metrics_clients_daily_v1/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/experimenter_cirrus_derived/metrics_clients_daily_v1/metadata.yaml	2025-10-03 20:47:18.000000000 +0000
@@ -34,7 +34,7 @@
   enabled: true
   collection: null
   partition_column: null
-  partition_column_set: true
+  partition_column_set: false
   freshness:
     blocking: false
   volume:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/baseline_clients_city_seen/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/baseline_clients_city_seen/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/baseline_clients_city_seen/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/baseline_clients_city_seen/metadata.yaml	2025-10-03 20:47:18.000000000 +0000
@@ -0,0 +1,14 @@
+friendly_name: Baseline Clients City Seen
+description: |-
+  Please provide a description for the query
+owners: []
+labels: {}
+bigquery: null
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+references:
+  view.sql:
+  - moz-fx-data-shared-prod.fenix_derived.baseline_clients_city_seen_v1
+require_column_descriptions: false
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/baseline_clients_city_seen/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/baseline_clients_city_seen/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/baseline_clients_city_seen/schema.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/baseline_clients_city_seen/schema.yaml	2025-10-03 20:45:09.000000000 +0000
@@ -0,0 +1,58 @@
+fields:
+- name: app_id
+  type: STRING
+  mode: NULLABLE
+  description: ID of the browser application.
+- name: client_id
+  type: STRING
+  mode: NULLABLE
+  description: A UUID uniquely identifying the client.
+- name: sample_id
+  type: INTEGER
+  mode: NULLABLE
+  description: A number, 0-99, that samples by client_id and allows filtering data
+    for analysis.
+- name: first_seen_city_date
+  type: DATE
+  mode: NULLABLE
+  description: Date when the first seen city was captured.
+- name: first_seen_city
+  type: STRING
+  mode: NULLABLE
+  description: City captured on first_seen_city_date.
+- name: first_seen_subdivision1
+  type: STRING
+  mode: NULLABLE
+  description: Major country subdivision, typically a state, province, or county captured
+    on first_seen_city_date.
+- name: first_seen_subdivision2
+  type: STRING
+  mode: NULLABLE
+  description: Second major country subdivision; not applicable for most countries
+    captured on first_seen_city_date.
+- name: first_seen_country
+  type: STRING
+  mode: NULLABLE
+  description: An ISO 3166-1 alpha-2 country code captured on first_seen_city_date.
+- name: last_seen_city_date
+  type: DATE
+  mode: NULLABLE
+  description: Date when the last seen city was captured.
+- name: last_seen_city
+  type: STRING
+  mode: NULLABLE
+  description: City captured on last_seen_city_date.
+- name: last_seen_subdivision1
+  type: STRING
+  mode: NULLABLE
+  description: Major country subdivision, typically a state, province, or county captured
+    on last_seen_city_date.
+- name: last_seen_subdivision2
+  type: STRING
+  mode: NULLABLE
+  description: Second major country subdivision; not applicable for most countries
+    captured on last_seen_city_date.
+- name: last_seen_country
+  type: STRING
+  mode: NULLABLE
+  description: An ISO 3166-1 alpha-2 country code captured on last_seen_city_date.
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/baseline_clients_city_seen/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/baseline_clients_city_seen/view.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/baseline_clients_city_seen/view.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/baseline_clients_city_seen/view.sql	2025-10-03 20:36:52.000000000 +0000
@@ -0,0 +1,8 @@
+--- User-facing view. Generated via sql_generators.baseline_clients_city_seen.
+CREATE OR REPLACE VIEW
+  `moz-fx-data-shared-prod.fenix.baseline_clients_city_seen`
+AS
+SELECT
+  *
+FROM
+  `moz-fx-data-shared-prod.fenix_derived.baseline_clients_city_seen_v1`
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/broken_site_report/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/broken_site_report/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/broken_site_report/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/broken_site_report/metadata.yaml	2025-10-03 20:47:18.000000000 +0000
@@ -1,6 +1,10 @@
-friendly_name: Broken Site Report
+friendly_name: App-specific view for Glean ping "broken-site-report"
 description: |-
-  Please provide a description for the query
+  This a view that UNIONs the stable ping tables
+  across all channels of the Glean application "Firefox for Android"
+  (org_mozilla_firefox.broken_site_report, org_mozilla_firefox_beta.broken_site_report, org_mozilla_fenix.broken_site_report, org_mozilla_fenix_nightly.broken_site_report, org_mozilla_fennec_aurora.broken_site_report).
+
+  It is used by Looker.
 owners: []
 labels: {}
 bigquery: null
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/crash/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/crash/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/crash/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/crash/metadata.yaml	2025-10-03 20:47:18.000000000 +0000
@@ -1,6 +1,10 @@
-friendly_name: Crash
+friendly_name: App-specific view for Glean ping "crash"
 description: |-
-  Please provide a description for the query
+  This a view that UNIONs the stable ping tables
+  across all channels of the Glean application "Firefox for Android"
+  (org_mozilla_firefox.crash, org_mozilla_firefox_beta.crash, org_mozilla_fenix.crash, org_mozilla_fenix_nightly.crash, org_mozilla_fennec_aurora.crash).
+
+  It is used by Looker.
 owners: []
 labels: {}
 bigquery: null
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml	2025-10-03 20:38:11.000000000 +0000
@@ -26,6 +26,9 @@
 - name: adjust_network
   type: STRING
   mode: NULLABLE
+- name: install_source
+  type: STRING
+  mode: NULLABLE
 - name: retained_week_2
   type: BOOLEAN
   mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml	2025-10-03 20:38:11.000000000 +0000
@@ -48,6 +48,9 @@
   description: 'The type of source of a client installation.
 
     '
+- name: install_source
+  type: STRING
+  mode: NULLABLE
 - name: new_profiles
   type: INTEGER
   mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/use_counters/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/use_counters/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/use_counters/schema.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/use_counters/schema.yaml	2025-10-03 20:41:27.000000000 +0000
@@ -1,11 +1,11 @@
 fields:
 - name: normalized_app_id
-  type: STRING
   mode: NULLABLE
+  type: STRING
   description: App ID of the channel data was received from
 - name: normalized_channel
-  type: STRING
   mode: NULLABLE
+  type: STRING
   description: Normalized channel name
 - name: additional_properties
   type: STRING
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attribution_clients_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attribution_clients_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attribution_clients_v1/metadata.yaml	2025-10-03 20:31:04.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/attribution_clients_v1/metadata.yaml	2025-10-03 20:47:17.000000000 +0000
@@ -41,7 +41,7 @@
   enabled: true
   collection: Operational Checks
   partition_column: null
-  partition_column_set: true
+  partition_column_set: false
   freshness:
     blocking: false
   volume:
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/baseline_clients_city_seen_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/baseline_clients_city_seen_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/baseline_clients_city_seen_v1/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/baseline_clients_city_seen_v1/metadata.yaml	2025-10-03 20:47:17.000000000 +0000
@@ -0,0 +1,50 @@
+friendly_name: Baseline Clients City Seen
+description: |-
+  This table stores the first-seen and last-seen geo attributes for each client_id.
+  The table was initialized from stable tables (with ~2 years of retention),
+  so the initial dates reflect the earliest/latest observations within that window.
+  After initialization, the city fields in the stable tables are nullified and
+  the one-time initialization code is no longer relevant.  The table is updated daily using live tables,
+  updating last-seen fields for existing clients and adding new clients with first-seen geo attributes
+  that do not yet exist in the table.
+  This table should not be backfilled beyond 30 days which is the maximum retention of live tables.
+
+  JIRA ticket: https://mozilla-hub.atlassian.net/browse/DENG-9468
+owners:
+- [email protected]
+labels:
+  incremental: true
+  schedule: daily
+  table_type: client_level
+  dag: bqetl_baseline_clients_city_seen
+  owner1: wichan
+scheduling:
+  dag_name: bqetl_baseline_clients_city_seen
+  task_name: fenix_baseline_clients_city_seen_v1
+  depends_on_past: true
+  date_partition_parameter: null
+  parameters:
+  - submission_date:DATE:{{ds}}
+bigquery:
+  time_partitioning:
+    type: day
+    field: first_seen_city_date
+    require_partition_filter: false
+    expiration_days: null
+  range_partitioning: null
+  clustering:
+    fields:
+    - sample_id
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+references:
+  query.sql:
+  - moz-fx-data-shared-prod.fenix_derived.baseline_clients_city_seen_v1
+  - moz-fx-data-shared-prod.org_mozilla_fenix_live.baseline_v1
+  - moz-fx-data-shared-prod.org_mozilla_fenix_nightly_live.baseline_v1
+  - moz-fx-data-shared-prod.org_mozilla_fennec_aurora_live.baseline_v1
+  - moz-fx-data-shared-prod.org_mozilla_firefox_beta_live.baseline_v1
+  - moz-fx-data-shared-prod.org_mozilla_firefox_live.baseline_v1
+require_column_descriptions: false
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/baseline_clients_city_seen_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/baseline_clients_city_seen_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/baseline_clients_city_seen_v1/query.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/baseline_clients_city_seen_v1/query.sql	2025-10-03 20:36:52.000000000 +0000
@@ -0,0 +1,1243 @@
+ -- Query generated via sql_generators.baseline_clients_city_seen.
+ -- this mimics the logic used in baseline_clients_daily_v1.
+WITH base_org_mozilla_firefox AS (
+  -- Live table dedup logic mimics copy_deduplicate
+  SELECT
+    submission_timestamp,
+    DATE(submission_timestamp) AS submission_date,
+    LOWER(client_info.client_id) AS client_id,
+    sample_id,
+    mozfun.glean.parse_datetime(ping_info.end_time) AS parsed_end_time,
+    `moz-fx-data-shared-prod.udf.glean_timespan_seconds`(
+      metrics.timespan.glean_baseline_duration
+    ) AS duration,
+    metadata.geo.city AS city,
+    metadata.geo.subdivision1 AS subdivision1,
+    metadata.geo.subdivision2 AS subdivision2,
+    metadata.geo.country AS country
+  FROM
+    {% if is_init() %}
+      `moz-fx-data-shared-prod.org_mozilla_firefox_stable.baseline_v1`
+    {% else %}
+      `moz-fx-data-shared-prod.org_mozilla_firefox_live.baseline_v1`
+    {% endif %}
+  WHERE
+    client_info.client_id IS NOT NULL
+    {% if is_init() %}
+      AND sample_id = @sample_id
+      AND DATE(submission_timestamp) <= CURRENT_DATE()
+    {% else %}
+      AND DATE(submission_timestamp) = @submission_date
+      AND 'automation' NOT IN (
+        SELECT
+          TRIM(t)
+        FROM
+          UNNEST(SPLIT(metadata.header.x_source_tags, ',')) t
+      )
+      QUALIFY
+        ROW_NUMBER() OVER (PARTITION BY document_id ORDER BY submission_timestamp) = 1
+    {% endif %}
+),
+with_dates_org_mozilla_firefox AS (
+  SELECT
+    *,
+    -- For explanation of session start time calculation, see Glean docs:
+    -- https://mozilla.github.io/glean/book/user/pings/baseline.html#contents
+    DATE(SAFE.TIMESTAMP_SUB(parsed_end_time, INTERVAL duration SECOND)) AS session_start_date,
+    DATE(parsed_end_time) AS session_end_date
+  FROM
+    base_org_mozilla_firefox
+),
+with_date_offsets_org_mozilla_firefox AS (
+  SELECT
+    *,
+    DATE_DIFF(submission_date, session_start_date, DAY) AS session_start_date_offset,
+    DATE_DIFF(submission_date, session_end_date, DAY) AS session_end_date_offset,
+  FROM
+    with_dates_org_mozilla_firefox
+),
+overactive_org_mozilla_firefox AS (
+  -- Find client_ids with over 150 000 pings in a day,
+  -- which could cause errors in the next step due to aggregation overflows.
+  SELECT
+    submission_date,
+    client_id
+  FROM
+    with_date_offsets_org_mozilla_firefox
+  WHERE
+    {% if is_init() %}
+      submission_date >= '2018-01-01'
+    {% else %}
+      submission_date = @submission_date
+    {% endif %}
+  GROUP BY
+    submission_date,
+    client_id
+  HAVING
+    COUNT(*) > 150000
+),
+clients_daily_org_mozilla_firefox AS (
+  SELECT
+    "org_mozilla_firefox" AS app_id,
+    submission_date,
+    client_id,
+    sample_id,
+    `moz-fx-data-shared-prod.udf.mode_last`(
+      ARRAY_AGG(STRUCT(city, subdivision1, subdivision2, country) ORDER BY submission_timestamp)
+    ) AS geo
+  FROM
+    with_date_offsets_org_mozilla_firefox
+  LEFT JOIN
+    overactive_org_mozilla_firefox
+    USING (submission_date, client_id)
+  WHERE
+    overactive_org_mozilla_firefox.client_id IS NULL
+    -- `mode_last` can result in struct with all null values if it’s most frequent (or latest among ties).
+    -- This exclude structs with all null values so there will always be one non-NULL field.
+    AND COALESCE(city, subdivision1, subdivision2, country) IS NOT NULL
+    {% if is_init() %}
+      AND submission_date >= '2018-01-01'
+    {% else %}
+      AND submission_date = @submission_date
+    {% endif %}
+  GROUP BY
+    submission_date,
+    client_id,
+    sample_id
+),
+{% if is_init() %}
+  clients_city_first_seen_org_mozilla_firefox AS (
+    SELECT
+      app_id,
+      client_id,
+      sample_id,
+      submission_date AS first_seen_city_date,
+      geo.city AS first_seen_city,
+      geo.subdivision1 AS first_seen_subdivision1,
+      geo.subdivision2 AS first_seen_subdivision2,
+      geo.country AS first_seen_country
+    FROM
+      clients_daily_org_mozilla_firefox
+    WHERE
+      geo.city IS NOT NULL
+    QUALIFY
+      ROW_NUMBER() OVER (PARTITION BY client_id, sample_id ORDER BY submission_date) = 1
+  ),
+  clients_city_last_seen_org_mozilla_firefox AS (
+    SELECT
+      app_id,
+      client_id,
+      sample_id,
+      submission_date AS last_seen_city_date,
+      geo.city AS last_seen_city,
+      geo.subdivision1 AS last_seen_subdivision1,
+      geo.subdivision2 AS last_seen_subdivision2,
+      geo.country AS last_seen_country
+    FROM
+      clients_daily_org_mozilla_firefox
+    WHERE
+      geo.city IS NOT NULL
+    QUALIFY
+      ROW_NUMBER() OVER (PARTITION BY client_id, sample_id ORDER BY submission_date DESC) = 1
+  ),
+{% else %}
+  _previous_org_mozilla_firefox AS (
+    SELECT
+      *
+    FROM
+      `moz-fx-data-shared-prod.fenix_derived.baseline_clients_city_seen_v1`
+    WHERE
+      app_id = "org_mozilla_firefox"
+  ),
+  _current_org_mozilla_firefox AS (
+    SELECT
+      app_id,
+      client_id,
+      sample_id,
+      submission_date AS first_seen_city_date,
+      geo.city AS first_seen_city,
+      geo.subdivision1 AS first_seen_subdivision1,
+      geo.subdivision2 AS first_seen_subdivision2,
+      geo.country AS first_seen_country,
+      submission_date AS last_seen_city_date,
+      geo.city AS last_seen_city,
+      geo.subdivision1 AS last_seen_subdivision1,
+      geo.subdivision2 AS last_seen_subdivision2,
+      geo.country AS last_seen_country
+    FROM
+      clients_daily_org_mozilla_firefox
+    WHERE
+      geo.city IS NOT NULL
+  ),
+{% endif %}
+base_org_mozilla_firefox_beta AS (
+  -- Live table dedup logic mimics copy_deduplicate
+  SELECT
+    submission_timestamp,
+    DATE(submission_timestamp) AS submission_date,
+    LOWER(client_info.client_id) AS client_id,
+    sample_id,
+    mozfun.glean.parse_datetime(ping_info.end_time) AS parsed_end_time,
+    `moz-fx-data-shared-prod.udf.glean_timespan_seconds`(
+      metrics.timespan.glean_baseline_duration
+    ) AS duration,
+    metadata.geo.city AS city,
+    metadata.geo.subdivision1 AS subdivision1,
+    metadata.geo.subdivision2 AS subdivision2,
+    metadata.geo.country AS country
+  FROM
+    {% if is_init() %}
+      `moz-fx-data-shared-prod.org_mozilla_firefox_beta_stable.baseline_v1`
+    {% else %}
+      `moz-fx-data-shared-prod.org_mozilla_firefox_beta_live.baseline_v1`
+    {% endif %}
+  WHERE
+    client_info.client_id IS NOT NULL
+    {% if is_init() %}
+      AND sample_id = @sample_id
+      AND DATE(submission_timestamp) <= CURRENT_DATE()
+    {% else %}
+      AND DATE(submission_timestamp) = @submission_date
+      AND 'automation' NOT IN (
+        SELECT
+          TRIM(t)
+        FROM
+          UNNEST(SPLIT(metadata.header.x_source_tags, ',')) t
+      )
+      QUALIFY
+        ROW_NUMBER() OVER (PARTITION BY document_id ORDER BY submission_timestamp) = 1
+    {% endif %}
+),
+with_dates_org_mozilla_firefox_beta AS (
+  SELECT
+    *,
+    -- For explanation of session start time calculation, see Glean docs:
+    -- https://mozilla.github.io/glean/book/user/pings/baseline.html#contents
+    DATE(SAFE.TIMESTAMP_SUB(parsed_end_time, INTERVAL duration SECOND)) AS session_start_date,
+    DATE(parsed_end_time) AS session_end_date
+  FROM
+    base_org_mozilla_firefox_beta
+),
+with_date_offsets_org_mozilla_firefox_beta AS (
+  SELECT
+    *,
+    DATE_DIFF(submission_date, session_start_date, DAY) AS session_start_date_offset,
+    DATE_DIFF(submission_date, session_end_date, DAY) AS session_end_date_offset,
+  FROM
+    with_dates_org_mozilla_firefox_beta
+),
+overactive_org_mozilla_firefox_beta AS (
+  -- Find client_ids with over 150 000 pings in a day,
+  -- which could cause errors in the next step due to aggregation overflows.
+  SELECT
+    submission_date,
+    client_id
+  FROM
+    with_date_offsets_org_mozilla_firefox_beta
+  WHERE
+    {% if is_init() %}
+      submission_date >= '2018-01-01'
+    {% else %}
+      submission_date = @submission_date
+    {% endif %}
+  GROUP BY
+    submission_date,
+    client_id
+  HAVING
+    COUNT(*) > 150000
+),
+clients_daily_org_mozilla_firefox_beta AS (
+  SELECT
+    "org_mozilla_firefox_beta" AS app_id,
+    submission_date,
+    client_id,
+    sample_id,
+    `moz-fx-data-shared-prod.udf.mode_last`(
+      ARRAY_AGG(STRUCT(city, subdivision1, subdivision2, country) ORDER BY submission_timestamp)
+    ) AS geo
+  FROM
+    with_date_offsets_org_mozilla_firefox_beta
+  LEFT JOIN
+    overactive_org_mozilla_firefox_beta
+    USING (submission_date, client_id)
+  WHERE
+    overactive_org_mozilla_firefox_beta.client_id IS NULL
+    -- `mode_last` can result in struct with all null values if it’s most frequent (or latest among ties).
+    -- This exclude structs with all null values so there will always be one non-NULL field.
+    AND COALESCE(city, subdivision1, subdivision2, country) IS NOT NULL
+    {% if is_init() %}
+      AND submission_date >= '2018-01-01'
+    {% else %}
+      AND submission_date = @submission_date
+    {% endif %}
+  GROUP BY
+    submission_date,
+    client_id,
+    sample_id
+),
+{% if is_init() %}
+  clients_city_first_seen_org_mozilla_firefox_beta AS (
+    SELECT
+      app_id,
+      client_id,
+      sample_id,
+      submission_date AS first_seen_city_date,
+      geo.city AS first_seen_city,
+      geo.subdivision1 AS first_seen_subdivision1,
+      geo.subdivision2 AS first_seen_subdivision2,
+      geo.country AS first_seen_country
+    FROM
+      clients_daily_org_mozilla_firefox_beta
+    WHERE
+      geo.city IS NOT NULL
+    QUALIFY
+      ROW_NUMBER() OVER (PARTITION BY client_id, sample_id ORDER BY submission_date) = 1
+  ),
+  clients_city_last_seen_org_mozilla_firefox_beta AS (
+    SELECT
+      app_id,
+      client_id,
+      sample_id,
+      submission_date AS last_seen_city_date,
+      geo.city AS last_seen_city,
+      geo.subdivision1 AS last_seen_subdivision1,
+      geo.subdivision2 AS last_seen_subdivision2,
+      geo.country AS last_seen_country
+    FROM
+      clients_daily_org_mozilla_firefox_beta
+    WHERE
+      geo.city IS NOT NULL
+    QUALIFY
+      ROW_NUMBER() OVER (PARTITION BY client_id, sample_id ORDER BY submission_date DESC) = 1
+  ),
+{% else %}
+  _previous_org_mozilla_firefox_beta AS (
+    SELECT
+      *
+    FROM
+      `moz-fx-data-shared-prod.fenix_derived.baseline_clients_city_seen_v1`
+    WHERE
+      app_id = "org_mozilla_firefox_beta"
+  ),
+  _current_org_mozilla_firefox_beta AS (
+    SELECT
+      app_id,
+      client_id,
+      sample_id,
+      submission_date AS first_seen_city_date,
+      geo.city AS first_seen_city,
+      geo.subdivision1 AS first_seen_subdivision1,
+      geo.subdivision2 AS first_seen_subdivision2,
+      geo.country AS first_seen_country,
+      submission_date AS last_seen_city_date,
+      geo.city AS last_seen_city,
+      geo.subdivision1 AS last_seen_subdivision1,
+      geo.subdivision2 AS last_seen_subdivision2,
+      geo.country AS last_seen_country
+    FROM
+      clients_daily_org_mozilla_firefox_beta
+    WHERE
+      geo.city IS NOT NULL
+  ),
+{% endif %}
+base_org_mozilla_fenix AS (
+  -- Live table dedup logic mimics copy_deduplicate
+  SELECT
+    submission_timestamp,
+    DATE(submission_timestamp) AS submission_date,
+    LOWER(client_info.client_id) AS client_id,
+    sample_id,
+    mozfun.glean.parse_datetime(ping_info.end_time) AS parsed_end_time,
+    `moz-fx-data-shared-prod.udf.glean_timespan_seconds`(
+      metrics.timespan.glean_baseline_duration
+    ) AS duration,
+    metadata.geo.city AS city,
+    metadata.geo.subdivision1 AS subdivision1,
+    metadata.geo.subdivision2 AS subdivision2,
+    metadata.geo.country AS country
+  FROM
+    {% if is_init() %}
+      `moz-fx-data-shared-prod.org_mozilla_fenix_stable.baseline_v1`
+    {% else %}
+      `moz-fx-data-shared-prod.org_mozilla_fenix_live.baseline_v1`
+    {% endif %}
+  WHERE
+    client_info.client_id IS NOT NULL
+    {% if is_init() %}
+      AND sample_id = @sample_id
+      AND DATE(submission_timestamp) <= CURRENT_DATE()
+    {% else %}
+      AND DATE(submission_timestamp) = @submission_date
+      AND 'automation' NOT IN (
+        SELECT
+          TRIM(t)
+        FROM
+          UNNEST(SPLIT(metadata.header.x_source_tags, ',')) t
+      )
+      QUALIFY
+        ROW_NUMBER() OVER (PARTITION BY document_id ORDER BY submission_timestamp) = 1
+    {% endif %}
+),
+with_dates_org_mozilla_fenix AS (
+  SELECT
+    *,
+    -- For explanation of session start time calculation, see Glean docs:
+    -- https://mozilla.github.io/glean/book/user/pings/baseline.html#contents
+    DATE(SAFE.TIMESTAMP_SUB(parsed_end_time, INTERVAL duration SECOND)) AS session_start_date,
+    DATE(parsed_end_time) AS session_end_date
+  FROM
+    base_org_mozilla_fenix
+),
+with_date_offsets_org_mozilla_fenix AS (
+  SELECT
+    *,
+    DATE_DIFF(submission_date, session_start_date, DAY) AS session_start_date_offset,
+    DATE_DIFF(submission_date, session_end_date, DAY) AS session_end_date_offset,
+  FROM
+    with_dates_org_mozilla_fenix
+),
+overactive_org_mozilla_fenix AS (
+  -- Find client_ids with over 150 000 pings in a day,
+  -- which could cause errors in the next step due to aggregation overflows.
+  SELECT
+    submission_date,
+    client_id
+  FROM
+    with_date_offsets_org_mozilla_fenix
+  WHERE
+    {% if is_init() %}
+      submission_date >= '2018-01-01'
+    {% else %}
+      submission_date = @submission_date
+    {% endif %}
+  GROUP BY
+    submission_date,
+    client_id
+  HAVING
+    COUNT(*) > 150000
+),
+clients_daily_org_mozilla_fenix AS (
+  SELECT
+    "org_mozilla_fenix" AS app_id,
+    submission_date,
+    client_id,
+    sample_id,
+    `moz-fx-data-shared-prod.udf.mode_last`(
+      ARRAY_AGG(STRUCT(city, subdivision1, subdivision2, country) ORDER BY submission_timestamp)
+    ) AS geo
+  FROM
+    with_date_offsets_org_mozilla_fenix
+  LEFT JOIN
+    overactive_org_mozilla_fenix
+    USING (submission_date, client_id)
+  WHERE
+    overactive_org_mozilla_fenix.client_id IS NULL
+    -- `mode_last` can result in struct with all null values if it’s most frequent (or latest among ties).
+    -- This exclude structs with all null values so there will always be one non-NULL field.
+    AND COALESCE(city, subdivision1, subdivision2, country) IS NOT NULL
+    {% if is_init() %}
+      AND submission_date >= '2018-01-01'
+    {% else %}
+      AND submission_date = @submission_date
+    {% endif %}
+  GROUP BY
+    submission_date,
+    client_id,
+    sample_id
+),
+{% if is_init() %}
+  clients_city_first_seen_org_mozilla_fenix AS (
+    SELECT
+      app_id,
+      client_id,
+      sample_id,
+      submission_date AS first_seen_city_date,
+      geo.city AS first_seen_city,
+      geo.subdivision1 AS first_seen_subdivision1,
+      geo.subdivision2 AS first_seen_subdivision2,
+      geo.country AS first_seen_country
+    FROM
+      clients_daily_org_mozilla_fenix
+    WHERE
+      geo.city IS NOT NULL
+    QUALIFY
+      ROW_NUMBER() OVER (PARTITION BY client_id, sample_id ORDER BY submission_date) = 1
+  ),
+  clients_city_last_seen_org_mozilla_fenix AS (
+    SELECT
+      app_id,
+      client_id,
+      sample_id,
+      submission_date AS last_seen_city_date,
+      geo.city AS last_seen_city,
+      geo.subdivision1 AS last_seen_subdivision1,
+      geo.subdivision2 AS last_seen_subdivision2,
+      geo.country AS last_seen_country
+    FROM
+      clients_daily_org_mozilla_fenix
+    WHERE
+      geo.city IS NOT NULL
+    QUALIFY
+      ROW_NUMBER() OVER (PARTITION BY client_id, sample_id ORDER BY submission_date DESC) = 1
+  ),
+{% else %}
+  _previous_org_mozilla_fenix AS (
+    SELECT
+      *
+    FROM
+      `moz-fx-data-shared-prod.fenix_derived.baseline_clients_city_seen_v1`
+    WHERE
+      app_id = "org_mozilla_fenix"
+  ),
+  _current_org_mozilla_fenix AS (
+    SELECT
+      app_id,
+      client_id,
+      sample_id,
+      submission_date AS first_seen_city_date,
+      geo.city AS first_seen_city,
+      geo.subdivision1 AS first_seen_subdivision1,
+      geo.subdivision2 AS first_seen_subdivision2,
+      geo.country AS first_seen_country,
+      submission_date AS last_seen_city_date,
+      geo.city AS last_seen_city,
+      geo.subdivision1 AS last_seen_subdivision1,
+      geo.subdivision2 AS last_seen_subdivision2,
+      geo.country AS last_seen_country
+    FROM
+      clients_daily_org_mozilla_fenix
+    WHERE
+      geo.city IS NOT NULL
+  ),
+{% endif %}
+base_org_mozilla_fenix_nightly AS (
+  -- Live table dedup logic mimics copy_deduplicate
+  SELECT
+    submission_timestamp,
+    DATE(submission_timestamp) AS submission_date,
+    LOWER(client_info.client_id) AS client_id,
+    sample_id,
+    mozfun.glean.parse_datetime(ping_info.end_time) AS parsed_end_time,
+    `moz-fx-data-shared-prod.udf.glean_timespan_seconds`(
+      metrics.timespan.glean_baseline_duration
+    ) AS duration,
+    metadata.geo.city AS city,
+    metadata.geo.subdivision1 AS subdivision1,
+    metadata.geo.subdivision2 AS subdivision2,
+    metadata.geo.country AS country
+  FROM
+    {% if is_init() %}
+      `moz-fx-data-shared-prod.org_mozilla_fenix_nightly_stable.baseline_v1`
+    {% else %}
+      `moz-fx-data-shared-prod.org_mozilla_fenix_nightly_live.baseline_v1`
+    {% endif %}
+  WHERE
+    client_info.client_id IS NOT NULL
+    {% if is_init() %}
+      AND sample_id = @sample_id
+      AND DATE(submission_timestamp) <= CURRENT_DATE()
+    {% else %}
+      AND DATE(submission_timestamp) = @submission_date
+      AND 'automation' NOT IN (
+        SELECT
+          TRIM(t)
+        FROM
+          UNNEST(SPLIT(metadata.header.x_source_tags, ',')) t
+      )
+      QUALIFY
+        ROW_NUMBER() OVER (PARTITION BY document_id ORDER BY submission_timestamp) = 1
+    {% endif %}
+),
+with_dates_org_mozilla_fenix_nightly AS (
+  SELECT
+    *,
+    -- For explanation of session start time calculation, see Glean docs:
+    -- https://mozilla.github.io/glean/book/user/pings/baseline.html#contents
+    DATE(SAFE.TIMESTAMP_SUB(parsed_end_time, INTERVAL duration SECOND)) AS session_start_date,
+    DATE(parsed_end_time) AS session_end_date
+  FROM
+    base_org_mozilla_fenix_nightly
+),
+with_date_offsets_org_mozilla_fenix_nightly AS (
+  SELECT
+    *,
+    DATE_DIFF(submission_date, session_start_date, DAY) AS session_start_date_offset,
+    DATE_DIFF(submission_date, session_end_date, DAY) AS session_end_date_offset,
+  FROM
+    with_dates_org_mozilla_fenix_nightly
+),
+overactive_org_mozilla_fenix_nightly AS (
+  -- Find client_ids with over 150 000 pings in a day,
+  -- which could cause errors in the next step due to aggregation overflows.
+  SELECT
+    submission_date,
+    client_id
+  FROM
+    with_date_offsets_org_mozilla_fenix_nightly
+  WHERE
+    {% if is_init() %}
+      submission_date >= '2018-01-01'
+    {% else %}
+      submission_date = @submission_date
+    {% endif %}
+  GROUP BY
+    submission_date,
+    client_id
+  HAVING
+    COUNT(*) > 150000
+),
+clients_daily_org_mozilla_fenix_nightly AS (
+  SELECT
+    "org_mozilla_fenix_nightly" AS app_id,
+    submission_date,
+    client_id,
+    sample_id,
+    `moz-fx-data-shared-prod.udf.mode_last`(
+      ARRAY_AGG(STRUCT(city, subdivision1, subdivision2, country) ORDER BY submission_timestamp)
+    ) AS geo
+  FROM
+    with_date_offsets_org_mozilla_fenix_nightly
+  LEFT JOIN
+    overactive_org_mozilla_fenix_nightly
+    USING (submission_date, client_id)
+  WHERE
+    overactive_org_mozilla_fenix_nightly.client_id IS NULL
+    -- `mode_last` can result in struct with all null values if it’s most frequent (or latest among ties).
+    -- This exclude structs with all null values so there will always be one non-NULL field.
+    AND COALESCE(city, subdivision1, subdivision2, country) IS NOT NULL
+    {% if is_init() %}
+      AND submission_date >= '2018-01-01'
+    {% else %}
+      AND submission_date = @submission_date
+    {% endif %}
+  GROUP BY
+    submission_date,
+    client_id,
+    sample_id
+),
+{% if is_init() %}
+  clients_city_first_seen_org_mozilla_fenix_nightly AS (
+    SELECT
+      app_id,
+      client_id,
+      sample_id,
+      submission_date AS first_seen_city_date,
+      geo.city AS first_seen_city,
+      geo.subdivision1 AS first_seen_subdivision1,
+      geo.subdivision2 AS first_seen_subdivision2,
+      geo.country AS first_seen_country
+    FROM
+      clients_daily_org_mozilla_fenix_nightly
+    WHERE
+      geo.city IS NOT NULL
+    QUALIFY
+      ROW_NUMBER() OVER (PARTITION BY client_id, sample_id ORDER BY submission_date) = 1
+  ),
+  clients_city_last_seen_org_mozilla_fenix_nightly AS (
+    SELECT
+      app_id,
+      client_id,
+      sample_id,
+      submission_date AS last_seen_city_date,
+      geo.city AS last_seen_city,
+      geo.subdivision1 AS last_seen_subdivision1,
+      geo.subdivision2 AS last_seen_subdivision2,
+      geo.country AS last_seen_country
+    FROM
+      clients_daily_org_mozilla_fenix_nightly
+    WHERE
+      geo.city IS NOT NULL
+    QUALIFY
+      ROW_NUMBER() OVER (PARTITION BY client_id, sample_id ORDER BY submission_date DESC) = 1
+  ),
+{% else %}
+  _previous_org_mozilla_fenix_nightly AS (
+    SELECT
+      *
+    FROM
+      `moz-fx-data-shared-prod.fenix_derived.baseline_clients_city_seen_v1`
+    WHERE
+      app_id = "org_mozilla_fenix_nightly"
+  ),
+  _current_org_mozilla_fenix_nightly AS (
+    SELECT
+      app_id,
+      client_id,
+      sample_id,
+      submission_date AS first_seen_city_date,
+      geo.city AS first_seen_city,
+      geo.subdivision1 AS first_seen_subdivision1,
+      geo.subdivision2 AS first_seen_subdivision2,
+      geo.country AS first_seen_country,
+      submission_date AS last_seen_city_date,
+      geo.city AS last_seen_city,
+      geo.subdivision1 AS last_seen_subdivision1,
+      geo.subdivision2 AS last_seen_subdivision2,
+      geo.country AS last_seen_country
+    FROM
+      clients_daily_org_mozilla_fenix_nightly
+    WHERE
+      geo.city IS NOT NULL
+  ),
+{% endif %}
+base_org_mozilla_fennec_aurora AS (
+  -- Live table dedup logic mimics copy_deduplicate
+  SELECT
+    submission_timestamp,
+    DATE(submission_timestamp) AS submission_date,
+    LOWER(client_info.client_id) AS client_id,
+    sample_id,
+    mozfun.glean.parse_datetime(ping_info.end_time) AS parsed_end_time,
+    `moz-fx-data-shared-prod.udf.glean_timespan_seconds`(
+      metrics.timespan.glean_baseline_duration
+    ) AS duration,
+    metadata.geo.city AS city,
+    metadata.geo.subdivision1 AS subdivision1,
+    metadata.geo.subdivision2 AS subdivision2,
+    metadata.geo.country AS country
+  FROM
+    {% if is_init() %}
+      `moz-fx-data-shared-prod.org_mozilla_fennec_aurora_stable.baseline_v1`
+    {% else %}
+      `moz-fx-data-shared-prod.org_mozilla_fennec_aurora_live.baseline_v1`
+    {% endif %}
+  WHERE
+    client_info.client_id IS NOT NULL
+    {% if is_init() %}
+      AND sample_id = @sample_id
+      AND DATE(submission_timestamp) <= CURRENT_DATE()
+    {% else %}
+      AND DATE(submission_timestamp) = @submission_date
+      AND 'automation' NOT IN (
+        SELECT
+          TRIM(t)
+        FROM
+          UNNEST(SPLIT(metadata.header.x_source_tags, ',')) t
+      )
+      QUALIFY
+        ROW_NUMBER() OVER (PARTITION BY document_id ORDER BY submission_timestamp) = 1
+    {% endif %}
+),
+with_dates_org_mozilla_fennec_aurora AS (
+  SELECT
+    *,
+    -- For explanation of session start time calculation, see Glean docs:
+    -- https://mozilla.github.io/glean/book/user/pings/baseline.html#contents
+    DATE(SAFE.TIMESTAMP_SUB(parsed_end_time, INTERVAL duration SECOND)) AS session_start_date,
+    DATE(parsed_end_time) AS session_end_date
+  FROM
+    base_org_mozilla_fennec_aurora
+),
+with_date_offsets_org_mozilla_fennec_aurora AS (
+  SELECT
+    *,
+    DATE_DIFF(submission_date, session_start_date, DAY) AS session_start_date_offset,
+    DATE_DIFF(submission_date, session_end_date, DAY) AS session_end_date_offset,
+  FROM
+    with_dates_org_mozilla_fennec_aurora
+),
+overactive_org_mozilla_fennec_aurora AS (
+  -- Find client_ids with over 150 000 pings in a day,
+  -- which could cause errors in the next step due to aggregation overflows.
+  SELECT
+    submission_date,
+    client_id
+  FROM
+    with_date_offsets_org_mozilla_fennec_aurora
+  WHERE
+    {% if is_init() %}
+      submission_date >= '2018-01-01'
+    {% else %}
+      submission_date = @submission_date
+    {% endif %}
+  GROUP BY
+    submission_date,
+    client_id
+  HAVING
+    COUNT(*) > 150000
+),
+clients_daily_org_mozilla_fennec_aurora AS (
+  SELECT
+    "org_mozilla_fennec_aurora" AS app_id,
+    submission_date,
+    client_id,
+    sample_id,
+    `moz-fx-data-shared-prod.udf.mode_last`(
+      ARRAY_AGG(STRUCT(city, subdivision1, subdivision2, country) ORDER BY submission_timestamp)
+    ) AS geo
+  FROM
+    with_date_offsets_org_mozilla_fennec_aurora
+  LEFT JOIN
+    overactive_org_mozilla_fennec_aurora
+    USING (submission_date, client_id)
+  WHERE
+    overactive_org_mozilla_fennec_aurora.client_id IS NULL
+    -- `mode_last` can result in struct with all null values if it’s most frequent (or latest among ties).
+    -- This exclude structs with all null values so there will always be one non-NULL field.
+    AND COALESCE(city, subdivision1, subdivision2, country) IS NOT NULL
+    {% if is_init() %}
+      AND submission_date >= '2018-01-01'
+    {% else %}
+      AND submission_date = @submission_date
+    {% endif %}
+  GROUP BY
+    submission_date,
+    client_id,
+    sample_id
+),
+{% if is_init() %}
+  clients_city_first_seen_org_mozilla_fennec_aurora AS (
+    SELECT
+      app_id,
+      client_id,
+      sample_id,
+      submission_date AS first_seen_city_date,
+      geo.city AS first_seen_city,
+      geo.subdivision1 AS first_seen_subdivision1,
+      geo.subdivision2 AS first_seen_subdivision2,
+      geo.country AS first_seen_country
+    FROM
+      clients_daily_org_mozilla_fennec_aurora
+    WHERE
+      geo.city IS NOT NULL
+    QUALIFY
+      ROW_NUMBER() OVER (PARTITION BY client_id, sample_id ORDER BY submission_date) = 1
+  ),
+  clients_city_last_seen_org_mozilla_fennec_aurora AS (
+    SELECT
+      app_id,
+      client_id,
+      sample_id,
+      submission_date AS last_seen_city_date,
+      geo.city AS last_seen_city,
+      geo.subdivision1 AS last_seen_subdivision1,
+      geo.subdivision2 AS last_seen_subdivision2,
+      geo.country AS last_seen_country
+    FROM
+      clients_daily_org_mozilla_fennec_aurora
+    WHERE
+      geo.city IS NOT NULL
+    QUALIFY
+      ROW_NUMBER() OVER (PARTITION BY client_id, sample_id ORDER BY submission_date DESC) = 1
+  )
+{% else %}
+  _previous_org_mozilla_fennec_aurora AS (
+    SELECT
+      *
+    FROM
+      `moz-fx-data-shared-prod.fenix_derived.baseline_clients_city_seen_v1`
+    WHERE
+      app_id = "org_mozilla_fennec_aurora"
+  ),
+  _current_org_mozilla_fennec_aurora AS (
+    SELECT
+      app_id,
+      client_id,
+      sample_id,
+      submission_date AS first_seen_city_date,
+      geo.city AS first_seen_city,
+      geo.subdivision1 AS first_seen_subdivision1,
+      geo.subdivision2 AS first_seen_subdivision2,
+      geo.country AS first_seen_country,
+      submission_date AS last_seen_city_date,
+      geo.city AS last_seen_city,
+      geo.subdivision1 AS last_seen_subdivision1,
+      geo.subdivision2 AS last_seen_subdivision2,
+      geo.country AS last_seen_country
+    FROM
+      clients_daily_org_mozilla_fennec_aurora
+    WHERE
+      geo.city IS NOT NULL
+  )
+{% endif %}
+{% if is_init() %}
+  SELECT
+    app_id,
+    client_id,
+    sample_id,
+    first_seen_city_date,
+    first_seen_city,
+    first_seen_subdivision1,
+    first_seen_subdivision2,
+    first_seen_country,
+    last_seen_city_date,
+    last_seen_city,
+    last_seen_subdivision1,
+    last_seen_subdivision2,
+    last_seen_country
+  FROM
+    clients_city_first_seen_org_mozilla_firefox cfs
+  FULL OUTER JOIN
+    clients_city_last_seen_org_mozilla_firefox cls
+    USING (client_id, sample_id, app_id)
+  UNION ALL
+  SELECT
+    app_id,
+    client_id,
+    sample_id,
+    first_seen_city_date,
+    first_seen_city,
+    first_seen_subdivision1,
+    first_seen_subdivision2,
+    first_seen_country,
+    last_seen_city_date,
+    last_seen_city,
+    last_seen_subdivision1,
+    last_seen_subdivision2,
+    last_seen_country
+  FROM
+    clients_city_first_seen_org_mozilla_firefox_beta cfs
+  FULL OUTER JOIN
+    clients_city_last_seen_org_mozilla_firefox_beta cls
+    USING (client_id, sample_id, app_id)
+  UNION ALL
+  SELECT
+    app_id,
+    client_id,
+    sample_id,
+    first_seen_city_date,
+    first_seen_city,
+    first_seen_subdivision1,
+    first_seen_subdivision2,
+    first_seen_country,
+    last_seen_city_date,
+    last_seen_city,
+    last_seen_subdivision1,
+    last_seen_subdivision2,
+    last_seen_country
+  FROM
+    clients_city_first_seen_org_mozilla_fenix cfs
+  FULL OUTER JOIN
+    clients_city_last_seen_org_mozilla_fenix cls
+    USING (client_id, sample_id, app_id)
+  UNION ALL
+  SELECT
+    app_id,
+    client_id,
+    sample_id,
+    first_seen_city_date,
+    first_seen_city,
+    first_seen_subdivision1,
+    first_seen_subdivision2,
+    first_seen_country,
+    last_seen_city_date,
+    last_seen_city,
+    last_seen_subdivision1,
+    last_seen_subdivision2,
+    last_seen_country
+  FROM
+    clients_city_first_seen_org_mozilla_fenix_nightly cfs
+  FULL OUTER JOIN
+    clients_city_last_seen_org_mozilla_fenix_nightly cls
+    USING (client_id, sample_id, app_id)
+  UNION ALL
+  SELECT
+    app_id,
+    client_id,
+    sample_id,
+    first_seen_city_date,
+    first_seen_city,
+    first_seen_subdivision1,
+    first_seen_subdivision2,
+    first_seen_country,
+    last_seen_city_date,
+    last_seen_city,
+    last_seen_subdivision1,
+    last_seen_subdivision2,
+    last_seen_country
+  FROM
+    clients_city_first_seen_org_mozilla_fennec_aurora cfs
+  FULL OUTER JOIN
+    clients_city_last_seen_org_mozilla_fennec_aurora cls
+    USING (client_id, sample_id, app_id)
+{% else %}
+  SELECT
+-- _p.* fields are NULL for clients that are not yet captured in the baseline_city_seen derived table.
+    IF(_p.app_id IS NULL, _c.app_id, _p.app_id) AS app_id,
+    IF(_p.client_id IS NULL, _c.client_id, _p.client_id) AS client_id,
+    IF(_p.sample_id IS NULL, _c.sample_id, _p.sample_id) AS sample_id,
+    IF(
+      _p.client_id IS NULL,
+      _c.first_seen_city_date,
+      _p.first_seen_city_date
+    ) AS first_seen_city_date,
+    IF(_p.client_id IS NULL, _c.first_seen_city, _p.first_seen_city) AS first_seen_city,
+    IF(
+      _p.client_id IS NULL,
+      _c.first_seen_subdivision1,
+      _p.first_seen_subdivision1
+    ) AS first_seen_subdivision1,
+    IF(
+      _p.client_id IS NULL,
+      _c.first_seen_subdivision2,
+      _p.first_seen_subdivision2
+    ) AS first_seen_subdivision2,
+    IF(_p.client_id IS NULL, _c.first_seen_country, _p.first_seen_country) AS first_seen_country,
+    IF(
+      _p.client_id IS NULL
+      OR _p.last_seen_city_date < _c.last_seen_city_date,
+      _c.last_seen_city_date,
+      _p.last_seen_city_date
+    ) AS last_seen_city_date,
+    IF(
+      _p.client_id IS NULL
+      OR _p.last_seen_city_date < _c.last_seen_city_date,
+      _c.last_seen_city,
+      _p.last_seen_city
+    ) AS last_seen_city,
+    IF(
+      _p.client_id IS NULL
+      OR _p.last_seen_city_date < _c.last_seen_city_date,
+      _c.last_seen_subdivision1,
+      _p.last_seen_subdivision1
+    ) AS last_seen_subdivision1,
+    IF(
+      _p.client_id IS NULL
+      OR _p.last_seen_city_date < _c.last_seen_city_date,
+      _c.last_seen_subdivision2,
+      _p.last_seen_subdivision2
+    ) AS last_seen_subdivision2,
+    IF(
+      _p.client_id IS NULL
+      OR _p.last_seen_city_date < _c.last_seen_city_date,
+      _c.last_seen_country,
+      _p.last_seen_country
+    ) AS last_seen_country
+  FROM
+    _current_org_mozilla_firefox AS _c
+  FULL JOIN
+    _previous_org_mozilla_firefox AS _p
+    USING (client_id, sample_id, app_id)
+  UNION ALL
+  SELECT
+-- _p.* fields are NULL for clients that are not yet captured in the baseline_city_seen derived table.
+    IF(_p.app_id IS NULL, _c.app_id, _p.app_id) AS app_id,
+    IF(_p.client_id IS NULL, _c.client_id, _p.client_id) AS client_id,
+    IF(_p.sample_id IS NULL, _c.sample_id, _p.sample_id) AS sample_id,
+    IF(
+      _p.client_id IS NULL,
+      _c.first_seen_city_date,
+      _p.first_seen_city_date
+    ) AS first_seen_city_date,
+    IF(_p.client_id IS NULL, _c.first_seen_city, _p.first_seen_city) AS first_seen_city,
+    IF(
+      _p.client_id IS NULL,
+      _c.first_seen_subdivision1,
+      _p.first_seen_subdivision1
+    ) AS first_seen_subdivision1,
+    IF(
+      _p.client_id IS NULL,
+      _c.first_seen_subdivision2,
+      _p.first_seen_subdivision2
+    ) AS first_seen_subdivision2,
+    IF(_p.client_id IS NULL, _c.first_seen_country, _p.first_seen_country) AS first_seen_country,
+    IF(
+      _p.client_id IS NULL
+      OR _p.last_seen_city_date < _c.last_seen_city_date,
+      _c.last_seen_city_date,
+      _p.last_seen_city_date
+    ) AS last_seen_city_date,
+    IF(
+      _p.client_id IS NULL
+      OR _p.last_seen_city_date < _c.last_seen_city_date,
+      _c.last_seen_city,
+      _p.last_seen_city
+    ) AS last_seen_city,
+    IF(
+      _p.client_id IS NULL
+      OR _p.last_seen_city_date < _c.last_seen_city_date,
+      _c.last_seen_subdivision1,
+      _p.last_seen_subdivision1
+    ) AS last_seen_subdivision1,
+    IF(
+      _p.client_id IS NULL
+      OR _p.last_seen_city_date < _c.last_seen_city_date,
+      _c.last_seen_subdivision2,
+      _p.last_seen_subdivision2
+    ) AS last_seen_subdivision2,
+    IF(
+      _p.client_id IS NULL
+      OR _p.last_seen_city_date < _c.last_seen_city_date,
+      _c.last_seen_country,
+      _p.last_seen_country
+    ) AS last_seen_country
+  FROM
+    _current_org_mozilla_firefox_beta AS _c
+  FULL JOIN
+    _previous_org_mozilla_firefox_beta AS _p
+    USING (client_id, sample_id, app_id)
+  UNION ALL
+  SELECT
+-- _p.* fields are NULL for clients that are not yet captured in the baseline_city_seen derived table.
+    IF(_p.app_id IS NULL, _c.app_i

⚠️ Only part of the diff is displayed.

Link to full diff

wwyc · 2025-10-03T21:02:43Z

@BenWu @soGaussian could you do a final review/approval on this PR? I’d like to merge early next week.

@BenWu is there a time of day you recommend for the merge so initialization doesn’t affect artifact deployments? Thanks!

wwyc requested a review from a team as a code owner August 19, 2025 23:01

wwyc marked this pull request as draft August 19, 2025 23:02

wwyc commented Aug 19, 2025

View reviewed changes

sql/moz-fx-data-shared-prod/fenix_derived/clients_city_seen_v1/schema.yaml Outdated Show resolved Hide resolved

wwyc requested a review from BenWu August 19, 2025 23:07

This comment has been minimized.

Sign in to view

lucia-vargas-a reviewed Aug 20, 2025

View reviewed changes

sql/moz-fx-data-shared-prod/fenix_derived/clients_city_seen_v1/metadata.yaml Outdated Show resolved Hide resolved

lucia-vargas-a reviewed Aug 20, 2025

View reviewed changes

sql/moz-fx-data-shared-prod/fenix_derived/clients_city_seen_v1/schema.yaml Outdated Show resolved Hide resolved

lucia-vargas-a reviewed Aug 20, 2025

View reviewed changes

sql/moz-fx-data-shared-prod/fenix_derived/clients_city_seen_v1/schema.yaml Outdated Show resolved Hide resolved

lucia-vargas-a reviewed Aug 20, 2025

View reviewed changes

sql/moz-fx-data-shared-prod/firefox_desktop_derived/clients_city_seen_v1/query.sql Outdated Show resolved Hide resolved

lucia-vargas-a reviewed Aug 20, 2025

View reviewed changes

sql/moz-fx-data-shared-prod/firefox_desktop_derived/clients_city_seen_v1/query.sql Outdated Show resolved Hide resolved

lucia-vargas-a reviewed Aug 20, 2025

View reviewed changes

sql_generators/clients_city_seen_v1/templates/metadata.yaml Outdated Show resolved Hide resolved

lucia-vargas-a reviewed Aug 20, 2025

View reviewed changes

sql/moz-fx-data-shared-prod/fenix_derived/clients_city_seen_v1/metadata.yaml Outdated Show resolved Hide resolved

lucia-vargas-a reviewed Aug 20, 2025

View reviewed changes

sql_generators/clients_city_seen_v1/templates/metadata.yaml Outdated Show resolved Hide resolved

BenWu reviewed Aug 20, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

wwyc changed the title ~~DO-2075 Added fenix and desktop city seen initial table - DO NOT MERGE~~ DO-2075 Added fenix and desktop city seen initial table - DO NOT MERGE (DRAFT) Aug 27, 2025

This comment has been minimized.

Sign in to view

wwyc force-pushed the do-2075-initialize-client-city-seen branch from 64349bd to dc75f81 Compare August 28, 2025 21:29

This comment has been minimized.

Sign in to view

wwyc requested a review from BenWu August 28, 2025 22:31

wwyc force-pushed the do-2075-initialize-client-city-seen branch from ce4fe26 to ff85557 Compare August 29, 2025 19:16

This comment has been minimized.

Sign in to view

wwyc requested a review from soGaussian September 2, 2025 17:48

BenWu reviewed Sep 2, 2025

View reviewed changes

sql_generators/baseline_clients_city_seen_v1/templates/query.sql Outdated Show resolved Hide resolved

sql_generators/baseline_clients_city_seen_v1/templates/metadata.yaml Outdated Show resolved Hide resolved

BenWu mentioned this pull request Sep 22, 2025

DENG-9727: Added mode last struct retain nulls udf #8130

Closed

This comment has been minimized.

Sign in to view

wwyc force-pushed the do-2075-initialize-client-city-seen branch from 24fd5b0 to 33d0b9f Compare September 23, 2025 00:08

This comment has been minimized.

Sign in to view

Fixed dry run error

20bd7b5

This comment has been minimized.

Sign in to view

Fixed geo alias

df125cb

This comment has been minimized.

Sign in to view

Added country fields

adee237

This comment has been minimized.

Sign in to view

wwyc added 3 commits September 23, 2025 18:24

Added tests

59ea268

Updated query

a166c86

Added query comments

3bfd2de

This comment has been minimized.

Sign in to view

Merge branch 'main' into do-2075-initialize-client-city-seen

3be374b

This comment has been minimized.

Sign in to view

Merge branch 'main' into do-2075-initialize-client-city-seen

880f231

This comment has been minimized.

Sign in to view

BenWu reviewed Sep 25, 2025

View reviewed changes

wwyc and others added 2 commits September 25, 2025 13:47

removed depends on

287f78a

Update sql_generators/baseline_clients_city_seen_v1/__init__.py

416ebe8

Co-authored-by: Ben Wu <[email protected]>

This comment has been minimized.

Sign in to view

Added allow apps in config

7b18c89

This comment has been minimized.

Sign in to view

wwyc and others added 2 commits October 3, 2025 13:04

Added sample_id parameter

ea8f209

Merge branch 'main' into do-2075-initialize-client-city-seen

b2ede49

wwyc changed the title ~~DO-2075 Added fenix and desktop city seen initial table - DO NOT MERGE (DRAFT)~~ DO-2075 Added fenix and desktop baseline city seen initial tables Oct 3, 2025

wwyc marked this pull request as ready for review October 3, 2025 20:06

wwyc changed the title ~~DO-2075 Added fenix and desktop baseline city seen initial tables~~ DO-2075 Added fenix and desktop baseline city seen tables Oct 3, 2025

Merge branch 'main' into do-2075-initialize-client-city-seen

44e2f77

DO-2075 Added fenix and desktop baseline city seen tables #7974

Are you sure you want to change the base?

DO-2075 Added fenix and desktop baseline city seen tables #7974

Conversation

wwyc commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Tickets & Documents

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

BenWu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

BenWu Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

wwyc Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

dataops-ci-bot commented Oct 3, 2025

Integration report for "Merge branch 'main' into do-2075-initialize-client-city-seen"

sql.diff

Uh oh!

wwyc commented Oct 3, 2025

Uh oh!

Uh oh!

wwyc commented Aug 19, 2025 •

edited

Loading

`sql.diff`