Use AIRFLOW_ENV=development on .development.env file

[erikamov]

Description

Based on researching the past commits, the only place that may need to be changed is on .development.env file. Since there was not a staging environment when it was created, there are tests only for two options:development or cal-itp-data-infra.

This PR sets AIRFLOW_ENV=development on .development.env file.

Resolves [#4426]

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

How has this been tested?

Post-merge follow-ups

• No action required
• Actions required (specified below)

[lauriemerrell]

One comment, otherwise LGTM. I had made the fix in dags.py but I can just rebase on this

[lauriemerrell]

Ope, missed a save -- here's the comment

[lauriemerrell]

As noted in the ticket, I am just not sure what this env var does in this context, not sure if there's been a functional issue on the test archiver?

[erikamov]

Yeah, I haven't work with that yet. I asked about your tests to see if any were related to this on.
We definitely need to test it before merging this change.

[erikamov]

@lauriemerrell I reviewed my previous changes, and I think they were wrong.
The only place that looks like we would need to change is on this .development.env file.
Can you test your test archiver with this change?

[lauriemerrell]

@erikamov I think you and I went in opposite directions -- here, I went in the direction of having the code check for the new value: github.com/cal-itp/data-infra/commit/df5a106421e2ddadd3fe4a35129e7b7042fd6b17. I guess either way is fine, I wasn't sure if changing it back to development would cause other problems either

I don't have a local archiver instance so not sure best way to test that

[lauriemerrell]

(Can try to set something up but not sure of overhead)

[lauriemerrell]

Basically I discovered this issue on #4350 and I resolved it for that use case by just checking for the updated value, I didn't want to scope creep on that PR so wrote this ticket, but I do not have a handle on the other places that variable is checked

[github-actions]

Terraform plan in iac/cal-itp-data-infra-staging/composer/us

No changes. Your infrastructure matches the configuration.

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1108

[github-actions]

Terraform plan in iac/cal-itp-data-infra-staging/airflow/us

Plan: 0 to add, 2 to change, 0 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-staging-composer-catalog will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-catalog" {
!~      content             = (sensitive value)
!~      crc32c              = "2FXNxw==" -> (known after apply)
!~      detect_md5hash      = "bE9f/PjreIWPig75/NS6Vw==" -> "different hash"
!~      generation          = 1764122637987817 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/target/catalog.json"
!~      md5hash             = "bE9f/PjreIWPig75/NS6Vw==" -> (known after apply)
        name                = "data/warehouse/target/catalog.json"
#        (16 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-manifest will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-manifest" {
!~      content             = (sensitive value)
!~      crc32c              = "5pENHQ==" -> (known after apply)
!~      detect_md5hash      = "3KrGaUvOP8W/6eVlzvOsOw==" -> "different hash"
!~      generation          = 1764122639162554 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/target/manifest.json"
!~      md5hash             = "3KrGaUvOP8W/6eVlzvOsOw==" -> (known after apply)
        name                = "data/warehouse/target/manifest.json"
#        (16 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1108

[github-actions]

Terraform plan in iac/cal-itp-data-infra/airflow/us

Plan: 0 to add, 2 to change, 0 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-composer-catalog will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-catalog" {
!~      content             = (sensitive value)
!~      crc32c              = "00iJ/w==" -> (known after apply)
!~      detect_md5hash      = "yfkZas/eBQMAqfz2eXnuDw==" -> "different hash"
!~      generation          = 1764100054963994 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/target/catalog.json"
!~      md5hash             = "yfkZas/eBQMAqfz2eXnuDw==" -> (known after apply)
        name                = "data/warehouse/target/catalog.json"
#        (16 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-manifest will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-manifest" {
!~      content             = (sensitive value)
!~      crc32c              = "S9xfig==" -> (known after apply)
!~      detect_md5hash      = "GEAgWaMaB3xMcxPCFeJXSg==" -> "different hash"
!~      generation          = 1764100056657852 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/target/manifest.json"
!~      md5hash             = "GEAgWaMaB3xMcxPCFeJXSg==" -> (known after apply)
        name                = "data/warehouse/target/manifest.json"
#        (16 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1108

[lauriemerrell]

Post #4459, should figure out if we need to re-apply the change that was made in 9b73b4d but it might be obsolete post #4548