Skip to content

feat: add Application Insights for OTEL tracing and uat->staging rename #68

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JustAGhosT merged 11 commits into from
Mar 15, 2026
Merged

feat: add Application Insights for OTEL tracing and uat->staging rename #68

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
8eef6e1
refactor: rename UAT environment to staging
JustAGhosT Mar 15, 2026
901509d
refactor: change environment validation from "uat" to "staging"
JustAGhosT Mar 15, 2026
3945195
refactor(infra): replace "uat" with "staging" in environment validation
JustAGhosT Mar 15, 2026
8f6c24a
refactor: rename UAT environment to staging
JustAGhosT Mar 15, 2026
4f315b2
feat: add Application Insights for OTEL tracing and complete uat->sta…
JustAGhosT Mar 15, 2026
eb07474
fix: address PR review comments - terminology consistency
JustAGhosT Mar 15, 2026
9d5b99b
docs: add SLM architecture and guides documentation
JustAGhosT Mar 15, 2026
cab0205
docs: add architecture documentation for all systems and SLM manageme…
JustAGhosT Mar 15, 2026
ef8619a
docs: update architecture docs with detailed SLM implementation patterns
JustAGhosT Mar 15, 2026
c7d997a
I'll analyze this pull request which reorganizes and expands the arch…
JustAGhosT Mar 15, 2026
4c5e506
ci: add reusable deploy-environment workflow
JustAGhosT Mar 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/actions/import-container-app/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ inputs:
description: Project name component of the Container App name (TF_VAR_projname)
env:
required: true
description: Environment name (dev|uat|prod)
description: Environment name (dev|staging|prod)
location_short:
required: true
description: Short location code (TF_VAR_location_short)
Expand Down
6 changes: 3 additions & 3 deletions .github/pull_request_template.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@
- [ ] No environment/config changes required
- [ ] Environment/config changes required (describe below)

## UAT Toggle (PRs to `main`)
## Staging Toggle (PRs to `main`)

- Add label `run-uat` to this PR to enable UAT deployment (`deploy-uat`).
- Remove label `run-uat` to skip UAT deployment.
- Add label `run-staging` to this PR to enable staging deployment (`deploy-staging`).
- Remove label `run-staging` to skip staging deployment.

## Risk / Rollback

Expand Down
24 changes: 12 additions & 12 deletions .github/workflows/deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,23 +33,23 @@ env:

jobs:
plan:
# PR into dev → dev | PR into main + label 'run-uat' → uat | Push to main/workflow_dispatch → prod
# PR into dev → dev | PR into main + label 'run-staging' → staging | Push to main/workflow_dispatch → prod
# Skip plan for PRs from forks (no repo secrets; avoids AADSTS700213)
# Runtime UAT toggle: add PR label 'run-uat' to enable UAT on PRs into main.
# Runtime UAT toggle: add PR label 'run-staging' to enable UAT on PRs into main.
if: |
(github.event_name != 'pull_request' || github.event.pull_request.head.repo.fork == false) &&
(
(github.event_name == 'push' && github.ref == 'refs/heads/main') ||
(github.event_name == 'workflow_dispatch') ||
(github.event_name == 'pull_request' && github.event.pull_request.base.ref == 'dev') ||
(github.event_name == 'pull_request' && github.event.pull_request.base.ref == 'main' && contains(join(github.event.pull_request.labels.*.name, ','), 'run-uat'))
(github.event_name == 'pull_request' && github.event.pull_request.base.ref == 'main' && contains(join(github.event.pull_request.labels.*.name, ','), 'run-staging'))
)
name: Plan ${{ matrix.environment }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
environment: ${{ (github.event_name == 'workflow_dispatch' && fromJSON('["prod"]')) || (github.event_name == 'push' && github.ref == 'refs/heads/main' && fromJSON('["prod"]')) || (github.event_name == 'pull_request' && github.event.pull_request.base.ref == 'dev' && fromJSON('["dev"]')) || (github.event_name == 'pull_request' && github.event.pull_request.base.ref == 'main' && contains(join(github.event.pull_request.labels.*.name, ','), 'run-uat') && fromJSON('["uat"]')) || fromJSON('["prod"]') }}
environment: ${{ (github.event_name == 'workflow_dispatch' && fromJSON('["prod"]')) || (github.event_name == 'push' && github.ref == 'refs/heads/main' && fromJSON('["prod"]')) || (github.event_name == 'pull_request' && github.event.pull_request.base.ref == 'dev' && fromJSON('["dev"]')) || (github.event_name == 'pull_request' && github.event.pull_request.base.ref == 'main' && contains(join(github.event.pull_request.labels.*.name, ','), 'run-staging') && fromJSON('["staging"]')) || fromJSON('["prod"]') }}
environment: ${{ matrix.environment }}
defaults:
run:
Expand Down Expand Up @@ -324,18 +324,18 @@ jobs:

jq -e '.enabled == true' /tmp/selection-get.json > /dev/null

Comment on lines +154 to +170
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Mar 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🌐 Web query:

Does GitHub Actions allow environment:on a job that calls a reusable workflow viauses:? If not, what is the documented way to bind the called job to dev, staging, or prod?

💡 Result:

No. A job that calls a reusable workflow via jobs.<job_id>.uses: can only use a limited set of keywords, and environment is not one of them (so environment: on the calling job is not supported). [1]

Documented way to bind to dev / staging / prod:

  1. Pass the target environment name as an input from the caller (often via a matrix of dev/stage/prod). [2]
  2. Set environment: inside the reusable workflow’s job, using that input (the environment name can be an expression and can use the inputs context). [3]
  3. This is also why environment secrets can’t be passed from the caller via workflow_call (it doesn’t support environment), so the environment needs to be selected in the called workflow’s job. [2]

Example:

Caller:

jobs:
  deploy:
    strategy:
      matrix:
        target: [dev, staging, prod]
    uses: org/repo/.github/workflows/deploy.yml@main
    with:
      target: ${{ matrix.target }}

[2]

Reusable workflow (deploy.yml):

on:
  workflow_call:
    inputs:
      target:
        required: true
        type: string

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment:
      name: ${{ inputs.target }}
    steps:
      - run: echo "Deploying to ${{ inputs.target }}"

[3]

Sources: GitHub Docs on reusable-workflow keyword limits [1], reusable workflows + environment-secret note + matrix example [2], and workflow syntax for jobs.<job_id>.environment (name can use inputs) [3].

🏁 Script executed:

cd .github/workflows && head -n 220 deploy.yaml | tail -n 70

Repository: phoenixvc/ai-gateway

Length of output: 2993

Remove environment: from all three reusable-workflow caller jobs.

Jobs that call a reusable workflow via uses: cannot use the environment: key. Currently, deploy-dev, deploy-staging, and deploy-prod all specify environment: at the job level alongside uses:, which violates GitHub Actions job schema. Move environment selection into the called workflow (deploy-environment.yaml) by setting environment: there with the input value (e.g., environment: ${{ inputs.env_name }}). The caller jobs already pass the environment name correctly via env_name input and only need the invalid environment: key removed.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/deploy.yaml around lines 154 - 170, The job-level
environment key must be removed from the reusable-workflow caller jobs (e.g.,
deploy-dev, deploy-staging, deploy-prod) because jobs that use `uses:` cannot
include `environment:`; update each caller to only pass the env via the
`env_name` input (already present) and then add `environment: ${{
inputs.env_name }}` inside the called workflow `deploy-environment.yaml` (use
the `inputs.env_name` value in that workflow's job definition) so the
environment selection is handled by the reusable workflow rather than the
callers.

deploy-uat:
name: Deploy uat
deploy-staging:
name: Deploy staging
needs: plan
runs-on: ubuntu-latest
if: github.event_name == 'pull_request' && github.event.pull_request.base.ref == 'main' && contains(join(github.event.pull_request.labels.*.name, ','), 'run-uat')
environment: uat
if: github.event_name == 'pull_request' && github.event.pull_request.base.ref == 'main' && contains(join(github.event.pull_request.labels.*.name, ','), 'run-staging')
environment: staging
defaults:
run:
working-directory: infra/env/uat
working-directory: infra/env/staging

env:
TF_VAR_env: "uat"
TF_VAR_env: "staging"
TF_VAR_projname: "aigateway"
TF_VAR_location: "southafricanorth"
TF_VAR_location_short: "san"
Expand Down Expand Up @@ -407,7 +407,7 @@ jobs:
-backend-config="resource_group_name=${TF_BACKEND_RG}" \
-backend-config="storage_account_name=${TF_BACKEND_SA}" \
-backend-config="container_name=${TF_BACKEND_CONTAINER}" \
-backend-config="key=uat.terraform.tfstate"
-backend-config="key=staging.terraform.tfstate"

- name: Import existing Container App into Terraform state
uses: ./.github/actions/import-container-app
Expand All @@ -416,7 +416,7 @@ jobs:
env: ${{ env.TF_VAR_env }}
location_short: ${{ env.TF_VAR_location_short }}
subscription_id: ${{ env.AZURE_SUBSCRIPTION_ID }}
terraform_working_directory: infra/env/uat
terraform_working_directory: infra/env/staging

- name: Terraform Plan
run: |
Expand Down
18 changes: 9 additions & 9 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Creates the shared resource group, storage account, and container for Terraform

### 2. Add GitHub secrets

Add these secrets to each GitHub **Environment** (dev, uat, prod): **Settings → Environments → &lt;env&gt; → Environment secrets**.
Add these secrets to each GitHub **Environment** (dev, staging, prod): **Settings → Environments → &lt;env&gt; → Environment secrets**.

| Secret | Description | Example |
| ----------------------- | --------------------------------- | --------------------------------------------- |
Expand All @@ -53,16 +53,16 @@ Bootstrap prints these values. For local runs, copy `infra/.env.local.example` t
**Bash:**

```bash
./infra/scripts/terraform-init.sh dev # or uat, prod
./infra/scripts/terraform-init.sh dev # or staging, prod
```

**PowerShell:**

```powershell
.\infra\scripts\terraform-init.ps1 -Env dev # or uat, prod
.\infra\scripts\terraform-init.ps1 -Env dev # or staging, prod
```

Valid environments: `dev`, `uat`, `prod`.
Valid environments: `dev`, `staging`, `prod`.

### 4. Plan and apply

Expand All @@ -74,11 +74,11 @@ terraform apply

## Environments

| Env | Purpose |
| ---- | --------------- |
| dev | Development |
| uat | User acceptance |
| prod | Production |
| Env | Purpose |
| ------- | ----------- |
| dev | Development |
| staging | Staging |
| prod | Production |

## CI/CD

Expand Down
2 changes: 1 addition & 1 deletion dashboard/app.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ function escHtml(s) {

function deriveEnv(url) {
if (!url) return null;
const m = url.match(/pvc-(dev|uat|prod)-/);
const m = url.match(/pvc-(dev|staging|prod)-/);
return m ? m[1] : null;
}

Expand Down
20 changes: 10 additions & 10 deletions docs/AZURE_OIDC_SETUP.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ If you see:
Error: AADSTS700213: No matching federated identity record found for presented assertion subject 'repo:phoenixvc/ai-gateway:environment:dev'
```

**Cause:** The workflow uses `environment: dev` (and uat/prod), so the OIDC subject is `repo:org/repo:environment:dev`. Azure must have a federated credential with that exact subject.
**Cause:** The workflow uses `environment: dev` (and staging/prod), so the OIDC subject is `repo:org/repo:environment:dev`. Azure must have a federated credential with that exact subject.

### Fix: Add environment federated credentials

Expand All @@ -32,21 +32,21 @@ az ad app list --display-name pvc-shared-github-actions-oidc --query "[0].appId"

1. Go to **Azure Portal** → **Microsoft Entra ID** → **App registrations** → your app (e.g. `pvc-shared-github-actions-oidc`)
2. **Certificates & secrets** → **Federated credentials** → **Add credential**
3. For each environment (dev, uat, prod), add:
3. For each environment (dev, staging, prod), add:
- **Federated credential scenario:** GitHub Actions deploying Azure resources
- **Organization:** phoenixvc
- **Repository:** ai-gateway
- **Entity type:** Environment
- **Environment name:** dev (or uat, prod)
- **Name:** github-actions-dev (or uat, prod)
- **Environment name:** dev (or staging, prod)
- **Name:** github-actions-dev (or staging, prod)

### Subject formats

| Workflow config | OIDC subject |
| -------------------- | ----------------------------------------------- |
| `environment: dev` | `repo:phoenixvc/ai-gateway:environment:dev` |
| `environment: uat` | `repo:phoenixvc/ai-gateway:environment:uat` |
| `environment: prod` | `repo:phoenixvc/ai-gateway:environment:prod` |
| Branch only (no env) | `repo:phoenixvc/ai-gateway:ref:refs/heads/main` |
| Workflow config | OIDC subject |
| ---------------------- | ----------------------------------------------- |
| `environment: dev` | `repo:phoenixvc/ai-gateway:environment:dev` |
| `environment: staging` | `repo:phoenixvc/ai-gateway:environment:staging` |
| `environment: prod` | `repo:phoenixvc/ai-gateway:environment:prod` |
| Branch only (no env) | `repo:phoenixvc/ai-gateway:ref:refs/heads/main` |

The federated credential **Subject** in Azure must match exactly.
10 changes: 5 additions & 5 deletions docs/CI_CD.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@ This document describes the current GitHub Actions deployment behavior for `ai-g

- PRs from forks are skipped for deployment-related jobs (no repo secrets).
- PRs targeting `dev` run `plan` + `deploy-dev`.
- PRs targeting `main` run UAT only when the PR has label `run-uat`.
- PRs targeting `main` run staging only when the PR has label `run-staging`.
- Push to `main` and `workflow_dispatch` run `plan` + `deploy-prod`.

## Runtime UAT toggle
## Runtime staging toggle

UAT deployment for PRs to `main` is controlled by PR label:
staging deployment for PRs to `main` is controlled by PR label:

- Add label `run-uat` to enable `deploy-uat` for that PR.
- Remove label `run-uat` to disable UAT for that PR.
- Add label `run-staging` to enable `deploy-staging` for that PR.
- Remove label `run-staging` to disable staging for that PR.

## Smoke test behavior

Expand Down
8 changes: 4 additions & 4 deletions docs/PRD.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Roo/Qoder currently struggles with Azure model/operation mismatches. A gateway n
2. Support:
- `POST /v1/responses` routed to Azure **Responses** endpoint for configurable model (default: `gpt-5.3-codex`).
- `POST /v1/embeddings` routed to Azure embeddings deployment.
3. Enable **multiple environments** (dev/uat/prod) and **multiple downstream projects**.
3. Enable **multiple environments** (dev/staging/prod) and **multiple downstream projects**.
4. Infrastructure managed with **Terraform**.
5. CI/CD via **GitHub Actions** using **Azure OIDC** (no long-lived secrets).
6. “Get it working” first; hardening follows.
Expand All @@ -29,7 +29,7 @@ Roo/Qoder currently struggles with Azure model/operation mismatches. A gateway n
## 3) Environments

- `dev`
- `uat`
- `staging`
- `prod`

Each env is independently deployable.
Expand Down Expand Up @@ -150,7 +150,7 @@ Gateway must expose:
- `docs/` - Documentation.
- `infra/`
- `modules/aigateway_aca` - Core Terraform module.
- `env/dev|uat|prod` - Environment-specific configurations.
- `env/dev|staging|prod` - Environment-specific configurations.
- `.github/workflows/` - CI/CD pipelines.
- `scripts/` - Helper scripts (bootstrap).

Expand All @@ -161,7 +161,7 @@ Gateway must expose:
- **Phase 1: Terraform & CI/CD**
- Terraform defines infra.
- GitHub Actions deploys using Azure OIDC.
- Dev auto-apply on merge; UAT/Prod gated with environment approvals.
- Dev auto-apply on merge; Staging/Prod gated with environment approvals.

## 10) Acceptance criteria

Expand Down
18 changes: 9 additions & 9 deletions docs/SECRETS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,17 @@

Copy this checklist when setting up environments for this repo.

For workflow behavior (dev/uat/prod triggers, PR label `run-uat`, and smoke-test flow), see [CI_CD.md](CI_CD.md).
For workflow behavior (dev/staging/prod triggers, PR label `run-staging`, and smoke-test flow), see [CI_CD.md](CI_CD.md).

## Where to add secrets

Add these as **Environment secrets** in GitHub:

- **Settings → Environments → dev → Environment secrets**
- **Settings → Environments → uat → Environment secrets**
- **Settings → Environments → staging → Environment secrets**
- **Settings → Environments → prod → Environment secrets**

> This workflow is environment-based (`environment: dev|uat|prod`), so each environment should have the full secret set.
> This workflow is environment-based (`environment: dev|staging|prod`), so each environment should have the full secret set.

## Required secrets (all environments)

Expand Down Expand Up @@ -53,7 +53,7 @@ When `STATE_SERVICE_CONTAINER_IMAGE` is set (state-service enabled), set this se

## Copy/paste template

Use this block as a setup checklist when creating/updating `dev`, `uat`, and `prod`:
Use this block as a setup checklist when creating/updating `dev`, `staging`, and `prod`:

```text
AZURE_CLIENT_ID=<GUID>
Expand Down Expand Up @@ -82,13 +82,13 @@ STATE_SERVICE_REGISTRY_PASSWORD=<ghcr-read-packages-token> # required for priv
- [ ] `AIGATEWAY_KEY` matches the key expected by the deployed gateway.
- [ ] OIDC federated credentials exist for each environment subject:
- `repo:phoenixvc/ai-gateway:environment:dev`
- `repo:phoenixvc/ai-gateway:environment:uat`
- `repo:phoenixvc/ai-gateway:environment:staging`
- `repo:phoenixvc/ai-gateway:environment:prod`

## Runtime UAT toggle
## Runtime staging toggle

- UAT deploy on PRs into `main` is controlled by PR label `run-uat`.
- Add label `run-uat` to enable `deploy-uat` for that PR.
- Remove label `run-uat` to skip UAT for that PR.
- Staging deploy on PRs into `main` is controlled by PR label `run-staging`.
- Add label `run-staging` to enable `deploy-staging` for that PR.
- Remove label `run-staging` to skip staging for that PR.

For OIDC troubleshooting, see [AZURE_OIDC_SETUP.md](AZURE_OIDC_SETUP.md).
10 changes: 5 additions & 5 deletions docs/Terraform_Blueprint.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
This canvas includes a working Terraform scaffold:

- `infra/modules/aigateway_aca`
- `infra/env/dev|uat|prod`
- `infra/env/dev|staging|prod`
- Shared state configured via `terraform init -backend-config=...` in GitHub Actions

> Notes:
Expand All @@ -27,7 +27,7 @@ infra/
main.tf
variables.tf
terraform.tfvars
uat/
staging/
main.tf
variables.tf
terraform.tfvars
Expand All @@ -44,7 +44,7 @@ infra/
```hcl
variable "env" {
type = string
description = "Environment name (dev|uat|prod)"
description = "Environment name (dev|staging|prod)"
}

variable "projname" {
Expand Down Expand Up @@ -343,7 +343,7 @@ output "key_vault_name" {

## 5) Env stacks

### 5.1 `infra/env/dev/variables.tf` (repeat for uat/prod)
### 5.1 `infra/env/dev/variables.tf` (repeat for staging/prod)

```hcl
variable "env" { type = string }
Expand Down Expand Up @@ -441,7 +441,7 @@ tags = {
}
```

Repeat the env folders for `uat` and `prod`, changing only `env` and tags.
Repeat the env folders for `staging` and `prod`, changing only `env` and tags.

---

Expand Down
27 changes: 14 additions & 13 deletions docs/planning/request_to_token_attribution.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,11 @@ LiteLLM's OTEL callback automatically emits spans with:

### Phase 2: Correlation ID Propagation

**Status: In Progress**
**Status: ✅ Done**

Correlation IDs flow through the system in two ways:

**Method A: Via Request Metadata (Recommended)**
**Method A: Via Request Metadata (Implemented)**
Pass correlation IDs in the request body `metadata` field:

```json
Expand Down Expand Up @@ -207,31 +207,32 @@ _Note: Method B requires additional LiteLLM configuration or middleware._

## Acceptance Criteria

| Criterion | Status | Notes |
| -------------------------------------------- | ---------- | ----------------------------------------- |
| 100% of LLM calls emit token telemetry | ✅ Done | Via OTEL callback |
| 100% include workflow + stage | ⚠️ Partial | Requires upstream to pass metadata |
| Support KQL joins by operation_Id/request_id | ✅ Done | OTEL spans include metadata |
| Request-completion rollup totals | 🔜 Future | Requires Phase 3 (downstream aggregation) |
| Criterion | Status | Notes |
| -------------------------------------------- | --------- | ----------------------------------------- |
| 100% of LLM calls emit token telemetry | ✅ Done | Via OTEL callback |
| 100% include workflow + stage | ✅ Done | Metadata passed through to OTEL spans |
| Support KQL joins by operation_Id/request_id | ✅ Done | OTEL spans include metadata |
| Request-completion rollup totals | 🔜 Future | Requires Phase 3 (downstream aggregation) |

## Dependencies

- cognitive-mesh: Must pass correlation headers to gateway
- cognitive-mesh: Must pass correlation metadata to gateway
- pvc-costops-analytics: Must create KQL queries for new event shape
- infra: May need custom LiteLLM container image or OTEL collector
- infra: Application Insights added for trace storage

## Action Items

### Completed

1. ✅ ai-gateway: Add OTEL callback for token telemetry (Phase 1)
2. ✅ ai-gateway: Document correlation ID requirements (Phase 2)
3. ✅ ai-gateway: Add Application Insights for trace storage (Phase 1b)

### Pending

3. cognitive-mesh: Pass correlation IDs in request metadata
4. pvc-costops-analytics: Create KQL queries for OTEL span joins
5. pvc-costops-analytics: Implement request rollup aggregation (Phase 3)
4. cognitive-mesh: Pass correlation metadata in request body
5. pvc-costops-analytics: Create KQL queries for OTEL span joins
6. pvc-costops-analytics: Implement request rollup aggregation (Phase 3)

---

Expand Down
Loading
Loading