Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v7

- name: Enable Corepack
run: corepack enable
Expand All @@ -59,7 +59,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v7

- name: Enable Corepack
run: corepack enable
Expand All @@ -82,7 +82,7 @@ jobs:
needs: [lint, typecheck]
steps:
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@v7

- name: Enable Corepack
run: corepack enable
Expand Down
30 changes: 14 additions & 16 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ name: Deploy - Frontend to Vercel and Artifacts
on:
push:
branches: [main]
workflow_dispatch:

permissions:
contents: read
Expand Down Expand Up @@ -59,7 +60,7 @@ jobs:
retention-days: 30

- name: Deploy to Vercel
uses: vercel/action@v5
uses: amondnet/vercel-action@v42
with:
vercel-token: ${{ secrets.VERCEL_TOKEN }}
vercel-org-id: ${{ secrets.VERCEL_ORG_ID }}
Expand All @@ -83,51 +84,51 @@ jobs:
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
targets: wasm32-unknown-unknown
targets: wasm32v1-none

- name: Cache cargo registry
uses: actions/cache@v6
with:
path: \~/.cargo/registry
key: \( {{ runner.os }}-cargo-registry- \){{ hashFiles('**/Cargo.lock') }}
path: ~/.cargo/registry
key: ${{ runner.os }}-cargo-registry-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-registry-

- name: Cache cargo index
uses: actions/cache@v6
with:
path: \~/.cargo/git
key: \( {{ runner.os }}-cargo-git- \){{ hashFiles('**/Cargo.lock') }}
path: ~/.cargo/git
key: ${{ runner.os }}-cargo-git-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-git-

- name: Cache cargo build
uses: actions/cache@v6
with:
path: target
key: \( {{ runner.os }}-cargo-build-target- \){{ hashFiles('**/Cargo.lock') }}
key: ${{ runner.os }}-cargo-build-target-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-build-target-

- name: Build contract
run: cargo build --release -p prompt-hash --target wasm32-unknown-unknown
run: cargo build --release -p prompt-hash --target wasm32v1-none
env:
CARGO_TERM_COLOR: always

- name: List WASM artifacts
run: ls -lah target/wasm32-unknown-unknown/release/*.wasm
run: ls -lah target/wasm32v1-none/release/*.wasm

- name: Generate WASM checksums
run: |
cd target/wasm32-unknown-unknown/release
cd target/wasm32v1-none/release
sha256sum *.wasm > ../../../wasm-checksums.txt
sha256sum ../../../wasm-checksums.txt > ../../../wasm-checksums.txt.sha256

- name: Upload WASM artifacts
uses: actions/upload-artifact@v7
with:
name: contract-wasm-${{ github.sha }}
path: target/wasm32-unknown-unknown/release/*.wasm
path: target/wasm32v1-none/release/*.wasm
retention-days: 30
if-no-files-found: warn

Expand Down Expand Up @@ -240,18 +241,15 @@ jobs:
- name: Install dependencies
run: yarn install --immutable

- name: Install CycloneDX tooling
run: npm install -g @cyclonedx/cyclonedx-npm

- name: Generate frontend SBOM (CycloneDX)
run: |
mkdir -p sbom
cyclonedx-npm --output-format JSON --output-file sbom/frontend-sbom.json
yarn dlx -q @cyclonedx/yarn-plugin-cyclonedx --output-format JSON --output-file sbom/frontend-sbom.json

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
targets: wasm32-unknown-unknown
targets: wasm32v1-none

- name: Install cargo-cyclonedx
run: cargo install cargo-cyclonedx
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/frontend.yml
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ jobs:

- name: Upload visual regression artifacts
if: always()
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: playwright-visual-regression
path: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
fetch-depth: 0

- name: Setup Node.js
uses: actions/setup-node@v4
uses: actions/setup-node@v7
with:
node-version: "22"
cache: "npm"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/soroban-schema-validation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:
- name: Set up Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
toolchain: 1.89.0
toolchain: 1.91.1
targets: wasm32v1-none

- name: Set up Node.js
Expand Down
Binary file modified .yarn/install-state.gz
Binary file not shown.
2 changes: 2 additions & 0 deletions api/bundles/unlock.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,7 @@ describe("bundle unlock API abuse protection & lockout", () => {
bundleId,
address: buyer.publicKey(),
signedMessage: wrongSignature,
captchaToken: i >= 3 ? "test-captcha-token-valid" : undefined,
});
expect(statusCode).toBe(401);
expect(responseData.code).toBe(ErrorCode.INVALID_SIGNATURE);
Expand All @@ -188,6 +189,7 @@ describe("bundle unlock API abuse protection & lockout", () => {
bundleId,
address: buyer.publicKey(),
signedMessage: wrongSignature,
captchaToken: "test-captcha-token-valid",
});

expect(statusCode).toBe(423);
Expand Down
90 changes: 90 additions & 0 deletions api/health.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
// @vitest-environment node

import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";

const { connectDbMock } = vi.hoisted(() => ({
connectDbMock: vi.fn().mockResolvedValue(undefined),
}));

vi.mock("../server/src/db/connectDb", () => ({
default: connectDbMock,
}));

vi.mock("../server/src/models/IndexerState", () => ({
IndexerState: {
findOne: vi.fn().mockResolvedValue({ lastIndexedLedger: 12345 }),
},
}));

import handler from "./health";

function makeReq(method: string = "GET") {
return {
method,
headers: {},
url: "/api/health",
socket: { remoteAddress: "127.0.0.1" },
};
}

function makeRes() {
const res = {
statusCode: 0 as number,
body: undefined as any,
writableEnded: false,
status(code: number) {
res.statusCode = code;
return res;
},
json(data: any) {
res.body = data;
res.writableEnded = true;
return res;
},
setHeader(_name: string, _value: string) {
return res;
},
};
return res;
}

describe("GET /api/health", () => {
const origContractId = process.env.PUBLIC_PROMPT_HASH_CONTRACT_ID;

beforeEach(() => {
process.env.PUBLIC_PROMPT_HASH_CONTRACT_ID =
"CAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
vi.clearAllMocks();
});

afterEach(() => {
process.env.PUBLIC_PROMPT_HASH_CONTRACT_ID = origContractId;
vi.restoreAllMocks();
});

it("returns 200 and healthy status when contract and RPC are reachable", async () => {
vi.spyOn(globalThis, "fetch").mockImplementation(async () => {
return new Response(JSON.stringify({ result: { status: "healthy" } }), { status: 200 });
});

const res = makeRes();
await handler(makeReq("GET"), res);

expect(res.statusCode).toBe(200);
expect(res.body.status).toBe("ok");
expect(res.body.rpc.status).toBe("up");
expect(res.body.contractConfig.configured).toBe(true);
expect(res.body.indexer.lastProcessedLedger).toBe(12345);
});

it("returns 503 degraded when RPC is unreachable", async () => {
vi.spyOn(globalThis, "fetch").mockRejectedValue(new Error("network error"));

const res = makeRes();
await handler(makeReq("GET"), res);

expect(res.statusCode).toBe(503);
expect(res.body.status).toBe("degraded");
expect(res.body.rpc.status).toBe("down");
});
});
6 changes: 4 additions & 2 deletions api/sitemap.test.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
import { beforeEach, describe, expect, it, vi } from "vitest";

const connectDbMock = vi.fn();
const promptFindMock = vi.fn();
const { connectDbMock, promptFindMock } = vi.hoisted(() => ({
connectDbMock: vi.fn(),
promptFindMock: vi.fn(),
}));

vi.mock("../src/lib/observability/wrapper", () => ({
withObservability: (handler: unknown) => handler,
Expand Down
9 changes: 6 additions & 3 deletions api/sitemap.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@ import { withObservability } from "../src/lib/observability/wrapper";
import connectDb from "../server/src/db/connectDb";
import Prompt from "../server/src/models/Prompt";

const APP_URL = process.env.APP_URL ?? "https://prompthash.io";
function getAppUrl(): string {
return process.env.APP_URL ?? "https://prompthash.io";
}

function escapeXml(value: string): string {
return value
Expand Down Expand Up @@ -30,7 +32,8 @@ async function handler(req: any, res: any) {
return;
}

const urls = getBaseUrls().map((path) => `${APP_URL}${path}`);
const appUrl = getAppUrl();
const urls = getBaseUrls().map((path) => `${appUrl}${path}`);
let error: string | null = null;

try {
Expand All @@ -43,7 +46,7 @@ async function handler(req: any, res: any) {
const promptId = String(prompt._id);
const lastMod = toIsoDate(prompt.updatedAt);
return {
loc: `${APP_URL}/browse?promptId=${encodeURIComponent(promptId)}`,
loc: `${appUrl}/browse?promptId=${encodeURIComponent(promptId)}`,
lastMod,
};
});
Expand Down
83 changes: 83 additions & 0 deletions docs/operations/incidents/2026-08-31-887acea-deploy-failure.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Incident Post-Mortem: 2026-08-31 Deploy Pipeline Breakage & Auto-Rollback (887acea & f9b966e)

**Severity**: SEV-1 (Critical)
**Status**: Fix Prepared Locally / Deployment Pending Remote Push
**Affected Workflow**: `.github/workflows/deploy.yml` (Deploy - Frontend to Vercel and Artifacts)
**Production Host**: `https://prompt-hash-stellar.vercel.app`

---

## 1. Incident Summary

Following a series of merges to `main` (beginning with `f9b966e` and subsequent PR merges `887acea` and `f31cb60`), the production deployment workflow (**Deploy - Frontend to Vercel and Artifacts**) repeatedly failed on every commit. Each failure triggered `.github/workflows/auto-rollback.yml` to execute an automated rollback on Vercel.

Root cause analysis confirmed the failures were caused by:
1. An invalid action reference (`vercel/action@v5`) in the `deploy-frontend` job.
2. An outdated Rust toolchain (`1.89.0`) incompatible with `soroban-sdk@26.1.1` in `contract-build`, along with the deprecated `wasm32-unknown-unknown` target.
3. Invocation of `cyclonedx-npm` against a Yarn Berry dependency tree in `generate-sbom`.
4. Pre-existing TypeScript syntax errors in `src/components/BuyerLibrary.tsx` on `main`.

A previous fix branch (`fix/sev1-deploy-887acea`) had addressed these issues locally on the fork `Cent-Dave/prompt-mint`, but was never merged into `main`. The fix branch has now been fast-forward merged into local `main`, with WASM target standardization (`wasm32v1-none`) applied to `deploy.yml`.

---

## 2. Incident Timeline (UTC)

*All timestamps sourced directly from GitHub Actions workflow runs and git commit metadata.*

| Timestamp (UTC) | Source | Event Description |
| :--- | :--- | :--- |
| **2026-08-31 07:51:15Z** | Git Commit `f9b966e` | Commit authored on `main`. |
| **2026-08-31 07:51:18Z** | GitHub Actions Run `33370214738` | Deploy workflow triggered on commit `f9b966e`. |
| **2026-08-31 07:51:55Z** | GitHub Actions Run `33370214738` | Deploy workflow failed (`deploy-frontend`, `contract-build`, `generate-sbom` failed). |
| **2026-08-31 07:51:57Z** | GitHub Actions Run `33370261029` | Auto-rollback workflow triggered and executed. |
| **2026-08-31 07:55:02Z** | Git Commit `887acea` | Commit authored on `main`. |
| **2026-08-31 07:55:05Z** | GitHub Actions Run `33370492695` | Deploy workflow triggered on commit `887acea`. |
| **2026-08-31 07:55:50Z** | GitHub Actions Run `33370492695` | Deploy workflow failed. |
| **2026-08-31 07:55:31Z** | Git Commit `f31cb60` | Commit authored on `main`. |
| **2026-08-31 07:55:35Z** | GitHub Actions Run `33370529587` | Deploy workflow triggered on commit `f31cb60`. |
| **2026-08-31 07:56:46Z** | GitHub Actions Run `33370529587` | Deploy workflow failed. |
| **2026-08-31 09:33:14Z** | Git Commit `9ae4066` | Initial remediation commit on branch `fix/sev1-deploy-887acea`. |
| **2026-08-31 11:22:54Z** | Git Commit `8bc642f` | Corrected broken `vercel/action` to `amondnet/vercel-action@v42`. |
| **2026-08-31 11:43:35Z** | Git Commit `f11d363` | Bumped Rust toolchain to `1.91.1` for `soroban-sdk@26.1.1` compatibility. |
| **2026-08-31 12:24:21Z** | Git Commit `87dd5cf` | Switched to yarn-native `cyclonedx` plugin for SBOM generation. |
| **2026-08-31 12:27:39Z** | Git Commit `ad9cc30` | Reverted unnecessary action version downgrades in `deploy.yml`. |
| **2026-08-31 12:38:03Z** | Git Commit `c400e73` | Reverted unnecessary action version downgrades across remaining workflows. |
| **2026-08-31 12:46:47Z** | Git Commit `22193ca` | Bumped schema validation Rust toolchain to `1.91.1` (branch remained unmerged). |
| **2026-08-31 20:34:59Z (approx)** | Local Operation | Fast-forward merged `fix/sev1-deploy-887acea` into local `main`. |
| **2026-08-31 21:29:23Z** | Git Commit `a5cec5a` | Standardized `deploy.yml` target to `wasm32v1-none` and recorded incident post-mortem. |
| **2026-08-31 21:36:59Z** | Live HTTP Probe | Current production endpoints (`https://prompt-hash-stellar.vercel.app/api/health` and `/api/status`) probed, confirming 500 error on pre-fix deployment. Recovery pending deployment. |

---

## 3. Root Cause Analysis (5 Whys)

1. **Why did the deployment pipeline fail on `f9b966e`, `887acea`, and `f31cb60`?**
Three jobs in `.github/workflows/deploy.yml` (`deploy-frontend`, `contract-build`, and `generate-sbom`) crashed.

2. **Why did each job fail?**
- `deploy-frontend`: Referenced non-existent action `vercel/action@v5`.
- `contract-build`: `rust-toolchain.toml` had `1.89.0`, incompatible with `soroban-sdk@26.1.1` (which requires Rust `1.91.0+`).
- `generate-sbom`: Ran `cyclonedx-npm` which failed on the Yarn Berry lockfile/tree.

3. **Why did the issue recur on `main` after fixes were developed?**
Branch `fix/sev1-deploy-887acea` contained the complete fixes, but was not merged into upstream `main`. Subsequent PRs merged into `main` continued to trigger the unpatched workflow.

4. **Why did `soroban-sdk@26.1.1` panic under Rust 1.91.1 with `wasm32-unknown-unknown`?**
Rust 1.84+ enables WebAssembly feature extensions under `wasm32-unknown-unknown` that are unsupported by the Soroban runtime; Soroban SDK 26+ enforces `wasm32v1-none`.

5. **Why was the unmerged state not caught automatically?**
Pull requests did not run a dry-run of the deployment workflow prior to merge.

---

## 4. Corrective & Preventative Actions

| Action Item | Type | Owner | Status |
| :--- | :--- | :--- | :--- |
| Fast-forward merge `fix/sev1-deploy-887acea` to `main` | Remediation | On-Call | Completed locally |
| Update `deploy.yml` targets to `wasm32v1-none` | Fix | On-Call | Completed locally |
| Fix `BuyerLibrary.tsx` JSX syntax and missing imports | Code Quality | Frontend Lead | Completed locally |
| Add `/api/health` unit test coverage in `api/health.test.ts` | Test | Backend Lead | Completed locally |
| Deploy merged commit to Vercel and verify production `/api/health` | Validation | On-Call | Pending Remote Push & Deploy |

4 changes: 2 additions & 2 deletions rust-toolchain.toml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
[toolchain]
channel = "1.89.0"
targets = ["wasm32v1-none"]
channel = "1.91.1"
targets = ["wasm32v1-none", "wasm32-unknown-unknown"]
Loading