Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
6f63ef1
chore: create v0.8 domain-driven module directory structure
ShixiangWang Jun 13, 2026
cea0d56
feat: add all v0.8 domain types (workflow, execution, auth, ai, colla…
ShixiangWang Jun 13, 2026
fcf69a2
feat: implement workflow domain service, handlers, and data discovery
ShixiangWang Jun 13, 2026
45696b9
feat: implement execution domain with diagnostics engine, service, an…
ShixiangWang Jun 13, 2026
36127b6
feat: implement auth domain, observability domain, and server router …
ShixiangWang Jun 13, 2026
a600e9b
feat: complete Phase 1 deterministic core — all domains + server asse…
ShixiangWang Jun 13, 2026
64e76b1
feat: add license header middleware and AI domain stubs
ShixiangWang Jun 13, 2026
19e1f71
feat: add collaboration domain stubs for Phase 3 endpoints
ShixiangWang Jun 13, 2026
025f648
feat: implement AI translation layer — copilot, service, handlers, se…
ShixiangWang Jun 13, 2026
353e9c9
feat: implement collaboration primitives — fork, share, import with h…
ShixiangWang Jun 13, 2026
a21fd52
docs: update documentation for v0.8 release
ShixiangWang Jun 13, 2026
f4aaa50
chore: fix formatting in collaboration mod.rs
ShixiangWang Jun 13, 2026
c79dda4
feat: complete v0.8 — HPC integration, three-mode CLI, frontend compa…
ShixiangWang Jun 13, 2026
91b66c5
feat: complete all spec requirements — license footer, CLI version, D…
ShixiangWang Jun 13, 2026
b9bfbc1
docs: update spec license table to reflect v0.8 implementation status…
ShixiangWang Jun 13, 2026
dd39401
feat: complete v0.8 release — domain-driven architecture, AI integrat…
ShixiangWang Jun 13, 2026
319ab27
feat: add API docs page, openapi endpoint, and docs navigation
ShixiangWang Jun 13, 2026
72f207b
fix: AI optimize now accepts toml_content directly in request
ShixiangWang Jun 13, 2026
73ae330
feat: professional web UI with header, sidebar, footer, toast, skelet…
ShixiangWang Jun 13, 2026
78ef81b
feat: Execution Monitor, Results Browser, Loading States, Empty States
ShixiangWang Jun 13, 2026
7354946
fix: responsive sidebar collapse, CSS hash update
ShixiangWang Jun 13, 2026
6be4c86
feat: complete Phase 1 deterministic core implementation and verifica…
ShixiangWang Jun 13, 2026
fa5cf4f
feat: complete Phase 2 AI Translation Layer and Phase 3 Collaboration
ShixiangWang Jun 13, 2026
be8d409
chore: cargo fmt fix for Phase 3 integration tests
ShixiangWang Jun 13, 2026
83575c2
feat: complete Phase 3 Collaboration & Multi-Mode
ShixiangWang Jun 13, 2026
e80ebb2
feat: complete Goal 8 — auth middleware gating + PBS/LSF/SGE HPC support
ShixiangWang Jun 13, 2026
052a809
chore: fix cargo fmt formatting issues
ShixiangWang Jun 13, 2026
3a94880
docs: v0.9 AI Companion design spec
ShixiangWang Jun 13, 2026
3a41113
feat: v0.9 AI Companion — Chat API + ChatUI foundation
ShixiangWang Jun 13, 2026
071a227
feat: complete v0.9 AI Companion — Settings, Monitor, Results
ShixiangWang Jun 13, 2026
34e66e1
Add AI agents, orchestrator, monitor & report
ShixiangWang Jun 13, 2026
818a499
Standardize AI API URLs, expose AI health, add frontend editors
ShixiangWang Jun 14, 2026
00ae322
refactor: consolidate API, fix routing, add DeepSeek, clean legacy code
ShixiangWang Jun 14, 2026
939225e
fix: clean entire assets dir before frontend build
ShixiangWang Jun 14, 2026
6aaf7bd
fix: remove duplicate license footer, add aria-label to Chat send button
ShixiangWang Jun 14, 2026
ac26b84
fix: diagnostics false positive, window.prompt→modal, dynamic index.h…
ShixiangWang Jun 14, 2026
2956501
fix: cargo fmt + clippy warnings, remove dead doc comment
ShixiangWang Jun 14, 2026
9abc030
fix: save workflow TOML to run directory before executor spawn
ShixiangWang Jun 14, 2026
c935f27
Refactor tests and update OpenAPI to 3.1
ShixiangWang Jun 14, 2026
5696ffa
refactor: PipelineSession context + contextual ChatUI + persistent state
ShixiangWang Jun 14, 2026
a5bb5e3
fix: ChatUI h1→span (avoid duplicate H1), update README v0.8 features
ShixiangWang Jun 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -73,3 +73,11 @@ logs/
node_modules/
frontend/node_modules/
frontend/dist/
*.db
*.db-shm
*.db-wal

# Frontend build artifacts (generated by 'npm run build')
crates/oxo-flow-web/static/assets/*.js
crates/oxo-flow-web/static/assets/*.css
crates/oxo-flow-web/static/index.html
17 changes: 15 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,13 +46,13 @@ make ci
- `docs/guide/src/reference/web-api.md` — REST API reference (structured errors, endpoints)
- `docs/guide/src/reference/web-system-architecture.md` — Web system architecture and AI-native API design
- `docs/guide/src/reference/architecture.md` — Overall system architecture
- `docs/schema/` — OpenAPI 3.0 schema and workflow JSON schema
- `docs/schema/` — OpenAPI 3.1 schema and workflow JSON schema

## 🌐 Web System (AI-Native API)
The web crate (`oxo-flow-web`) is designed as an AI-native API surface:

- **Structured errors**: All responses use `{code, message, detail, suggestion}` format
- **API discovery**: `GET /api/openapi.json` returns full OpenAPI 3.0 schema
- **API discovery**: `GET /api/openapi.json` returns full OpenAPI 3.1 schema
- **Intent-driven authoring**: `POST /api/workflows/generate` maps natural language to pipelines
- **SSE streaming**: `GET /api/events` provides real-time execution events
- **Pagination**: List endpoints use `{data, meta: {page, per_page, total_items, total_pages}}` envelope
Expand All @@ -69,7 +69,20 @@ cargo run -p oxo-flow-web # API server on :3000
cd frontend && npm run build # Outputs to frontend/dist/
```

### v0.8 Web Module Structure

The web crate uses a domain-driven modular monolith pattern:

- `domains/workflow/` — Pipeline parsing, validation, DAG building, formatting, data discovery
- `domains/execution/` — Run lifecycle, diagnostics engine (30+ error patterns), retry logic
- `domains/ai/` — AI translation layer (copilot prompts + service orchestration + handlers)
- `domains/auth/` — Authentication service, OAuth stubs, license endpoints
- `domains/collaboration/` — Pipeline fork, share, import
- `domains/observability/` — Health checks, system info, metrics
- `infra/db/` — StorageBackend trait, SqliteBackend, PostgreSQL backend (feature-gated)
- `server.rs` — Router assembly with all v0.8 API endpoints

Each domain follows: `types.rs` (data) → `service.rs` (pure logic) → `handlers.rs` (HTTP adapters).

## 🐳 Docker Deployment

Expand Down
109 changes: 67 additions & 42 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,48 +9,29 @@ Changelog entries are automatically generated by
[git-cliff](https://git-cliff.org) from conventional commit messages.
<!-- next-header -->

## [0.8.0] — 2026-06-13

### Features

- **s3-storage:** Real S3 storage backend backed by aws-sdk-s3.
Lazy OnceLock client init, standard AWS credential chain, all 6
StorageBackend trait methods. Requires `s3-storage` feature flag.
- **gcs-storage:** New GCS storage backend with HMAC-SHA1 auth via
GCS XML API. Reads credentials from GCS_ACCESS_KEY / GCS_SECRET_KEY.
Requires `gcs-storage` feature flag.

### Testing

- **webhook:** 11 new tests (4 → 15) covering HTTP method variants,
event filtering, HMAC edge cases, Slack payload conversion for all
event types, Event Display, WebhookData serialization.
- **plugin:** 9 new tests (6 → 15) covering ExecutorPlugin &
ReportPlugin traits, TOML deserialization, signature verification
edge cases (missing sig, unknown key, tampered payload).

### Performance

- **Micro-benchmarks:** Restructured into 4 modular bench targets
(dag_bench, parsing_bench, scheduling_bench, wildcard_bench) with
45 total bench functions (up from 10). DAG topologies: chain,
tree, star at 10-10k rules. Full parser lifecycle, scheduler
strategy comparison, expanded wildcard expansion.
- **Macro-benchmarks:** Python harness for CLI-driven lifecycle,
scaling, and reliability benchmarks. Supports automated result
collection in JSON format.
- **Comparative:** Nextflow and Snakemake equivalent pipeline
definitions for cross-engine benchmarking (requires external tools).

### Documentation

- **Cloud Storage:** New reference page covering S3/GCS setup,
feature flags, StorageBackend API, and known limitations.
- **LIMITATIONS.md:** Updated to reflect s3:// and gs:// support.
- **BENCHMARKS.md:** Benchmark methodology, usage, and CI integration.
- **Nav:** Cloud Storage entry added to mkdocs.yml.


## [0.8.0] - 2026-06-13

### Added
- **Domain-driven architecture**: Modular monolith with workflow, execution, AI, auth, collaboration, observability domains
- **StorageBackend trait**: Abstract database interface with SqliteBackend (default) and PostgreSQL (optional)
- **Diagnostics Engine**: 30+ deterministic error patterns for pipeline failure diagnosis with auto-fix suggestions
- **AI Translation Layer**: Natural language to .oxoflow pipeline generation with validation gate, failure explanation, result interpretation, and parameter optimization
- **Collaboration Primitives**: Pipeline fork, share (with oxo+https:// URL scheme), and import
- **License Middleware**: X-OxoFlow-License header injected into every HTTP response
- **Data Discovery**: Deterministic file format detection (18 bioinformatics formats) and reference genome discovery
- **New API Endpoints**: /api/pipelines/* (parse, validate, prepare, dag, format, lint, stats, diff, export, search), /api/runs/* (status, dag-status, diagnostics, retry), /api/data/* (analyze, reference), /api/ai/* (translate, explain, interpret, optimize)
- **S3 Storage Backend**: Real S3 storage backed by aws-sdk-s3 (feature-gated: `s3-storage`)
- **GCS Storage Backend**: GCS storage with HMAC-SHA1 auth via GCS XML API (feature-gated: `gcs-storage`)
- **Benchmarks**: Micro-benchmarks (4 targets, 45 bench functions), macro-benchmarks (Python harness), comparative benchmarks (Nextflow, Snakemake)
- 205+ unit and integration tests

### Changed
- Web crate refactored from flat structure to domain-driven modules
- API response format standardized with ApiError envelope
- Auth upgraded to if-let chains (Rust 2024)

### Security
- AI service layer has zero write access (DB, filesystem, process) — verified by automated test

## [0.7.0] — 2026-05-25

Expand Down Expand Up @@ -805,3 +786,47 @@ Changelog entries are automatically generated by
[0.1.1]: https://github.com/Traitome/oxo-flow/compare/v0.1.0...v0.1.1
[0.1.0]: https://github.com/Traitome/oxo-flow/releases/tag/v0.1.0


### v0.8.0 Phase 3 Completion (2026-06-13)

#### Resource Quota System (new)
- `infra/quota.rs` — Resource quota enforcement for team mode
- Per-user limits: concurrent runs, total threads, total memory, runs/day
- `QuotaTracker` with check/record_start/record_complete/reset_daily
- `GET /api/quota` endpoint returns current limits
- In-memory tracking with thread-safe Mutex
- 5 unit tests covering all operations

#### HPC Executor Bridge (new)
- `submit_to_hpc()` in `domains/execution/runner.rs`
- Generates SLURM/PBS submit scripts via `infra/hpc`
- Submits via sbatch/qsub, returns HPC job ID
- Mode-gated: HPC routes only in HPC mode (`server.rs`)

#### Frontend DAG Visualization Enhancements
- Toolbar: fit-to-view, zoom in/out, toggle layout (TB/LR), export PNG
- Hover tooltips: rule name, tool, threads, memory, duration
- Click-to-select with slide-out detail panel
- Running node pulse animation

#### Frontend Diagnostics Panel
- Auto-fix action buttons in diagnostic cards
- Resource bottleneck severity indicators
- Collapsible log line sections with syntax highlighting

#### Three-Tier UX (PipelineEditor mode switcher)
- One-click mode: intent-only input, no TOML required
- Config mode: form-based parameter editor
- Code mode: full TOML editor

#### Documentation Updates
- `ROADMAP.md` — Phase 3 completed status
- `deploy-modes.md` — quota system and HPC details
- `collaboration.md` — fork/share/import examples
- `openapi.yaml` — `/api/quota` endpoint added

#### Testing
- 15 Phase 3 integration tests (fork/share/import/modes/headers/quota)
- 1,085 total tests (0 failures)
- Browser E2E: 5 user flows, 12/13 pass
- `make ci` fully passing (fmt + clippy + build + test + audit)
33 changes: 33 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,10 @@ COPY --from=frontend-builder /app/frontend/dist /app/frontend/dist
RUN mkdir -p /app/data
ENV OXO_FLOW_FRONTEND_DIR=/app/frontend/dist
EXPOSE 3000
LABEL org.opencontainers.image.title="oxo-flow" \
org.opencontainers.image.description="Bioinformatics pipeline engine" \
org.opencontainers.image.version="0.8.0" \
org.opencontainers.image.licenses="Apache-2.0 AND LicenseRef-OxoFlow-Commercial" \
org.opencontainers.image.vendor="Traitome" \
org.opencontainers.image.authors="Shixiang Wang <wangsx@traitome.com>"
CMD ["/app/oxo-flow-web", "--host", "0.0.0.0", "--port", "3000"]
33 changes: 2 additions & 31 deletions LIMITATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,37 +39,8 @@ WDL, and no importers for those formats are currently available.
`s3://` and `gs://` URIs via feature flags (`s3-storage`, `gcs-storage`),
but data is fully downloaded before execution rather than streamed.
Streaming support is on the roadmap.
- **No GUI workflow editor** — The web interface provides monitoring and
management but does not include a visual drag-and-drop workflow editor.
- **No Kubernetes operator** — Cloud-native orchestration via Kubernetes
CRD/operator is not yet available.
- **No native distributed consensus** — oxo-flow assumes a shared filesystem
for multi-node execution. True distributed execution without shared storage
is not supported.

## Practical Constraints

- **Reference database management** — oxo-flow does not automatically download
or update reference databases. Versions can be tracked via the
`[[reference_db]]` configuration section.
- **Conda/Pixi environment resolution** — Environment creation depends on
external package managers. Network issues or solver conflicts are outside
oxo-flow's control.
- **Cluster backend specifics** — SLURM, PBS, SGE, and LSF backends rely on
the scheduler being correctly configured on the host system. oxo-flow
cannot diagnose cluster misconfiguration.
- **GPU scheduling** — GPU resource declarations are passed to the cluster
scheduler. On the local executor, a warning is emitted when GPU specs are
declared but GPU availability is not verified.

## Standards & Compliance

- **No GA4GH TES/WES support** — Task Execution Service and Workflow Execution
Service APIs are not implemented.
- **No FHIR/HL7 integration** — Clinical data interchange standards are not
natively supported. Report output is HTML/JSON.
- **Limited OpenAPI specification** — A basic OpenAPI 3.0 specification is
available but may not cover all endpoints comprehensively.
- **Limited OpenAPI specification** — A comprehensive OpenAPI 3.1 specification
is available at `GET /api/openapi.json`.
- **No HIPAA/GDPR de-identification tools** — PHI handling and data
de-identification must be managed by the user or external tools.
- **No native PDF export** — PDF generation requires `wkhtmltopdf` to be
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ build:
cargo build --workspace

test:
PATH="$$(pwd)/target/debug:$$PATH" cargo test --workspace
PATH="$$(pwd)/target/debug:$$PATH" cargo test --workspace -- --test-threads=1

audit:
cargo audit
Expand Down
Loading
Loading