Work on benchmark #62

ocots · 2025-10-11T12:24:28Z

@jbcaillau I was working on the benchmark too, considering your branch as starting point. I have added storage of needed info. If you want, based on what you are doing, I can update to set the pipeline from the bench to the doc.

For the stale dependencies:

Test that package loads all dependencies listed in Project.toml. Note that this does not imply that package loads the dependencies directly, this can be achieved via transitivity as well.

Generated by automated benchmark workflow Results saved to docs/src/assets/benchmark-minimal/data.json Ready for documentation generation

- Add solve_and_extract_data and benchmark_minimal_data functions - Support multiple models (JuMP, adnlp, exa) and solvers (Ipopt, MadNLP) - Add comprehensive tests for the new functionality - Update project dependencies - Add support for multiple discretization methods

- Moved all imports to CTBenchmarks.jl with consistent comments - Added Tables.jl as a dependency - Relaxed version constraints in Project.toml - Removed mini.jl as it's no longer needed

- Renamed benchmark_minimal to benchmark and benchmark_minimal_data to benchmark_data - Added max_iter and max_wall_time parameters to solve function - Removed default values from generic functions - Updated tests to match new function signatures - Moved benchmark script to scripts/ directory - Added comprehensive test suite for benchmark utilities

- Add benchmark_minimal function in src/utils.jl - Create GitHub Actions workflow for benchmark execution - Add documentation page for benchmark results - Update project dependencies and test suite

@timed

- Refactor API: replace separate solvers/models with solver=>models pairs - Add :exa_gpu model for GPU benchmarking with MadNLP + CUDA backend - Implement CUDA.@timed for GPU timing and memory tracking - Add automatic CUDA detection and filtering of exa_gpu when unavailable - Update format_benchmark_line to display both CPU and GPU metrics - Store full benchmark objects (@btimed or CUDA.@timed) in DataFrame - Add assertion: exa_gpu requires madnlp solver - Update all tests to use new API and verify GPU assertions - Update benchmark script with new solver_models structure

- Update _print_results to pass full benchmark object to format_benchmark_line - Remove direct access to time/allocs/memory fields which are now in benchmark object - Maintain same display format but with unified data access

@btimed

- Add helper function getval() to access fields from both Dict and NamedTuple - Support both String and Symbol keys when reading from JSON - Handle nested structures (cpu_gcstats, gpu_memstats) from Dict - Maintain compatibility with native benchmark objects (@btimed, CUDA.@timed) - Fix documentation build error when displaying benchmark results from JSON

- Add grid_size_max_cpu parameter to benchmark_data and benchmark functions - Filter CPU models (not ending with _gpu) when N > grid_size_max_cpu - GPU models run on all grid sizes regardless of grid_size_max_cpu - Update display logic to skip empty grid sizes and handle spacing correctly - Add test case verifying CPU/GPU filtering behavior - Update documentation with GPU benchmarking features and usage example - Update benchmark-core.jl script with grid_size_max_cpu=200 - Rename CI job from 'call' to 'cpu-tests' for clarity

…enchmarks.jl into 48-minimal-benchmark

Generated by reusable benchmark workflow Results saved to /scratch/github-actions/actions_runner_control_toolbox/_work/CTBenchmarks.jl/CTBenchmarks.jl/docs/src/assets/benchmark-core-moonshot/data.json Includes environment TOMLs

Generated by reusable benchmark workflow Results saved to /home/runner/work/CTBenchmarks.jl/CTBenchmarks.jl/docs/src/assets/benchmark-core-ubuntu-cpu/data.json Includes environment TOMLs

github-actions · 2025-10-13T14:00:06Z

✅ Benchmark and Documentation Complete

The automated workflow has completed successfully! 🎉

✅ Completed Tasks

📊 Benchmarks: Core benchmark executed and results saved to your branch
📚 Documentation: Documentation updated successfully
🔄 Integration: All changes integrated properly

📖 Documentation Preview

🌐 📚 View Documentation Preview ← Click to see your changes!

📋 Results

🎯 Benchmark results have been committed to your feature branch
📄 The docs/src/assets/benchmark-core/data.json file is now part of your PR changes
📚 Documentation has been regenerated with the latest benchmark data

🔗 Links

📊 View workflow run
🌿 View your feature branch

🤖 This notification was automatically generated

Generated by reusable benchmark workflow Results saved to /home/runner/work/CTBenchmarks.jl/CTBenchmarks.jl/docs/src/assets/benchmark-core-ubuntu-latest/data.json Includes environment TOMLs

github-actions · 2025-10-13T20:23:05Z

✅ Benchmark and Documentation Complete

The automated workflow has completed successfully! 🎉

✅ Completed Tasks

📊 Benchmarks: Core benchmark executed and results saved to your branch
📚 Documentation: Documentation updated successfully
🔄 Integration: All changes integrated properly

📖 Documentation Preview

🌐 📚 View Documentation Preview ← Click to see your changes!

📋 Results

🎯 Benchmark results have been committed to your feature branch
📄 The docs/src/assets/benchmark-core/data.json file is now part of your PR changes
📚 Documentation has been regenerated with the latest benchmark data

🔗 Links

📊 View workflow run
🌿 View your feature branch

🤖 This notification was automatically generated

jbcaillau · 2025-10-13T20:53:14Z

src/utils.jl

            end

            # Use CUDA.@timed for GPU benchmarking
+            madnlp(nlp_model_oc; opt...) # run for warmup


looks like the run times i see with Ipopt on CPU are better than those with MadNLP, which contradicts the tests i've made with tol = 1e-8 (not tested with `tol = 1e-6 ). to be checked

Generated by reusable benchmark workflow Results saved to /home/runner/work/CTBenchmarks.jl/CTBenchmarks.jl/docs/src/assets/benchmark-core-ubuntu-latest/data.json Includes environment TOMLs

…enchmarks.jl into 48-minimal-benchmark

Generated by reusable benchmark workflow Results saved to /scratch/github-actions/actions_runner_control_toolbox/_work/CTBenchmarks.jl/CTBenchmarks.jl/docs/src/assets/benchmark-core-moonshot/data.json Includes environment TOMLs

ocots and others added 12 commits September 20, 2025 15:34

foo

459939b

Merge 459939b into b722e68

2977985

📊 Add minimal benchmark results

a03a06b

Generated by automated benchmark workflow Results saved to docs/src/assets/benchmark-minimal/data.json Ready for documentation generation

added mini.jl

be15439

added mini.jl

9ec66b9

added mini.jl

d08ac87

added mini.jl

f36b11f

added madnlp

c7e2172

Merge branch '47-general-minimal-benchmark' into 48-minimal-benchmark

2412adf

Merge 2412adf into fc67523

4ab6570

📊 Add minimal benchmark results

fb6669b

Generated by automated benchmark workflow Results saved to docs/src/assets/benchmark-minimal/data.json Ready for documentation generation

github-actions bot assigned ocots Oct 11, 2025

ocots marked this pull request as draft October 11, 2025 12:29

ocots added 16 commits October 11, 2025 14:39

Refactor imports and update dependencies

b76d009

- Moved all imports to CTBenchmarks.jl with consistent comments - Added Tables.jl as a dependency - Relaxed version constraints in Project.toml - Removed mini.jl as it's no longer needed

foo

27757ae

Refactor benchmark workflows and formatting

8e60857

foo

c751341

foo

08971a1

Describe your changes

fbf95a5

test script

676dc11

feat: implement minimal benchmark pipeline

4dd7c14

- Add benchmark_minimal function in src/utils.jl - Create GitHub Actions workflow for benchmark execution - Add documentation page for benchmark results - Update project dependencies and test suite

ci: add moonshot gpu job

ec78a4e

update workflows with new runner and grid parameters

e33cd67

refactor: centralize CUDA guards and extend benchmark workflows

5d6ef4d

ocots and others added 3 commits October 13, 2025 15:42

Merge branch '48-minimal-benchmark' of github.com:control-toolbox/CTB…

59a150b

…enchmarks.jl into 48-minimal-benchmark

📊 Add benchmark results

8a8c292

Generated by reusable benchmark workflow Results saved to /scratch/github-actions/actions_runner_control_toolbox/_work/CTBenchmarks.jl/CTBenchmarks.jl/docs/src/assets/benchmark-core-moonshot/data.json Includes environment TOMLs

📊 Add benchmark results

763d0e8

Generated by reusable benchmark workflow Results saved to /home/runner/work/CTBenchmarks.jl/CTBenchmarks.jl/docs/src/assets/benchmark-core-ubuntu-cpu/data.json Includes environment TOMLs

ocots removed the run bench core ubuntu Trigger the workflow to make the core benchmark on ubuntu latest and the documentation. label Oct 13, 2025

ocots added 3 commits October 13, 2025 20:57

Remove grid_size_max_cpu handling

f3e0bd0

Restrict JuMP models to trapeze

7a6b143

split workflows

c34abeb

ocots added the run bench core ubuntu Trigger the workflow to make the core benchmark on ubuntu latest and the documentation. label Oct 13, 2025

📊 Add benchmark results

30be0bc

Generated by reusable benchmark workflow Results saved to /home/runner/work/CTBenchmarks.jl/CTBenchmarks.jl/docs/src/assets/benchmark-core-ubuntu-latest/data.json Includes environment TOMLs

ocots removed the run bench core ubuntu Trigger the workflow to make the core benchmark on ubuntu latest and the documentation. label Oct 13, 2025

Restructure orchestrator for multiple benchmarks

e53c45e

jbcaillau reviewed Oct 13, 2025

View reviewed changes

fix doc trigger

27cea8d

ocots closed this Oct 13, 2025

ocots reopened this Oct 13, 2025

fix doc trigger

1486472

ocots added the run bench core ubuntu Trigger the workflow to make the core benchmark on ubuntu latest and the documentation. label Oct 13, 2025

renamed

4a4ddcb

ocots added the run bench core moonshot Trigger the workflow to make the core benchmark on moonshot and the documentation. label Oct 13, 2025

github-actions bot and others added 9 commits October 13, 2025 21:11

📊 Add benchmark results

946c0d1

Generated by reusable benchmark workflow Results saved to /home/runner/work/CTBenchmarks.jl/CTBenchmarks.jl/docs/src/assets/benchmark-core-ubuntu-latest/data.json Includes environment TOMLs

all problems

195694d

test print bytes

b8d2bbf

📊 Add benchmark results

255681a

Generated by reusable benchmark workflow Results saved to /home/runner/work/CTBenchmarks.jl/CTBenchmarks.jl/docs/src/assets/benchmark-core-ubuntu-latest/data.json Includes environment TOMLs

add MadNLPGPU

656a85f

Merge branch '48-minimal-benchmark' of github.com:control-toolbox/CTB…

f977652

…enchmarks.jl into 48-minimal-benchmark

📊 Add benchmark results

acf4ac0

Generated by reusable benchmark workflow Results saved to /scratch/github-actions/actions_runner_control_toolbox/_work/CTBenchmarks.jl/CTBenchmarks.jl/docs/src/assets/benchmark-core-moonshot/data.json Includes environment TOMLs

remove MadNLP

e6b1561

add print trace

d66374c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Work on benchmark #62

Work on benchmark #62

Uh oh!

ocots commented Oct 11, 2025

Uh oh!

github-actions bot commented Oct 13, 2025

Uh oh!

github-actions bot commented Oct 13, 2025

Uh oh!

jbcaillau Oct 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Work on benchmark #62

Are you sure you want to change the base?

Work on benchmark #62

Uh oh!

Conversation

ocots commented Oct 11, 2025

Uh oh!

github-actions bot commented Oct 13, 2025

✅ Benchmark and Documentation Complete

✅ Completed Tasks

📖 Documentation Preview

📋 Results

🔗 Links

Uh oh!

github-actions bot commented Oct 13, 2025

✅ Benchmark and Documentation Complete

✅ Completed Tasks

📖 Documentation Preview

📋 Results

🔗 Links

Uh oh!

jbcaillau Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants