Skip to content

Conversation

Jim8y
Copy link
Contributor

@Jim8y Jim8y commented Sep 22, 2025

Description

  • Establish a dedicated benchmarking harness that exercises every Neo VM opcode, syscall, and native contract method across baseline, single-call, and saturated profiles so we can observe how execution behaves as inputs scale.
  • Record end-to-end runtime, allocations, stack depth, and gas deltas for each scenario via a shared BenchmarkResultRecorder, then consolidate the data into component-specific CSVs plus a global scaling report to highlight complexity trends.
  • Add coverage tracking (with enforcement in the final report) to flag any opcode, syscall, or native manifest method that lacks a benchmark, eliminating silent gaps when the VM surface evolves.
  • Seed native benchmarks with deterministic protocol settings and snapshot helpers to remove the multisig bootstrap crash and keep scenarios consistent regardless of user configuration.
  • Document the benchmarking workflow, coverage expectations, and dynamic pricing methodology so teams can reproduce runs and interpret the resulting artifacts.

Fixes # (issue)

Type of change

  • Optimization (the change is only an optimization)
  • Style (the change is only a code style for better maintenance or standard purpose)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • env NEO_BENCHMARK_ARTIFACTS=$(pwd)/benchmarks/Neo.VM.Benchmarks/bin/Release/net9.0/BenchmarkArtifacts_Lite dotnet run -c Release --framework net9.0 --project benchmarks/Neo.VM.Benchmarks -- --filter "Opcode:DROP (Micro)" --job short
  • env NEO_BENCHMARK_ARTIFACTS=$(pwd)/benchmarks/Neo.VM.Benchmarks/bin/Release/net9.0/BenchmarkArtifacts_LiteSyscall dotnet run -c Release --framework net9.0 --project benchmarks/Neo.VM.Benchmarks -- --filter "Syscall:System.Runtime.Platform (Micro)" --job short
  • env NEO_BENCHMARK_ARTIFACTS=$(pwd)/benchmarks/Neo.VM.Benchmarks/bin/Release/net9.0/BenchmarkArtifacts_LiteNative dotnet run -c Release --framework net9.0 --project benchmarks/Neo.VM.Benchmarks -- --filter "Neo.VM.Benchmark.Native.NativeSuite.*" --job short

Test Configuration:

  • OS: Ubuntu 24.04.3 LTS (x86_64)
  • .NET SDK: 9.0.301
  • Hardware: Intel Core Ultra 9 285K (16 cores)

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

@Jim8y Jim8y marked this pull request as draft September 22, 2025 10:16
@Jim8y Jim8y force-pushed the fix/benchmark-native-fallback branch from 9fd9182 to e00d0b2 Compare September 22, 2025 10:17
@Jim8y Jim8y changed the title Fix fallback protocol settings for native benchmarks Instrument full Neo VM benchmarking coverage for opcodes, syscalls, and native methods Sep 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants