diff --git a/README.md b/README.md index 5d312a8b..6b80b071 100644 --- a/README.md +++ b/README.md @@ -24,13 +24,25 @@ A high-performance command-line interface for contributing proofs to the Nexus n [Nexus](https://nexus.xyz/) is a global distributed prover network that unites the world's computers to power a new and better Internet: the Verifiable Internet. +## Network Status + +**Current Version**: v0.10.17 + There have been several testnets so far: - Testnet 0: [October 8 – 28, 2024](https://blog.nexus.xyz/nexus-launches-worlds-first-open-prover-network/) - Testnet I: [December 9 – 13, 2024](https://blog.nexus.xyz/the-new-nexus-testnet-is-live/) - Testnet II: [February 18 – 22, 2025](https://blog.nexus.xyz/testnet-ii-is-open/) - Devnet: [February 22 - June 20, 2025](https://docs.nexus.xyz/layer-1/testnet/devnet) -- Testnet III: [Ongoing](https://blog.nexus.xyz/live-everywhere/) +- Testnet III: [March 2025 - Ongoing](https://blog.nexus.xyz/live-everywhere/) + +### Recent Improvements (v0.10.17) + +- **Enhanced Thread Management**: Automatic optimization based on CPU cores and available memory +- **Improved Memory Checking**: Real-time monitoring with detailed warnings +- **Better Error Handling**: Timeout protection and progressive error categorization +- **Performance Optimization**: More efficient parallel processing with `join_all` +- **Stability Improvements**: Enhanced error recovery and resource management --- @@ -76,7 +88,7 @@ nexus-cli register-node --node-id nexus-cli start ``` -To run the CLI noninteractively, you can also opt to start it in headless mode. +To run the CLI non-interactively, you can also opt to start it in headless mode. ```bash nexus-cli start --headless @@ -96,6 +108,27 @@ For troubleshooting or to see available command-line options, run: nexus-cli --help ``` +### Thread Management and Performance + +The Nexus CLI automatically optimizes thread usage based on your system's capabilities: + +- **Default threads**: Half of your CPU cores for optimal performance +- **Maximum threads**: Capped at 75% of total cores to maintain system stability +- **Memory checking**: Each thread requires ~4GB RAM, automatically adjusted based on available memory + +#### Thread Configuration + +```bash +# Let CLI auto-detect optimal thread count (recommended) +nexus-cli start + +# Manually specify thread count +nexus-cli start --max-threads 4 + +# Enable detailed memory checking +nexus-cli start --check-memory +``` + ### Adaptive Task Difficulty The Nexus CLI features an **adaptive difficulty system** that automatically adjusts task difficulty based on your node's performance. This ensures optimal resource utilization while preventing system overload. @@ -107,13 +140,13 @@ The Nexus CLI features an **adaptive difficulty system** that automatically adju #### When to Override Difficulty -**Lower Difficulty** (e.g. `Small` or `SmallMedium`): +**Lower Difficulty** (e.g. `small` or `small_medium`): - Resource-constrained systems - Background processing alongside other apps - Testing/development environments - Battery-powered devices -**Higher Difficulty** (e.g. `Large`, `ExtraLarge`, or `ExtraLarge2`): +**Higher Difficulty** (e.g. `large`, `extra_large`, or `extra_large_2`): - High-performance hardware (8+ cores, 16+ GB RAM) - Dedicated proving machines - Maximum reward optimization @@ -142,12 +175,13 @@ nexus-cli start --max-difficulty Medium #### Difficulty Guidelines -| Difficulty | Use Case | -|------------|----------| -| `small` | Default, starting task | -| `small_medium` | Building reputation | -| `medium` and `large` | Standard desktop/laptop | -| `extra_large` and above | High-performance systems, more points | +| Difficulty | RAM Required | CPU Cores | Use Case | +|------------|--------------|-----------|----------| +| `small` | 4-8GB | 1-2 cores | Default, starting task | +| `small_medium` | 8-12GB | 2-4 cores | Building reputation | +| `medium` | 12-16GB | 4-6 cores | Standard desktop/laptop | +| `large` | 16-24GB | 6-8 cores | High-performance desktop | +| `extra_large` and above | 24GB+ | 8+ cores | Dedicated proving machines, maximum points | > **Tip**: Use `nexus-cli start --help` to see the full auto-promotion details in the CLI help text. @@ -155,7 +189,7 @@ nexus-cli start --max-difficulty Medium **Tasks taking too long:** -Try a lower difficulty. +Try a lower difficulty: ```bash nexus-cli start --max-difficulty small_medium @@ -163,16 +197,18 @@ nexus-cli start --max-difficulty small_medium **Want more challenging tasks:** -Request a harder difficulty. It will still take time to build up reputation to get the requested difficulty. +Request a harder difficulty. Note: It will still take time to build up reputation to get the requested difficulty: ```bash nexus-cli start --max-difficulty extra_large_2 ``` **Unsure about system capabilities:** -- Use the default adaptive system (no `--max-difficulty` needed) +- Use the default adaptive system (no `--max-difficulty` flag needed) - The system will automatically find the optimal difficulty for your hardware -- Only override if you're fine-tuning performance +- CLI now defaults to optimal thread count based on your CPU cores +- Memory checking ensures stable operation without system overload +- Only override if you're fine-tuning performance for specific use cases ### Docker Installation @@ -189,6 +225,13 @@ docker compose logs # Check logs docker compose down # Shutdown ``` +For direct Docker usage: + +```bash +docker run -d --name nexus-node --restart unless-stopped --init \ + nexusxyz/nexus-cli:latest start --headless --node-id +``` + --- ## Terms of Use @@ -212,13 +255,37 @@ following format: --- +## Performance Optimization + +### System Requirements + +**Minimum Requirements:** +- 4GB RAM (for 1 thread) +- 2 CPU cores +- Stable internet connection + +**Recommended for Optimal Performance:** +- 16GB+ RAM (for 4+ threads) +- 8+ CPU cores +- SSD storage +- Dedicated proving machine + +### Memory Management + +The CLI includes intelligent memory management: +- Automatic thread count optimization based on available memory +- Real-time memory monitoring and warnings +- Graceful degradation on memory-constrained systems +- Each proving thread requires approximately 4GB RAM + + + ## Get Help - [Network FAQ](https://docs.nexus.xyz/layer-1/testnet/faq) - [Discord Community](https://discord.gg/nexus-xyz) - Technical issues? [Open an issue](https://github.com/nexus-xyz/nexus-cli/issues) -- To submit programs to the network for proving, contact - [growth@nexus.xyz](mailto:growth@nexus.xyz). +- To submit programs to the network for proving, contact [growth@nexus.xyz](mailto:growth@nexus.xyz) --- @@ -247,6 +314,9 @@ sudo apt update sudo apt upgrade sudo apt install build-essential pkg-config libssl-dev git-all sudo apt install protobuf-compiler + +# Verify installation +protoc --version ``` #### macOS @@ -261,14 +331,35 @@ protoc --version #### Windows -[Install WSL](https://learn.microsoft.com/en-us/windows/wsl/install), -then see Linux instructions above. +**Option 1: WSL (Recommended)** +[Install WSL](https://learn.microsoft.com/en-us/windows/wsl/install), then follow Linux instructions above. -```bash +**Option 2: Native Windows** +```powershell # Install using Chocolatey choco install protobuf + +# Or using winget +winget install protobuf + +# Verify installation +protoc --version ``` +**Option 3: Manual Installation** +Download protobuf compiler from [GitHub releases](https://github.com/protocolbuffers/protobuf/releases) and add to PATH. + +## Changelog + +### v0.10.17 (Current) +- **Enhanced Thread Management**: Intelligent default thread allocation (half of CPU cores) +- **Improved Memory Checking**: Real-time available memory calculation with detailed warnings +- **Better Parallel Processing**: Timeout protection and improved error handling for proving pipeline +- **Performance Optimization**: More efficient resource utilization and stability improvements +- **Bug Fixes**: Various stability and performance improvements + +For complete changelog, see [GitHub Releases](https://github.com/nexus-xyz/nexus-cli/releases). + ## License Nexus CLI is distributed under the terms of both the [MIT License](./LICENSE-MIT) and the [Apache License (Version 2.0)](./LICENSE-APACHE). diff --git a/clients/cli/src/main.rs b/clients/cli/src/main.rs index 1a99287f..3789bbde 100644 --- a/clients/cli/src/main.rs +++ b/clients/cli/src/main.rs @@ -104,7 +104,7 @@ enum Command { #[arg(long = "headless", action = ArgAction::SetTrue)] headless: bool, - /// Maximum number of threads to use for proving. Capped at the number of CPU cores. + /// Maximum number of threads to use for proving. Defaults to half of CPU cores, capped at 75% of total cores. #[arg(long = "max-threads", value_name = "MAX_THREADS")] max_threads: Option, @@ -237,7 +237,7 @@ async fn main() -> Result<(), Box> { /// * `env` - The environment to connect to. /// * `config_path` - Path to the configuration file. /// * `headless` - If true, runs without the terminal UI. -/// * `max_threads` - Optional maximum number of threads to use for proving. +/// * `max_threads` - Optional maximum number of threads to use for proving. Defaults to optimal count based on CPU cores. /// * `check_mem` - Whether to check risky memory usage. /// * `with_background` - Whether to use the alternate TUI background color. /// * `max_tasks` - Optional maximum number of tasks to prove. diff --git a/clients/cli/src/prover/pipeline.rs b/clients/cli/src/prover/pipeline.rs index b7c11b84..5d66bb60 100644 --- a/clients/cli/src/prover/pipeline.rs +++ b/clients/cli/src/prover/pipeline.rs @@ -107,13 +107,25 @@ impl ProvingPipeline { }) .collect(); - // Use join_all for better parallelization - let results = join_all(handles).await; + // Use join_all for better parallelization with timeout + let results = tokio::time::timeout( + std::time::Duration::from_secs(30 * 60), // 30 minute timeout + join_all(handles) + ).await; + + let results = match results { + Ok(results) => results, + Err(_) => { + cancellation_token.cancel(); + return Err(ProverError::MalformedTask("Task timeout exceeded".to_string())); + } + }; // Process results and collect verification failures for batch handling let mut all_proofs = Vec::new(); let mut proof_hashes = Vec::new(); let mut verification_failures = Vec::new(); + let mut critical_errors = Vec::new(); for (result_index, result) in results.into_iter().enumerate() { match result { @@ -122,7 +134,7 @@ impl ProvingPipeline { proof_hashes.push(proof_hash); } Ok(Err(e)) => { - // Collect verification failures for batch processing + // Categorize errors for better handling match e { ProverError::Stwo(_) | ProverError::GuestProgram(_) => { verification_failures.push(( @@ -133,18 +145,22 @@ impl ProvingPipeline { )); } _ => { - // Cancel remaining tasks on critical errors - cancellation_token.cancel(); - return Err(e); + critical_errors.push(e); } } } Err(join_error) => { - return Err(ProverError::JoinError(join_error)); + critical_errors.push(ProverError::JoinError(join_error)); } } } + // Handle critical errors first - cancel remaining tasks + if !critical_errors.is_empty() { + cancellation_token.cancel(); + return Err(critical_errors.into_iter().next().unwrap()); + } + // Handle all verification failures in batch (avoid nested spawns) let failure_count = verification_failures.len(); for (task, error_msg, env, client) in verification_failures { diff --git a/clients/cli/src/session/setup.rs b/clients/cli/src/session/setup.rs index e2817d46..7848847c 100644 --- a/clients/cli/src/session/setup.rs +++ b/clients/cli/src/session/setup.rs @@ -38,16 +38,32 @@ fn clamp_threads_by_memory(requested_threads: usize) -> usize { sysinfo.refresh_memory(); let total_system_memory = sysinfo.total_memory(); + let used_memory = sysinfo.used_memory(); let memory_per_thread = crate::consts::cli_consts::PROJECTED_MEMORY_REQUIREMENT; - // Calculate max threads based on total system memory - // Reserve 25% of system memory for OS and other processes - let available_memory = (total_system_memory as f64 * 0.75) as u64; - let max_threads_by_memory = (available_memory / memory_per_thread) as usize; + // Calculate available memory more accurately + // Use actual available memory instead of just reserving 25% + let available_memory = total_system_memory.saturating_sub(used_memory); + + // Reserve additional 2GB for OS and other processes to be safe + let reserved_memory = 2_147_483_648u64; // 2GB + let usable_memory = available_memory.saturating_sub(reserved_memory); + + let max_threads_by_memory = (usable_memory / memory_per_thread) as usize; // Return the minimum of requested threads and memory-limited threads - // Always allow at least 1 thread - requested_threads.min(max_threads_by_memory.max(1)) + // Always allow at least 1 thread, but warn if memory is very low + let result = requested_threads.min(max_threads_by_memory.max(1)); + + if max_threads_by_memory == 0 { + crate::print_cmd_warn!( + "Low memory", + "Available memory ({:.1}GB) is below recommended minimum. Performance may be degraded.", + usable_memory as f64 / 1_073_741_824.0 + ); + } + + result } /// Warn the user if their available memory seems insufficient for the task(s) at hand @@ -115,7 +131,17 @@ pub async fn setup_session( // Clamp the number of workers to [1, 75% of num_cores]. Leave room for other processes. let total_cores = crate::system::num_cores(); let max_workers = ((total_cores as f64 * 0.75).ceil() as usize).max(1); - let mut num_workers: usize = max_threads.unwrap_or(1).clamp(1, max_workers as u32) as usize; + + // Default to optimal thread count instead of just 1 + let default_threads = if total_cores >= 4 { + (total_cores / 2).max(1) + } else { + 1 + }; + + let mut num_workers: usize = max_threads + .unwrap_or(default_threads as u32) + .clamp(1, max_workers as u32) as usize; // Check memory and clamp threads if max-threads was explicitly set OR check-memory flag is set if max_threads.is_some() || check_mem { @@ -129,6 +155,18 @@ pub async fn setup_session( ); num_workers = memory_clamped_workers; } + + // Additional detailed memory check + let (sufficient, available_gb, required_gb) = crate::system::check_memory_for_threads(num_workers); + if !sufficient { + crate::print_cmd_warn!( + "Memory warning", + "Available memory ({:.1}GB) may be insufficient for {} threads (requires {:.1}GB). Consider reducing --max-threads.", + available_gb, + num_workers, + required_gb + ); + } } // Additional memory warning if explicitly requested diff --git a/clients/cli/src/system.rs b/clients/cli/src/system.rs index 2052c610..56c4e54c 100644 --- a/clients/cli/src/system.rs +++ b/clients/cli/src/system.rs @@ -149,6 +149,24 @@ pub fn process_memory_gb() -> f64 { memory as f64 / 1024.0 / 1024.0 / 1024.0 // Convert to GB (binary) } +/// Get available memory in GB (free + cached/buffered) +pub fn available_memory_gb() -> f64 { + let mut sys = System::new(); + sys.refresh_memory(); + + let available_memory = sys.available_memory(); // bytes + available_memory as f64 / 1024.0 / 1024.0 / 1024.0 // Convert to GB (binary) +} + +/// Check if system has enough memory for the requested number of threads +pub fn check_memory_for_threads(num_threads: usize) -> (bool, f64, f64) { + let available_gb = available_memory_gb(); + let required_gb = (num_threads as f64 * 4.0) + 2.0; // 4GB per thread + 2GB buffer + let sufficient = available_gb >= required_gb; + + (sufficient, available_gb, required_gb) +} + // We encode the memory usage to i32 type at client fn bytes_to_mb_i32(bytes: u64) -> i32 { // Convert to MB with 3 decimal places of precision