Skip to content

devicesimulator.md

Latest commit

 

History

History
382 lines (269 loc) · 18.5 KB

devicesimulator.md

File metadata and controls

382 lines (269 loc) · 18.5 KB

Device Simulator

devicesimulator is used to simulate the load generated by devices. It launches independent instances of the flightctl-agent process with configurable concurrency and monitoring capabilities.

The simulator automatically handles device enrollment, fleet assignment, and provides comprehensive metrics for performance testing. Certain fields of a device spec (for example OS image updates and applications) can cause issues during simulation. It is highly recommended to avoid including OS images or Applications in specs.

Setup

Copy the necessary setup files to run a devicesimulator.

mkdir -p ~/.flightctl/certs/ ~/.config/flightctl/ 
cp bin/agent/etc/flightctl/certs/* ~/.flightctl/certs/
cp bin/agent/etc/flightctl/config.yaml ~/.flightctl/agent.yaml

Changes to the configuration of all launched agents can be made in ~/.flightctl/agent.yaml Some key options available for tuning:

  • status-update-interval - How often the agent reports its status to the API. Higher intervals reduce server load.

See Installing the Agent for all available options.

CLI Reference

The device simulator supports various command-line options organized into logical groups:

Device Configuration

  • --count (default: 1): Number of devices to simulate
  • --initial-device-index (default: 0): Starting index for device name suffix (e.g., device-00000 for 0, device-00200 for 200)
  • --label: Label applied to simulated devices in the format key=value (can be specified multiple times)

Concurrency Configuration

  • --max-concurrency (default: 100): Maximum number of agents that can be creating simultaneously
  • --agent-startup-jitter (default: -1s): Maximum random delay when starting agents (negative = use status-update-interval, 0 = no jitter, positive = custom duration)

Runtime Configuration

  • --stop-after: Stop the simulator after the specified duration (format: 1h30m, 5m, etc.)
  • --log-level, -v (default: debug): Logger verbosity level (fatal, error, warn, warning, info, debug)
  • --metrics (default: localhost:9093): Address for the metrics endpoint
  • --source-ips: Comma-separated list of source IP addresses for device management HTTP connections

File and Directory Configuration

  • --config (default: ~/.flightctl/agent.yaml): Path of the agent configuration template
  • --data-dir (default: ~/.flightctl/data): Directory for storing simulator data

Output Configuration

  • --output, -o: Output format for version command (json, yaml)

Commands

  • version: Print device simulator version information
  • help: Show help message

Version Command Examples

# Default text output
bin/devicesimulator version

# JSON output
bin/devicesimulator version --output=json

# YAML output  
bin/devicesimulator version --output=yaml

Examples

# Basic usage - simulate 1 device
bin/devicesimulator

# Simulate 100 devices with custom labels
bin/devicesimulator --count=100 --label="environment=test" --label="region=us-east"

# Performance testing with high concurrency
bin/devicesimulator --count=1000 --max-concurrency=100 --log-level=error

# Distributed simulation starting at device index 500
bin/devicesimulator --count=200 --initial-device-index=500

# Time-limited simulation
bin/devicesimulator --count=50 --stop-after=1h

# Large-scale simulation with multiple source IPs to avoid port exhaustion
bin/devicesimulator --count=5000 --source-ips=192.168.1.10,192.168.1.11,192.168.1.12 --max-concurrency=100

# Using source IPs with custom network interface aliases
bin/devicesimulator --count=10000 --source-ips=10.0.0.100,10.0.0.101,10.0.0.102,10.0.0.103

# Control agent startup timing with custom jitter
bin/devicesimulator --count=1000 --agent-startup-jitter=30s

# Disable startup jitter for immediate agent launch  
bin/devicesimulator --count=100 --agent-startup-jitter=0

# Use default jitter behavior (status-update-interval)
bin/devicesimulator --count=500 --agent-startup-jitter=-1s

# High-frequency testing with minimal jitter
bin/devicesimulator --count=500 --agent-startup-jitter=5s --max-concurrency=50

Configuration

Agent Configuration Template

The simulator uses an agent configuration template (~/.flightctl/agent.yaml by default) that is applied to all simulated devices. This template is copied and customized for each agent instance.

Key configuration parameters that affect simulator performance:

  • status-update-interval: How often agents report status (default from template)
  • log-level: The simulator propagates its --log-level to agents unless log-level is explicitly set in the agent template, in which case the agent configuration takes precedence.

Device Naming

Devices are named using the format device-XXXXX where XXXXX is a zero-padded 5-digit number starting from the --initial-device-index value.

Agent Startup Timing

The simulator uses configurable random jitter when starting agents to prevent thundering herd effects. The jitter behavior is controlled by the --agent-startup-jitter flag:

  • Negative values (default: -1s): Use the agent's status-update-interval setting as the maximum jitter duration. This provides backward-compatible behavior with random delays between 0 and the status update interval (typically 60 seconds).
  • Zero (0): No jitter - agents start immediately without any random delay. Useful for testing scenarios requiring precise timing.
  • Positive values: Use the specified duration as the maximum jitter. For example, --agent-startup-jitter=30s creates random delays between 0 and 30 seconds.

Common Usage Patterns:

  • Default behavior: Omit the flag or use --agent-startup-jitter=-1s for production-like load distribution
  • Immediate startup: Use --agent-startup-jitter=0 for testing scenarios requiring synchronized agent launches
  • Custom timing: Use --agent-startup-jitter=30s or similar for specific load distribution needs

For testing scenarios where you need precise timing or want to minimize startup delays, setting jitter to 0 or a small positive value can be beneficial. For production-like load testing, the default behavior usually provides good load distribution.

Today the agent assumes a bootc host so the default disk alerts are based on /sysroot path. To avoid many errors in the logs as a result of being unable to read from /sysroot consider either:

  1. Create the empty directory sudo mkdir /sysroot
  2. Create a fleet with a disk alert with the path / and assign all created devices to that fleet

Limits

The number of devices runnable on a single host is determined by the resources available to the host and what each device is configured to do.

It is possible to run 10k devices on a single instance running the flightctl services and the simulator provided proper system tuning is applied.

Performance Tuning Recommendations

  1. CPU Isolation: Limit CPUs available to the simulator to ensure resources for flightctl services:

    taskset -c 0-7 bin/devicesimulator --count 10000

  2. Avoid Applications: Don't assign applications to simulated devices. Applications trigger podman events processes that consume significant CPU and reduce scalability.

  3. Tune Agent Intervals: Increase intervals in ~/.flightctl/agent.yaml to reduce API load:

    status-update-interval: 60s

  4. Optimize Logging: Use higher log levels to reduce I/O overhead:

    bin/devicesimulator --log-level=error --count 10000

  5. Concurrency Control: Tune simulator concurrency based on system capacity:

    # For systems with limited resources
bin/devicesimulator --max-concurrency=25

# For high-performance systems  
bin/devicesimulator --max-concurrency=100

  6. Source IP Distribution: For very large-scale simulations (>30,000 devices), use multiple source IPs to overcome ephemeral port limitations:

    # Setup network aliases first (requires root)
sudo ip addr add 192.168.1.100/24 dev eth0
sudo ip addr add 192.168.1.101/24 dev eth0
sudo ip addr add 192.168.1.102/24 dev eth0

# Run simulator with source IP distribution
bin/devicesimulator --count=50000 --source-ips=192.168.1.100,192.168.1.101,192.168.1.102 --max-concurrency=100

    Note: This method adds IP aliases to a physical interface, making them visible and accessible to other hosts on the same network. You must ensure the chosen IPs do not conflict with other devices on your LAN.

    Advanced: Using a Network Bridge for IP Isolation

    For a cleaner and more isolated setup, especially when adding a large number of IPs, you can create a virtual bridge interface. This avoids cluttering your main network interface and uses a dedicated, non-existent network range, preventing any potential IP conflicts on your LAN.

    Note: This creates a host-only bridge. It is isolated within the host machine and is not connected to any physical network. This is the desired setup for the simulator as it prevents network conflicts and is simple to manage.

    1. Create and activate a bridge:

      # Create a new bridge named br-sim
sudo ip link add name br-sim type bridge

# Bring the bridge interface up
sudo ip link set br-sim up

    2. Assign a block of IP addresses to the bridge: This example assigns IPs in the 10.100.0.0/24 range. The host's kernel will handle routing traffic from this bridge to the flightctl service.

      # Assign multiple IPs to the bridge for the simulator to use
for i in {100..103}; do sudo ip addr add 10.100.0.$i/24 dev br-sim; done

    3. Run the simulator with the new IPs:

      bin/devicesimulator \
  --count=10000 \
  --source-ips=10.100.0.100,10.100.0.101,10.100.0.102,10.100.0.103 \
  --max-concurrency=100

    4. Cleanup (when finished): When you are done with the simulation, you can remove the bridge and all its associated IPs with a single command:

      sudo ip link delete br-sim

System-level Connection Limits

When simulating a large number of devices, each running agent establishes multiple HTTP connections to the flightctl server. This can exhaust system resources related to network connections. Understanding these limits is crucial for large-scale simulations.

  • File Descriptors and ulimit: On Linux and other Unix-like systems, every network connection is treated as a file descriptor. Each process has a limit on the number of file descriptors it can have open simultaneously. You can check the current limit for your shell session with ulimit -n. This is often a soft limit that can be increased. If a process tries to open more file descriptors than its limit, it will fail, leading to connection errors. You can increase the soft limit for the simulator's session, but you might also need to adjust the hard limit.

  • Hard Limits: The hard limit is the maximum value to which a soft limit can be raised. It's a system-wide configuration that prevents any single user or process from consuming all available resources. Modifying hard limits usually requires root privileges and involves editing files like /etc/security/limits.conf.

  • Ephemeral Ports: When a client (like a flightctl agent) connects to a server, it is assigned a temporary (or ephemeral) port from a specific range. The operating system has a finite number of these ports available. Once this range is exhausted, no new outgoing connections can be made, and you'll see "address already in use" errors. You can view the ephemeral port range on a Linux system with sysctl net.ipv4.ip_local_port_range. While this range can be adjusted, it represents a hard ceiling on the number of simultaneous connections from a single IP address.

  • Source IP Distribution: To overcome ephemeral port limitations when simulating large numbers of devices (typically >30,000), you can use the --source-ips flag to distribute connections across multiple IP addresses. Each IP address gets its own ephemeral port range, effectively multiplying your connection capacity. For example, with 4 source IPs, you can theoretically support 4x the number of concurrent connections. The simulator automatically distributes agents across the provided source IPs using round-robin assignment.

In addition to the file descriptors opened by agent HTTP connections, agents also open many more traditional files for various purposes (persistence, system details, .etc).

When running the device simulator, you may need to tune these system parameters to support a high number of concurrent agents.

Running

    bin/devicesimulator --count=1

Automatic Fleet Creation

The device simulator automatically creates a fleet configuration named simulator-disk-monitoring if it doesn't already exist. This fleet:

  • Eliminates disk usage errors that would occur with default monitoring configurations
  • Uses the selector created_by: device-simulator to automatically match all devices created by the simulator
  • Loads configuration from examples/fleet-disk-simulator.yaml
  • Provides appropriate disk monitoring settings for simulated devices

All devices created by the simulator are automatically labeled with created_by: device-simulator, ensuring they are matched by this fleet without manual intervention.

Device Enrollment Process

The simulator handles the complete device lifecycle:

  1. Creates agent instances with unique device names (format: device-XXXXX)
  2. Starts agent processes that initiate enrollment requests
  3. Automatically approves enrollment requests when they appear
  4. Applies configured labels to enrolled devices
  5. Manages concurrent device creation with configurable limits

Monitoring

The device simulator provides metrics through a built-in HTTP endpoint for monitoring performance and tracking device states.

Metrics Endpoint

By default, metrics are exposed on localhost:9093. You can customize this with the --metrics flag:

bin/devicesimulator --metrics="0.0.0.0:8080" --count=100

Available Metrics

The simulator tracks various metrics including:

  • Number of active agents
  • Device creation success/failure rates
  • Enrollment completion times
  • Agent process lifecycle events

These metrics can be scraped by monitoring systems like Prometheus for observability during large-scale testing.

Configuration

The metrics endpoint is automatically configured when the simulator starts. No additional setup is required, but ensure the metrics port is accessible if running on remote hosts or in containerized environments.

Remote simulator

To simulate a large number of devices (10,000 for example) a devicesimulator has to be installed on a number of separate hosts to let it use more resources. Each remote host must be set up with the same files. Then you have to start simulator on each host.

Remote setup

Run the same commands per each host. Note: some commands must be run remotely.

    # on remote host
    mkdir -p ~/.flightctl/certs/ ~/.config/flightctl ~/bin
    scp bin/agent/etc/flightctl/certs/* your_username@remotehost:~/.flightctl/certs/
    scp bin/agent/etc/flightctl/config.yaml your_username@remotehost:~/.flightctl/agent.yaml
    scp ~/.config/flightctl/client.yaml your_username@remotehost:~/.config/flightctl/client.yaml
    scp bin/devicesimulator your_username@remotehost:~/bin/
    # on remote host 
    chmod -R 755 ~/.flightctl/ ~/.config/flightctl ~/bin/devicesimulator

Remote run

On each remote host run devicesimulator with at least two parameters:

  • count - number of devices to simulate on this host
  • initial-device-index - starting index for device name suffix. To simulate 600 devices on 3 hosts run:

    bin/devicesimulator --count=200 # device-00000 to device-00199

    bin/devicesimulator --count=200 --initial-device-index=200 # device-00200 to device-00399

    bin/devicesimulator --count=200 --initial-device-index=400 # device-00400 to device-00599

When running remotely it is important to consider connection limits when running large scale performance tests.

Source IP Considerations for Remote Hosts: When distributing device simulation across multiple remote hosts, each host can benefit from using multiple source IP addresses to avoid ephemeral port exhaustion. This is especially important when running high device counts per host (>10,000 devices).

When adding IPs, care must be taken to ensure they do not conflict with other hosts in the network. The recommended approach is to use the Network Bridge method described above on each remote host. This isolates the IPs and prevents conflicts.

Alternatively, you can add IPs directly to an existing physical interface. This makes the IPs visible to other hosts on the network, so you must ensure they do not cause IP conflicts.

# On each remote host, setup additional IP addresses, ensuring they don't conflict with other hosts.
sudo ip addr add 10.0.1.100/24 dev eth0
sudo ip addr add 10.0.1.101/24 dev eth0

# Run simulator with source IPs on each remote host
bin/devicesimulator --count=20000 --source-ips=10.0.1.100,10.0.1.101 --initial-device-index=0

Troubleshooting

Common Issues

Device Creation Failures

  • Symptom: Devices fail to enroll
  • Solution: Check enrollment service connectivity and certificate validity.

High Resource Usage

  • Symptom: High CPU/memory usage during simulation
  • Solution:
    • Reduce --max-concurrency to limit parallel device creation
    • Increase status-update-interval in agent config
    • Use higher log levels (--log-level=error or --log-level=fatal)

Connection Limits

  • Symptom: "Too many open files" or "Address already in use" errors
  • Solution:
    • Increase file descriptor limits: ulimit -n 65536
    • Tune ephemeral port range: sysctl net.ipv4.ip_local_port_range
    • Distribute load across multiple hosts

/sysroot Errors

  • Symptom: Repeated errors about unable to read /sysroot
  • Solution:
    • Create empty directory: sudo mkdir /sysroot
    • Or ensure the auto-created fleet has appropriate disk monitoring paths

Cleanup

Devices should clean themselves up on graceful exit (ctrl + c), but if they fail and the device previously had applications associated with it, some podman events processes could be lingering. If they are run the following:

    #!/bin/bash
    while true; do
      echo pkill -f -TERM 'podman events'
      sudo pkill -f -TERM 'podman events'
      sleep 30
    done