Release Testing

Jump to bottom

Aaron Molitor edited this page Feb 21, 2022 · 1 revision

Release Testing

This document will list the set of tests executed prior to the release of a version of FoundationDB.

Types of Release Testing

Correctness Testing (using fdb-joshua)

Regular
Binding Tester
Valgrind Enabled
Sanitizers Enabled

Performance Testing Matrix (using YCSB tests from fdb-kubernetes-tests)

Storage Engines
- SQLite
- RocksDB
- Redwood
Disk Types
- AWS EBS gp3
- AWS EC2 Instance Store
Scalability Tests (NOTE: "Node" definition is TBD)
- 10 Nodes
- 20 Nodes
- 30 Nodes
- etc.

Other Cluster Testing

High Availability configurations
Chaos testing
Backup/Restore Tests
- Backup/Restore performance
- Backup/Restore integrity

Performance Test Metrics

Data Load rate
Data Read rate
Transactional throughput (YCSB workloads A-F plus some custom/hybrid workloads)
Latency Measurements
- Mean Latency
- Tail Latency
Disk Usage Efficiency (Logical TB vs. Physical TB)
CPU Utilization (under load)
Memory Utilization (under load)

Release/Test Report Format

Format will attempt to be consistent
Format will attempt to allow for comparability between releases
Format is subject to change as testing capability, or system performance characteristics change
Test reports must be available to OSS community at time of release