Skip to content

Commit 5d2cfaa

Browse files
ternausternaus
committed
Finally works
1 parent 6182ce8 commit 5d2cfaa

22 files changed

+4255
-283
lines changed

‎.gitignore

-2
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,4 @@ conda_build/
167167

168168
data/
169169

170-
output/
171-
172170
.venv_*

‎.pre-commit-config.yaml

-4
Original file line numberDiff line numberDiff line change
@@ -58,10 +58,6 @@ repos:
5858
hooks:
5959
- id: codespell
6060
additional_dependencies: ["tomli"]
61-
- repo: https://github.com/igorshubovych/markdownlint-cli
62-
rev: v0.42.0
63-
hooks:
64-
- id: markdownlint
6561
- repo: https://github.com/tox-dev/pyproject-fmt
6662
rev: "v2.4.3"
6763
hooks:

‎README.md

+285-1
Original file line numberDiff line numberDiff line change
@@ -1 +1,285 @@
1-
# benchmark
1+
# Image Augmentation Library Benchmarks
2+
3+
A comprehensive benchmarking suite for comparing the performance of popular image augmentation libraries including [Albumentations](https://albumentations.ai/), [imgaug](https://imgaug.readthedocs.io/en/latest/), [torchvision](https://pytorch.org/vision/stable/index.html), [Kornia](https://kornia.readthedocs.io/en/latest/), and [Augly](https://github.com/facebookresearch/AugLy).
4+
5+
<details>
6+
<summary>Table of Contents</summary>
7+
8+
- [Image Augmentation Library Benchmarks](#image-augmentation-library-benchmarks)
9+
- [Overview](#overview)
10+
- [Benchmark Results](#benchmark-results)
11+
- [System Information](#system-information)
12+
- [Benchmark Parameters](#benchmark-parameters)
13+
- [Library Versions](#library-versions)
14+
- [Performance Comparison](#performance-comparison)
15+
- [Requirements](#requirements)
16+
- [Supported Libraries](#supported-libraries)
17+
- [Notes](#notes)
18+
- [Setup](#setup)
19+
- [Getting Started](#getting-started)
20+
- [Using Your Own Images](#using-your-own-images)
21+
- [Methodology](#methodology)
22+
- [Benchmark Process](#benchmark-process)
23+
- [Running Benchmarks](#running-benchmarks)
24+
- [Single Library](#single-library)
25+
- [Command Line Arguments](#command-line-arguments)
26+
- [All Libraries](#all-libraries)
27+
- [All Libraries Options](#all-libraries-options)
28+
- [Output Structure](#output-structure)
29+
- [Output Format](#output-format)
30+
31+
</details>
32+
33+
## Overview
34+
35+
This benchmark suite measures the throughput and performance characteristics of common image augmentation operations across different libraries. It features:
36+
37+
- Adaptive warmup to ensure stable measurements
38+
- Multiple runs for statistical significance
39+
- Detailed performance metrics and system information
40+
- Thread control settings for consistent single-threaded performance
41+
- Support for multiple image formats and loading methods
42+
43+
## Benchmark Results
44+
45+
### System Information
46+
47+
- Platform: macOS-15.0.1-arm64-arm-64bit
48+
- Processor: arm
49+
- CPU Count: 10
50+
- Python Version: 3.12.7
51+
52+
### Benchmark Parameters
53+
54+
- Number of images: 1000
55+
- Runs per transform: 10
56+
- Max warmup iterations: 1000
57+
58+
### Library Versions
59+
60+
- albumentations: 1.4.20
61+
- augly: 1.0.0
62+
- imgaug: 0.4.0
63+
- kornia: 0.7.3
64+
- torchvision: 0.20.0
65+
66+
### Performance Comparison
67+
68+
| Transform | albumentations<br>1.4.20 | augly<br>1.0.0 | imgaug<br>0.4.0 | kornia<br>0.7.3 | torchvision<br>0.20.0 |
69+
|:------------------|:---------------------------|:-----------------|:------------------|:------------------|:------------------------|
70+
| HorizontalFlip | **8325 ± 955** | 4807 ± 818 | 5585 ± 1146 | 390 ± 106 | 875 ± 69 |
71+
| VerticalFlip | **20493 ± 1134** | 9153 ± 1291 | 10390 ± 290 | 1212 ± 402 | 3131 ± 61 |
72+
| Rotate | **1272 ± 12** | 1119 ± 41 | 1054 ± 96 | 143 ± 11 | 147 ± 6 |
73+
| Affine | **967 ± 3** | - | 802 ± 55 | 147 ± 9 | 128 ± 6 |
74+
| Equalize | **961 ± 4** | - | 540 ± 39 | 152 ± 19 | 414 ± 64 |
75+
| RandomCrop80 | **118946 ± 741** | 25272 ± 1822 | 11009 ± 404 | 1510 ± 230 | 22499 ± 5532 |
76+
| ShiftRGB | **1873 ± 252** | - | 1563 ± 195 | - | - |
77+
| Resize | **2365 ± 153** | 611 ± 78 | 1699 ± 105 | 232 ± 24 | 166 ± 9 |
78+
| RandomGamma | **8608 ± 220** | - | 2315 ± 151 | 108 ± 13 | - |
79+
| Grayscale | **3050 ± 597** | 2720 ± 932 | 1670 ± 56 | 289 ± 75 | 1626 ± 156 |
80+
| RandomPerspective | 410 ± 20 | - | **537 ± 41** | 86 ± 11 | 96 ± 4 |
81+
| GaussianBlur | **1734 ± 204** | 242 ± 4 | 1047 ± 122 | 176 ± 18 | 73 ± 6 |
82+
| MedianBlur | **862 ± 30** | - | 814 ± 71 | 5 ± 0 | - |
83+
| MotionBlur | **2975 ± 52** | - | 583 ± 52 | 73 ± 2 | - |
84+
| Posterize | **5214 ± 101** | - | 2112 ± 281 | 430 ± 49 | 3063 ± 116 |
85+
| JpegCompression | **845 ± 61** | 778 ± 5 | 413 ± 29 | 71 ± 3 | 617 ± 24 |
86+
| GaussianNoise | 147 ± 10 | 67 ± 2 | **203 ± 10** | 75 ± 1 | - |
87+
| Elastic | 171 ± 15 | - | **227 ± 17** | 1 ± 0 | 2 ± 0 |
88+
| ColorJitter | **536 ± 41** | 255 ± 13 | - | 55 ± 18 | 45 ± 2 |
89+
| Brightness | **4443 ± 84** | 1163 ± 86 | - | 472 ± 101 | - |
90+
| Contrast | **4398 ± 143** | 736 ± 79 | - | 425 ± 52 | - |
91+
| Blur | **4816 ± 59** | 246 ± 3 | - | - | - |
92+
| RandomResizedCrop | **2952 ± 24** | - | - | 287 ± 58 | 466 ± 30 |
93+
| Normalize | **1016 ± 84** | - | - | 626 ± 40 | 422 ± 64 |
94+
| PlankianJitter | **1844 ± 208** | - | - | 813 ± 211 | - |
95+
| Clahe | **423 ± 10** | - | - | 94 ± 9 | - |
96+
| CoarseDropout | **11288 ± 609** | - | - | 536 ± 87 | - |
97+
98+
99+
## Requirements
100+
101+
The benchmark automatically creates isolated virtual environments for each library and installs the necessary dependencies. Base requirements:
102+
103+
- Python 3.10+
104+
- uv (for fast package installation)
105+
- Disk space for virtual environments
106+
- Image dataset in a supported format (JPEG, PNG)
107+
108+
## Supported Libraries
109+
110+
- [Albumentations](https://albumentations.ai/)
111+
- [imgaug](https://imgaug.readthedocs.io/en/latest/)
112+
- [torchvision](https://pytorch.org/vision/stable/index.html)
113+
- [Kornia](https://kornia.readthedocs.io/en/latest/)
114+
- [Augly](https://github.com/facebookresearch/AugLy)
115+
116+
Each library's specific dependencies are managed through separate requirements files in the `requirements/` directory.
117+
118+
## Notes
119+
120+
- The benchmark prioritizes consistent measurement over raw speed by enforcing single-threaded execution
121+
- Early stopping mechanisms prevent excessive time spent on slow transforms
122+
- Variance stability checks ensure meaningful measurements
123+
- System information and thread settings are captured to aid in reproducibility
124+
125+
## Setup
126+
127+
### Getting Started
128+
129+
For testing and comparison purposes, you can use the ImageNet validation set:
130+
131+
```bash
132+
wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar
133+
tar -xf ILSVRC2012_img_val.tar -C /path/to/your/target/directory
134+
```
135+
136+
### Using Your Own Images
137+
138+
While the ImageNet validation set provides a standardized benchmark, we strongly recommend running the benchmarks on your own dataset that matches your use case:
139+
140+
- Use images that are representative of your actual workload
141+
- Consider image sizes and formats you typically work with
142+
- Include edge cases specific to your application
143+
144+
This will give you more relevant performance metrics for your specific use case, as:
145+
146+
- Different image sizes can significantly impact performance
147+
- Some transforms may perform differently on different types of images
148+
- Your specific image characteristics might favor certain libraries over others
149+
150+
## Methodology
151+
152+
### Benchmark Process
153+
154+
1. **Image Loading**: Images are loaded using library-specific loaders to ensure optimal format compatibility:
155+
- OpenCV (BGR → RGB) for Albumentations and imgaug
156+
- torchvision for PyTorch-based operations
157+
- PIL for augly
158+
- Normalized tensors for Kornia
159+
160+
2. **Warmup Phase**:
161+
- Performs adaptive warmup until performance variance stabilizes
162+
- Uses configurable parameters for stability detection
163+
- Implements early stopping for slow transforms
164+
- Maximum time limits prevent hanging on problematic transforms
165+
166+
3. **Measurement Phase**:
167+
- Multiple runs of each transform
168+
- Measures throughput (images/second)
169+
- Calculates statistical metrics (median, standard deviation)
170+
171+
4. **Environment Control**:
172+
- Forces single-threaded execution across libraries
173+
- Captures detailed system information and library versions
174+
- Monitors thread settings for various numerical libraries
175+
176+
## Running Benchmarks
177+
178+
### Single Library
179+
180+
To benchmark a single library:
181+
182+
```bash
183+
./benchmark/run_single.sh -l albumentations -d /path/to/images -o /path/to/output
184+
```
185+
186+
#### Command Line Arguments
187+
188+
```bash
189+
Usage: run_single.sh -l LIBRARY -d DATA_DIR -o OUTPUT_DIR [-n NUM_IMAGES] [-r NUM_RUNS]
190+
[--max-warmup MAX_WARMUP] [--warmup-window WINDOW]
191+
[--warmup-threshold THRESHOLD] [--min-warmup-windows MIN_WINDOWS]
192+
Required arguments:
193+
-l LIBRARY Library to benchmark (albumentations, imgaug, torchvision, kornia, augly)
194+
-d DATA_DIR Directory containing images
195+
-o OUTPUT_DIR Directory for output files
196+
Optional arguments:
197+
-n NUM_IMAGES Number of images to process (default: 1000)
198+
-r NUM_RUNS Number of benchmark runs (default: 5)
199+
--max-warmup Maximum warmup iterations (default: 5000)
200+
--warmup-window Window size for variance check (default: 5)
201+
--warmup-threshold Variance stability threshold (default: 0.05)
202+
--min-warmup-windows Minimum windows to check (default: 3)
203+
```
204+
205+
### All Libraries
206+
207+
To run benchmarks for all supported libraries and generate a comparison:
208+
209+
```bash
210+
./run_all.sh -d /path/to/images -o /path/to/output
211+
```
212+
213+
#### All Libraries Options
214+
215+
```bash
216+
Usage: run_all.sh -d DATA_DIR -o OUTPUT_DIR [-n NUM_IMAGES] [-r NUM_RUNS]
217+
[--max-warmup MAX_WARMUP] [--warmup-window WINDOW]
218+
[--warmup-threshold THRESHOLD] [--min-warmup-windows MIN_WINDOWS]
219+
Required arguments:
220+
-d DATA_DIR Directory containing images
221+
-o OUTPUT_DIR Directory for output files
222+
Optional arguments:
223+
-n NUM_IMAGES Number of images to process (default: 2000)
224+
-r NUM_RUNS Number of benchmark runs (default: 5)
225+
--max-warmup Maximum warmup iterations (default: 1000)
226+
--warmup-window Window size for variance check (default: 5)
227+
--warmup-threshold Variance stability threshold (default: 0.05)
228+
--min-warmup-windows Minimum windows to check (default: 3)
229+
```
230+
231+
The `run_all.sh` script will:
232+
233+
1. Run benchmarks for each library ([albumentations](https://albumentations.ai/), [imgaug](https://imgaug.readthedocs.io/en/latest/), [torchvision](https://pytorch.org/vision/stable/index.html), [kornia](https://kornia.readthedocs.io/en/latest/), [augly](https://github.com/facebookresearch/AugLy))
234+
2. Save individual results as JSON files in the output directory
235+
3. Generate a comparison CSV file combining results from all libraries
236+
237+
### Output Structure
238+
239+
```tree
240+
output_directory/
241+
├── albumentations_results.json
242+
├── imgaug_results.json
243+
├── torchvision_results.json
244+
├── kornia_results.json
245+
└── augly_results.json
246+
```
247+
248+
When running all benchmarks, the output directory will contain:
249+
250+
### Output Format
251+
252+
The benchmark produces a JSON file containing:
253+
254+
```json
255+
{
256+
"metadata": {
257+
"system_info": {
258+
"python_version": "...",
259+
"platform": "...",
260+
"processor": "...",
261+
"cpu_count": "...",
262+
"timestamp": "..."
263+
},
264+
"library_versions": {...},
265+
"thread_settings": {...},
266+
"benchmark_params": {...}
267+
},
268+
"results": {
269+
"transform_name": {
270+
"supported": true,
271+
"warmup_iterations": 100,
272+
"throughputs": [...],
273+
"median_throughput": 123.45,
274+
"std_throughput": 1.23,
275+
"times": [...],
276+
"mean_time": 0.123,
277+
"std_time": 0.001,
278+
"variance_stable": true,
279+
"early_stopped": false,
280+
"early_stop_reason": null
281+
}
282+
// ... results for other transforms
283+
}
284+
}
285+
```

0 commit comments

Comments
 (0)