You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<palign="right">(<ahref="#top">back to top</a>)</p>
54
53
55
-
56
54
## Changelog <aname="changelog"></a>
57
-
-**`[2025/02/28]`** NAVSIM v2.0 release (official devkit version for 2025 warm-up phase)
55
+
56
+
-**`[2025/04/09]`** NAVSIM v2.1 release (official devkit version for 2025 warm-up phase)
57
+
- Added new dataset for the [Hugging Face Warmup leaderboard](https://huggingface.co/spaces/AGC2025/e2e-driving-warmup) (see [submission](docs/submission.md))
58
+
59
+
- ⚠️ **IMPORTANT**: To submit to the updated leaderboard, you need to re-download the synthetic dataset (see [download](docs/install.md))
60
+
- Introduced support for two-stage reactive traffic agents (see [traffic simulation](docs/metrics.md))
61
+
- Code refactoring and bug fixes
62
+
-**`[2025/02/28]`** NAVSIM v2.0 release
58
63
- Extends the PDM Score with more metrics and penalties (see [metrics](docs/metrics.md))
59
64
- Adds a new two-stage pseudo closed-loop simulation (see [metrics](docs/metrics.md))
60
65
- Adds support for reactive traffic agent policies (see [traffic simulation](docs/metrics.md))
@@ -82,11 +87,10 @@
82
87
- OpenScene-mini sensor blobs and annotation logs
83
88
- Naive `ConstantVelocity` agent
84
89
85
-
86
90
<palign="right">(<ahref="#top">back to top</a>)</p>
87
91
88
-
89
92
## License and citation <aname="licenseandcitation"></a>
93
+
90
94
All assets and code in this repository are under the [Apache 2.0 license](./LICENSE) unless specified otherwise. The datasets (including nuPlan and OpenScene) inherit their own distribution licenses. Please consider citing our paper and project if they help your research.
91
95
92
96
```BibTeX
@@ -109,7 +113,6 @@ All assets and code in this repository are under the [Apache 2.0 license](./LICE
109
113
110
114
<palign="right">(<ahref="#top">back to top</a>)</p>
Copy file name to clipboardExpand all lines: docs/install.md
+8-4Lines changed: 8 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -34,7 +34,7 @@ You can download the mini, trainval, test, private_test_e2e and warmup_synthetic
34
34
./download_trainval
35
35
./download_test
36
36
./download_private_test_e2e
37
-
./download_warmup_synthetic_scenes
37
+
./download_warmup_two_stage
38
38
```
39
39
40
40
Also, the script `./download_navtrain` can be used to download a small portion of the `trainval` dataset split which is needed for the `navtrain` training split.
@@ -57,12 +57,16 @@ This will download the splits into the download directory. From there, move it t
57
57
| ├── trainval
58
58
| ├── private_test_e2e
59
59
| └── mini
60
-
└── synthetic_scenes
61
-
├── scene_pickles
62
-
└── sythetic_sensor
60
+
└── warmup_two_stage
61
+
├── openscene_meta_datas
62
+
├── sensor_blobs
63
+
├── synthetic_scene_pickles
64
+
└── synthetic_scenes_attributes.csv
63
65
64
66
```
65
67
68
+
⚠️ **IMPORTANT:** If you have already downloaded the data for Navsim V2.0.1 and tried the Hugging Face Leaderboard, please replace the old `"synthetic_scenes"` folder with the new `"warmup_two_stage"` folder. In Navsim V2.1, the traffic agents' policy has been updated, and the old data is no longer compatible.
69
+
66
70
Set the required environment variables, by adding the following to your `~/.bashrc` file
67
71
Based on the structure above, the environment variables need to be defined as:
Copy file name to clipboardExpand all lines: docs/splits.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,7 +1,7 @@
1
1
# Dataset splits vs. filtered training / test splits
2
2
3
3
The NAVSIM framework utilizes several dataset splits for standardized training and evaluating agents.
4
-
All of them use the OpenScene dataset that is divided into the dataset splits `mini`,`trainval`,`test`,`private_test_e2e`, which can all be downloaded separately.
4
+
All of them use the OpenScene dataset that is divided into the dataset splits `mini`,`trainval`,`test`,`warmup_two_stage`, `private_test_e2e`, which can all be downloaded separately.
5
5
6
6
It is possible to run trainings and evaluations directly on these sets (see `Standard` in table below).
7
7
Alternatively, you can run trainings and evaluations on training and validation splits that were filtered for challenging scenarios (see `NAVSIM` in table below), which is the recommended option for producing comparable and competitive results efficiently.
@@ -34,7 +34,7 @@ In Navsim-v1.1, the training/test split can bet set with a single config paramet
34
34
<td>trainval</td>
35
35
<td>Large split for training and validating agents with regular driving recordings. Corresponds to nuPlan and downsampled to 2HZ.</td>
36
36
<td>14GB</td>
37
-
<td>>2000GB</td>
37
+
<td>>2000GB</td>
38
38
<td>
39
39
train_test_split=trainval
40
40
</td>
@@ -78,18 +78,18 @@ In Navsim-v1.1, the training/test split can bet set with a single config paramet
78
78
</tr>
79
79
<tr>
80
80
<td rowspan="2">Competition</td>
81
-
<td>warmup_test_e2e</td>
81
+
<td>warmup_two_stage</td>
82
82
<td>Warmup test split to validate submission on hugging face. Available as a filter for test split.</td>
83
83
<td>-</td>
84
84
<td>-</td>
85
85
<td>
86
-
train_test_split=warmup_test_e2e
86
+
train_test_split=warmup_two_stage
87
87
</td>
88
88
</tr>
89
89
<tr>
90
90
<td>private_test_e2e</td>
91
91
<td>Private test split for the challenge leaderboard on hugging face.</td>
92
-
<td><1GB</td>
92
+
<td><1GB</td>
93
93
<td>25GB</td>
94
94
<td>
95
95
train_test_split=private_test_e2e
@@ -105,7 +105,7 @@ The standard splits `trainval`, `test`, and `mini` are from the OpenScene datase
105
105
106
106
NAVSIM provides a subset and filter of the `trainval` split, called `navtrain`. The `navtrain` split facilitates a standardized training scheme and requires significantly less sensor data storage than `travel` (445GB vs. 2100GB). If your agents don't need historical sensor inputs, you can download `navtrain` without history, which requires 300GB of storage. Note that `navtrain` can be downloaded separately via [download_navtrain.sh](https://github.com/autonomousvision/navsim/blob/main/download/download_navtrain.sh) but still requires access to the `trainval` logs. Similarly, the `navtest` split enables a standardized set for testing agents with a provided scene filter. Both `navtrain` and `navtest` are filtered to increase interesting samples in the sets.
107
107
108
-
For the challenge on Hugging Face, we provide the `warmup_test_e2e` and `private_test_e2e` for the warm-up and challenge track, respectively. Note that `private_test_e2e` requires you to download the data, while `warmup_test_e2e` is a scene filter for the `mini` split.
108
+
For the challenge on Hugging Face, we provide the `warmup_two_stage` and `private_test_e2e` for the warm-up and challenge track, respectively. Note that `private_test_e2e` requires you to download the data, while `warmup_two_stage` is a scene filter for the `test` split.
Copy file name to clipboardExpand all lines: docs/submission.md
+12-9Lines changed: 12 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,24 +7,27 @@ For the [NAVSIM challenge 2025 warmup](https://huggingface.co/spaces/AGC2025/e2e
7
7
### Rules
8
8
9
9
-**Open-source code and models**:
10
+
10
11
- We will periodically (~every 6 months) be removing all entries on the leaderboard which **do not provide associated open-source training and inference code with the corresponding pre-trained checkpoints**. Even if removed for not having this information, an entry can be resubmitted once the code needed for reproducibility is made publicly available.
11
12
- Code must be provided by setting the `TEAM_NAME` variable of the submission file as `"<a href=Link/to/repository>Method name</a>"`. Note that this can also be edited on the leaderboard for an existing submission, if the repo is created (or updated) after the initial submission.
12
-
-**Multi-seed submissions**:
13
-
- Driving policies often differ significantly in performance when re-trained with different network initialization seeds.
14
-
- Therefore, the leaderboard now supports (1) regular single-seed submissions and (2) multi-seed submission, which we **strongly encourage** (with a minimum of 3 training seeds).
15
-
- The maximum, mean and standard deviations of our evaluation metrics will be displayed for multi-seed submissions.
13
+
14
+
### Run Score Locally
15
+
16
+
You can reproduce your test results locally in the [NAVSIM](https://github.com/autonomousvision/navsim/blob/main/docs/install.md) repository, and they should match the results you obtain on Hugging Face. Follow the steps below:
17
+
18
+
1.**Download the dataset** — refer to the [dataset](install.md) for instructions.
19
+
2.**Cache the data** — follow the script `scripts/evaluation/run_metric_caching.sh`, and set `TRAIN_TEST_SPLIT=warmup_two_stage`
20
+
3.**Run the evaluation** — follow the script `scripts/evaluation/run_cv_pdm_score_evaluation.sh` with your own model.
21
+
* If you specified `metric_cache_path` during caching, make sure to use the same path during evaluation.
22
+
* Set `TRAIN_TEST_SPLIT=warmup_two_stage` to ensure that the score matches the one returned by Hugging Face.
16
23
17
24
### Regular warmup submission
18
25
19
26
To submit to a leaderboard you need to create a pickle file that contains a trajectory for each test scenario. NAVSIM provides a script to create such a pickle file.
20
27
21
28
Have a look at `run_cv_create_submission_pickle.sh` in the [NAVSIM](https://github.com/autonomousvision/navsim/blob/main/docs/install.md) repository: this file creates the pickle file for the ConstantVelocity agent. You can run it for your own agent by replacing the `agent` override. **Note that you have to set the variables `TEAM_NAME`, `AUTHORS`, `EMAIL`, `INSTITUTION`, and `COUNTRY` in `run_create_submission_pickle.sh` to generate a valid submission file**
22
29
23
-
You should be able to obtain the same evaluation results as on the server by running the evaluation locally. To do so, use the override `train_test_split=warmup_navsafe_two_stage_extended` when executing the script to run the PDM scoring.
24
-
25
-
### Multi-seed warmup submission
26
-
27
-
For a multi-seed submission, you first have to create individual agents, i.e. trained on different seeds. Consequently, you can merge your entries to a single submission file with the `run_merge_submission_pickles.sh` bash script. Please set the override `train_test_split=warmup_navsafe_two_stage_extended ` to ensure all individual entries contain trajectories for the evaluation.
30
+
You should be able to obtain the same evaluation results as on the server by running the evaluation locally.
Copy file name to clipboardExpand all lines: docs/traffic_agents.md
+11-5Lines changed: 11 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,21 +5,27 @@ NAVSIM v2 introduces support for **reactive traffic agents**, allowing surroundi
5
5
### Available Traffic Agent Policies
6
6
7
7
1.**Log-Replay** (Non-Reactive)
8
-
- Identical to NAVSIM v1, traffic agents strictly follow recorded trajectories without reacting to the ego-vehicle.
9
8
9
+
- Identical to NAVSIM v1, traffic agents strictly follow recorded trajectories without reacting to the ego-vehicle.
10
10
2.**Constant-Velocity** (Debugging Only)
11
+
11
12
- Traffic agents move in a straight line at a fixed velocity, providing a simple baseline for debugging.
13
+
3.**IDM (Intelligent Driver Model)** (Reactive)
12
14
13
-
3.**IDM (Intelligent Driver Model)**
14
15
- Similar to nuPlan, this model simulates traffic agents with more realistic behavior, adjusting speed and spacing based on road conditions.
15
16
- Pedestrians, static objects, and other non-vehicle agents still follow pre-recorded log data.
16
17
17
18
### Selecting a Traffic Agents Policy
18
-
Traffic agent policies can be selected by specifying an override when running the evaluation. For example:
19
19
20
-
`traffic_agents_policy=navsim_IDM_traffic_agents`
20
+
For single-stage simulation, you can specify the traffic agent policy by providing an override when running the evaluation script `navsim/planning/script/run_pdm_score_one_stage.py`.
21
+
22
+
An example can be found in the commented section of the script `run_cv_pdm_score_evaluation.sh`. For instance:
23
+
24
+
`traffic_agents=non_reactive` or `traffic_agents=reactive`
25
+
26
+
This makes it easy to switch between different traffic agent policies depending on your evaluation requirements.
21
27
22
-
This allows to easily switch between different policies depending on your evaluation needs.
28
+
In two-stage simulations (e.g., for Hugging Face submissions), reactive traffic agents are used by default.
23
29
24
30
All available traffic agents policies can be found [here](navsim/planning/script/config/common/traffic_agents_policy.md)
0 commit comments