Skip to content

Latest commit

 

History

History
87 lines (65 loc) · 3.03 KB

train_eval.md

File metadata and controls

87 lines (65 loc) · 3.03 KB

Training and Evaluation Guide

Training

To train on the Nuplan Dataset, execute the following command:

sh ./scripts/training/train_nexus_nuplan.sh

To train on the Waymo Open Dataset, execute the following command:

sh ./scripts/training/train_nexus_wod.sh

Evaluation

Waymo Sim Agents Benchmark

Steps to Fix Bugs in Waymo Open Dataset Package

  1. Install the Waymo Open Dataset Package Make sure you have installed the Waymo Open Dataset package by running:

    pip install waymo-open-dataset-tf-2-12-0==1.6.4

  2. Modify the File Edit the file located at /usr/local/lib/python3.9/dist-packages/waymo_open_dataset/wdl_limited/sim_agents_metrics/metrics.py. Change the code at line 47 from:

    config_path = '{pyglib_resource}waymo_open_dataset/wdl_limited/sim_agents_metrics/challenge_2024_config.textproto'.format(pyglib_resource='')

    to:

    import os
# Get the resource path from an environment variable
pyglib_resource = os.getenv('PYGLIB_RESOURCE_PATH', '')

# Construct the full config path
config_path = f'{pyglib_resource}/waymo_open_dataset/wdl_limited/sim_agents_metrics/challenge_config.textproto'

  3. Set Environment Variable Set the environment variable for the resource path by running:

    export PYGLIB_RESOURCE_PATH=/usr/local/lib/python3.9/dist-packages

By following these steps, you will ensure that the configuration path is correctly set and the package functions properly.

Start Evaluation

To evaluate on the Waymo Sim Agents Benchmark, follow these steps:

  1. Running the testing script:

    sh ./scripts/testing/test_nexus_wod_rollouts.sh

  2. Running the offline evaluation script with multiprocessing

    python ./scripts/testing/offline_sim_agents.py --pkl_dir ${YOUR_PKL_DUMP_PATH} --nprocess 32 --output_dir ${OUTPUT_DIR}

  3. Please check ./scripts/testing/offline_sim_agents.py for more details about online submission

Nuplan Metrics(e.g. FDE, ADE, ...) Evaluation

To evaluate specific metrics like FDE, ADE, Collision rate, etc., run the following command:

sh ./scripts/testing/test_nexus_metrics.sh

Checkpoints

We provide several checkpoints for model reproduction. To use a checkpoint, download it and replace the checkpoint path in the bash script:

+checkpoint.ckpt_path=PATH_TO_CHECKPOINT \
+checkpoint.strict=True \

Checkpoint List

We provided the following CKPT:

Model Dataset Checkpoint
Nexus-nuplan NuPlan Hugging Face
Nexus-wod Waymo Motion dataset Hugging Face