README.md

Experiments on TAO Dataset

1. Download the data

a. Please follow TAO download instructions.

b. Please also prepare the LVIS dataset.

It is recommended to symlink the dataset root to $QDTrack/data.

If your folder structure is different, you may need to change the corresponding paths in config files.

Our folder structure follows

├── qdtrack
├── tools
├── configs
├── data
    ├── tao
        ├── frames
            ├── train
            ├── val
            ├── test
        ├── annotations
    ├── lvis
        ├── train2017
        ├── annotations    

2. Install the TAO API

pip install git+https://github.com/OceanPang/tao.git

We fork the TAO API to make the logger compatabile with our codebase.

We also print the AP of main classes such as "person" for reference.

3. Generate our annotation files

a. Generate TAO annotation files with 482 classes.

python tools/convert_datasets/tao2coco.py -t ./data/tao/annotations --filter-classes

b. Merge LVIS and COCO training sets.

Use the merge_coco_with_lvis.py script in the offical TAO API.

This operation follows the paper TAO.

cd ${TAP_API}
python ./scripts/detectors/merge_coco_with_lvis.py --lvis ${LVIS_PATH}/annotations/lvis_v0.5_train.json --coco ${COCO_PATH}/annotations/instances_train2017.json --mapping data/coco_to_lvis_synset.json --output-json ${LVIS_PATH}/annotations/lvisv0.5+coco_train.json

You can also get the merged annotation file from Google Drive or Baidu Yun (passcode: rkh2).

4. Pre-train the model on LVIS dataset

a. Pre-train our QDTrack on LVISv0.5+COCO2017 training set.

sh ./tools/dist_train.sh ./configs/tao/qdtrack_frcnn_r101_fpn_24e_lvis.py 8

The detection performance on TAO validation is

AP	AP50	AP75	AP_S	AP_M	AP_L
17.3	29.2	17.4	5.7	13.1	22.1

Here is a checkpoint at Google Drive. and Baidu Yun (passcode: i4rm).

b. Save the runned model to ckpts/tao/**.pth, and modify the configs for TAO accordingly.

5. Fine-tune the model on TAO dataset

sh ./tools/dist_train.sh ./configs/tao/qdtrack_frcnn_r101_fpn_12e_tao_ft.py 8

You can found a trained model at Google Drive and Baidu Yun (passcode: uhjq).

Results on TAO validation set:

AP(50:75)	AP50	AP75	AP_S(50:75)	AP_M (50:75)	AP_L(50:75)
11.2	15.9	6.4	7.9	9.2	14.7