SpaceNet2 doesn't work

[calebrob6]

Description

SpaceNet2 dataset doesn't work with real data.

Setup:

mkdir data
cd data
aws s3 cp --no-sign-request s3://spacenet-dataset/spacenet/SN2_buildings/tarballs/SN2_buildings_train_AOI_2_Vegas.tar.gz .
tar xzf SN2_buildings_train_AOI_2_Vegas.tar.gz

Steps to reproduce:

from torchgeo.datasets import SpaceNet2

ds = SpaceNet2(
    root='data/',
    split = 'train',
    aois = [2],
    image = "RGB-PanSharpen",
)

You can also see this by doing less SN2_buildings_train_AOI_2_Vegas.tar.gz which gives:

This file structure is not compatible with what the SpaceNet2 dataset is looking for.

Steps to reproduce

see above

Version

0.6.2

[calebrob6]

I'm pretty sure https://github.com/microsoft/torchgeo/blob/main/torchgeo/datasets/spacenet.py#L277 should just be:

product_glob = os.path.join(self.root, self.directory_glob)

[adamjstewart]

This is frustrating because I still have the data I downloaded 8 months ago when working on #2203 and it is compatible. Which means the data structure changed in the last 8 months, and possibly all of our SpaceNet datasets need to be changed. I really wish AWS had some kind of stable versioned checksummable download option.

Feel free to submit a PR to fix these. You probably don't need to actually download anything to check the directory structure for the other versions, you can just use aws ls.

[calebrob6]

The .tar.gz files I show above match the checksums we list in the dataset. The already un-tarred data on AWS is in a different directory structure though (which seems to be what the dataset object expects). Separately, our _verify method creates directories in self.root regardless of whether you want to download the data or not or whether the dataset doesn't exists.

I'll open a PR to fix SpaceNet2, but because everything is crammed into this super class I'm not sure if it will break others.

[adamjstewart]

Also see #2366 for more SpaceNet woes.