Add tests for various tiff file types

[kylebarron]

@geospatial-jeff @vincentsarago have thoughts for good ways to test a variety of TIFFs? Something like https://github.com/cogeotiff/cog-fixtures?

[kylebarron]

There are also a bunch of files upstream: https://github.com/image-rs/image-tiff/tree/master/tests

[kylebarron]

@maxrjones Not urgent, but maybe if you have some time down the line, you might be interested in implementing some tests on the Rust side? It doesn't look too hard to port some of the upstream tests like: https://github.com/image-rs/image-tiff/blob/master/tests/decode_bigtiff_images.rs

[vincentsarago]

in rio-tiler I use a pytest fixture https://github.com/cogeotiff/rio-tiler/blob/7db311c4dd0a839d6fb7634d5891979fea456935/tests/conftest.py#L26-L118 and then a matrix https://github.com/cogeotiff/rio-tiler/blob/7db311c4dd0a839d6fb7634d5891979fea456935/tests/benchmarks/benchmarks.py#L78-L81 to create a lot fixtures dataset 😅

[kylebarron]

Progress on the rust side with #28

[weiji14]

Originally posted by @kylebarron in #102 (comment)

yeah I think it would be better to either check in this file into this repo, or perhaps create an async-tiff-fixtures separate repo that we can check data into.

Assuming this file is large, it would be great if we could extract like a one-pixel or very small subset of this file, while maintaining all header metadata the same.

Or, we could find the size of the last IFD and just clip the existing buffer to that region. So all IFD parsing would succeed but any image loading would fail as the image data is past the byte region checked in to the test fixture.

Agree that we can keep some commit some small clipped subsets (<2KB) to the repo, but we should probably plan for larger TIFFs to be included too, either in a separate fixtures repo, or a DVC mirror like what @maxrjones has been doing over at https://www.dagshub.com/maxrjones/virtual-tiff.

[maxrjones]

Agree that we can keep some commit some small clipped subsets (<2KB) to the repo, but we should probably plan for larger TIFFs to be included too, either in a separate fixtures repo, or a DVC mirror like what @maxrjones has been doing over at dagshub.com/maxrjones/virtual-tiff.

Kyle and Jeff asked if I could move the TIFFs in DagsHub to a more accessible location. Since it should be possible to access them from DagsHub over HTTPS, my guess is that they're looking for a location that supports both S3 and HTTPS. Is that accurate for the motivation @kylebarron?

I like how DagsHub also understands most of the TIFFs so that you can see the "additional metadata" (e.g., see https://dagshub.com/maxrjones/virtual-tiff/src/main/tests/dvc/gdal_gcore/minimum_tiff_tags_no_warning.tif) and the versioning of the datasets, which I don't know how to do using DVC separately from DagsHub. My current opinion is that the best solution would be to link a S3 compatible bucket (e.g., on Source coop or OSN) to a DagsHub repository that versions datasets. What do you think about that approach @weiji14?

[weiji14]

I like how DagsHub also understands most of the TIFFs so that you can see the "additional metadata" (e.g., see https://dagshub.com/maxrjones/virtual-tiff/src/main/tests/dvc/gdal_gcore/minimum_tiff_tags_no_warning.tif) and the versioning of the datasets

Oo, that is super nice! It seems to know EXIF and GPS tags too looking at https://dagshub.com/maxrjones/virtual-tiff/src/8a54fdef774be71f42b0f5e2cb66b0262be3cd35/tests/dvc/gdal_gcore/exif_and_gps.tif:

My current opinion is that the best solution would be to link a S3 compatible bucket (e.g., on Source coop or OSN) to a DagsHub repository that versions datasets. What do you think about that approach @weiji14?

Yes, DagsHub comes with its own 100GB of free space, but we can configure it to use an external S3 store instead following https://dagshub.com/docs/integration_guide/dvc/#external-storage-buckets.

[maxrjones]

My current opinion is that the best solution would be to link a S3 compatible bucket (e.g., on Source coop or OSN) to a DagsHub repository that versions datasets. What do you think about that approach @weiji14?

Yes, DagsHubshould comes with its own 100GB of free space, but we can configure it to use an external S3 store following dagshub.com/docs/integration_guide/dvc#external-storage-buckets.

Nice! I asked Rich Signell if we could use the ESIP OSN allocation. Source could be another nice option if they remove the proxy so the tests don't run slow.

[jedsundwall]

The Source bucket is fully open at s3://us-west-2.opendata.source.coop/ – no need to go through the proxy for testing. Would love to have a data product like this in Source. Would be super useful for our own testing / product development needs too.

[maxrjones]

The Source bucket is fully open at s3://us-west-2.opendata.source.coop/ – no need to go through the proxy for testing. Would love to have a data product like this in Source. Would be super useful for our own testing / product development needs too.

That's great to know, thanks @jedsundwall! I just signed up for the beta program to set this up on Source