Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SDAP-484 - Cloud-optimized GeoTIFF support #307

Draft
wants to merge 139 commits into
base: develop
Choose a base branch
from
Draft
Changes from 2 commits
Commits
Show all changes
139 commits
Select commit Hold shift + click to select a range
6b1b7cc
Separated NTS backends
Jun 29, 2023
4f3f611
n/a
Jul 5, 2023
e32d5ad
More nts backend stuff
Jul 6, 2023
ccc0de4
Working(?) np backend
Jul 10, 2023
743fb1d
Merge branch 'apache:master' into SDAP-472-gridded-zarr
RKuttruff Jul 10, 2023
b77aa11
Working(?) np backend
Jul 10, 2023
4ccec2e
gitignore ini
Jul 10, 2023
736a44e
ASF headers
Jul 10, 2023
70bdab1
First functioning test of 2 simultaneous backends
Jul 11, 2023
f3981cd
Removed accidentally committed ini files
Jul 12, 2023
26f6220
Working zarr backend ds list
Jul 12, 2023
91de6ef
Capture and handle NTS requests routed to backend that doesn't (yet) …
Jul 12, 2023
df23919
analysis setup fails to find VERSION.txt when building locally
Jul 12, 2023
07404f0
Implemented more NTS functions in zarr backend
Jul 12, 2023
72888aa
Added misc backend time metrics record field in NCSH
Jul 12, 2023
1c4a0e4
fixes
Jul 13, 2023
0a7cd7f
Dynamic dataset management
Jul 17, 2023
c8e7dbb
Dynamic dataset management
Jul 18, 2023
e78f7ad
Dataset management
Jul 20, 2023
a84d77e
Timeseriesspark support
Jul 27, 2023
53190e2
Update backend dict on dataset mgmt query
Jul 31, 2023
2e7a0dc
Fixes and improvements
Jul 31, 2023
0869375
Adapted matchup to work with zarr backends
Jul 31, 2023
c156826
Merge branch 'apache:master' into SDAP-472-gridded-zarr
RKuttruff Jul 31, 2023
1eb680b
Zarr support
Aug 1, 2023
0aef0f1
DDAS adjustments
Aug 2, 2023
42b912e
find_tile_by_polygon_and_most_recent_day_of_year impl
Aug 3, 2023
1559fba
Don't sel by time if neither max nor min time are given
Aug 8, 2023
2bb52af
Fix not calling partial when needed
Aug 15, 2023
f9dc2ae
Pinned s3fs and fsspec versions
Aug 18, 2023
a6f602d
Fixed some dependencies to ensure image builds properly + s3fs works
Aug 18, 2023
1a451eb
Config override for backends
Aug 21, 2023
6f8f7b1
Deps update
Aug 21, 2023
5baf9ec
Merge remote-tracking branch 'RKuttruff/master' into SDAP-472-gridded…
Aug 21, 2023
8cc9d5d
Merge branch 'apache:master' into SDAP-472-gridded-zarr
RKuttruff Aug 22, 2023
e693d27
Merge branch 'apache:master' into SDAP-484-CoG
RKuttruff Aug 22, 2023
492be4b
Merge branch 'apache:master' into SDAP-472-gridded-zarr
RKuttruff Aug 23, 2023
8a9283a
Merge branch 'apache:master' into SDAP-484-CoG
RKuttruff Aug 23, 2023
483ad9f
Add metadata from Zarr collection to /list
Aug 31, 2023
e1dec65
CoG backend start
Sep 1, 2023
bb2fa00
Start of CoG work
Sep 5, 2023
d57a531
rioxarray dep
Sep 5, 2023
3ad5530
CoG URL work and some Solr queries
Sep 6, 2023
4b24ec3
Merge branch 'apache:master' into SDAP-472-gridded-zarr
RKuttruff Sep 6, 2023
87f7bc0
Merge branch 'apache:master' into SDAP-484-CoG
RKuttruff Sep 6, 2023
6712c82
Merge remote-tracking branch 'RKuttruff/SDAP-484-CoG' into SDAP-484-CoG
Sep 6, 2023
aee843b
More functions implemented
Sep 7, 2023
c7b938d
More functions implemented
Sep 7, 2023
9d2605b
Merge remote-tracking branch 'RKuttruff/SDAP-484-CoG' into SDAP-484-CoG
Sep 7, 2023
6077ac2
Merge branch 'apache:master' into SDAP-472-gridded-zarr
RKuttruff Sep 7, 2023
a43db23
Merge branch 'apache:master' into SDAP-484-CoG
RKuttruff Sep 7, 2023
d51cf63
More CoG stuff
Sep 11, 2023
0d3c0fc
Merge branch 'apache:master' into SDAP-472-gridded-zarr
RKuttruff Sep 14, 2023
f80496c
Merge branch 'apache:master' into SDAP-484-CoG
RKuttruff Sep 14, 2023
8d51337
Merge branch 'apache:master' into SDAP-472-gridded-zarr
RKuttruff Sep 14, 2023
d26f4a2
Merge branch 'apache:master' into SDAP-484-CoG
RKuttruff Sep 14, 2023
f5750c3
Zarr: Probe lat order and flip if necessary
Sep 14, 2023
b97677e
Zarr: Probe lat order and flip if necessary
Sep 14, 2023
4cdb485
Fixes for subsetting
Sep 18, 2023
1ee25c2
Warnings for geo subsetting
Sep 20, 2023
7fc260a
Strip quotes from variable names
Sep 20, 2023
62564f7
Strip quotes from variable names
Sep 20, 2023
b5a223b
Fixed find_tile_by_id not routing to correct backend
Sep 21, 2023
06504e0
Ensure geotiffs are sorted by dt
Sep 21, 2023
b5df944
removed resultSizeLimit param from matchup
stephenykp Sep 25, 2023
5e0fbb2
Add # of primaries/avergae secondaries to job output
stephenykp Sep 25, 2023
fbad6b7
rename to executionId
stephenykp Sep 25, 2023
e0a5999
update changelog
stephenykp Sep 25, 2023
8942afc
add totalSecondaryMatched field to /job output
stephenykp Sep 29, 2023
dd73036
num unique secondaries addition
stephenykp Sep 29, 2023
a7beb85
Reenabled TIFF geo subsetting
Oct 3, 2023
741d4b6
Ensure latitudes are in ascending order
Oct 3, 2023
5003420
Catch and report algs that are unsupported by ds backend
Oct 9, 2023
15fce2e
Catch and report algs that are unsupported by ds backend
Oct 9, 2023
d0742ad
Code for opening tiffs in S3
Oct 12, 2023
db68d4f
updated docs to use correct sea_water_temperature param name
stephenykp Oct 13, 2023
5e77bae
Merge remote-tracking branch 'origin/master' into SDAP-484-CoG
Oct 17, 2023
29f556f
Merge branch 'apache:master' into SDAP-484-CoG
RKuttruff Oct 26, 2023
7e11a4c
Merge remote-tracking branch 'origin' into SDAP-493
stephenykp Nov 1, 2023
a8be9b8
bugfix
stephenykp Nov 1, 2023
62de867
fix division by zero bug
stephenykp Nov 6, 2023
972f3dd
add params to dataset management handler classes
Nov 8, 2023
9f0a107
Merge remote-tracking branch 'origin/master' into SDAP-472-gridded-zarr
Nov 8, 2023
831ca37
add page number to default filename for matchup output
stephenykp Nov 16, 2023
4ab2f9b
pagination improvements
stephenykp Nov 16, 2023
3677c11
removed debugging line
stephenykp Nov 16, 2023
86f1348
changelog
stephenykp Nov 16, 2023
1e8cc4e
Update helm cassandra dependency (#289)
RKuttruff Nov 27, 2023
32ca3d7
stac catalog
stephenykp Jan 5, 2024
3563ae9
Updated openapi spec
stephenykp Jan 6, 2024
0691d87
move stac endpoints to matchup tag in openapi spec
stephenykp Jan 6, 2024
e02fc78
SDAP-507 - Changes to remove geos sub-dependency
Jan 11, 2024
51231ca
SDAP-507 - Changelog
Jan 11, 2024
5c75573
SDAP-507 - Changes to remove geos sub-dependency
Jan 11, 2024
7f717c0
SDAP-507 - Changelog
Jan 11, 2024
2b6efa6
Merge remote-tracking branch 'origin/SDAP-507' into SDAP-507
Jan 11, 2024
9779f40
delete instead of comment out
Jan 19, 2024
3e700b8
Merge branch 'SDAP-500' into SDAP-499
stephenykp Jan 19, 2024
9378760
Revert "Update helm cassandra dependency (#289)"
stephenykp Jan 19, 2024
092c87b
Merge branch 'SDAP-499' into SDAP-506
stephenykp Jan 19, 2024
e6730eb
Merge branch 'release/1.2.0' into SDAP-506
stephenykp Jan 19, 2024
3aafb7e
Merge remote-tracking branch 'origin/SDAP-506' into SDAP-472-gridded-…
Jan 19, 2024
5303146
deleted disabled endpoint files
Jan 19, 2024
d6a75e3
Merge branch 'SDAP-507' into SDAP-472-gridded-zarr
Jan 19, 2024
2ed29fd
Merge branch 'develop' into SDAP-493
stephenykp Jan 19, 2024
2a340dc
Merge branch 'SDAP-493' into SDAP-500
stephenykp Jan 19, 2024
718067c
Merge branch 'SDAP-500' into SDAP-499
stephenykp Jan 19, 2024
681ba5a
Merge branch 'SDAP-499' into SDAP-506
stephenykp Jan 19, 2024
935000b
Merge branch 'release/1.2.0' into SDAP-472-gridded-zarr
Jan 19, 2024
ee5e5c8
fix bug where still-running jobs failed /job endpoint due to missing …
stephenykp Jan 25, 2024
0f388a3
Merge branch 'release/1.2.0' into SDAP-506
stephenykp Jan 25, 2024
6bd5f0e
Merge remote-tracking branch 'origin/SDAP-506' into SDAP-472-gridded-…
Jan 29, 2024
639d7d7
Update .asf.yaml (#293)
RKuttruff Feb 1, 2024
5937db1
Merge branch 'apache:master' into SDAP-472-gridded-zarr
RKuttruff Feb 1, 2024
3050ef3
Merge remote-tracking branch 'origin/develop' into SDAP-472-gridded-zarr
Feb 1, 2024
3875f2d
Moved changelog entries
Feb 1, 2024
d989c66
SDAP-472 changelog entries
Feb 1, 2024
2c22be6
Merge branch 'SDAP-472-gridded-zarr' into SDAP-484-CoG
Feb 6, 2024
f766df1
Merge branch 'develop' into SDAP-472-gridded-zarr
Feb 8, 2024
3285c1e
Merge branch 'SDAP-472-gridded-zarr' into SDAP-484-CoG
Feb 12, 2024
ec2ed11
pyproj requirement
Feb 12, 2024
1392983
CoG: some minor fixes for docker build &c
Feb 13, 2024
1ce1732
Merge branch 'develop' into SDAP-472-gridded-zarr
RKuttruff Mar 6, 2024
42530a2
Merge branch 'SDAP-472-gridded-zarr' into SDAP-484-CoG
Apr 1, 2024
c1c4c49
Merge remote-tracking branch 'origin/develop' into SDAP-484-CoG
Apr 1, 2024
168ae73
Dependencies update to poetry
Apr 1, 2024
cefa8ab
Handling of GeoTIFFs with non-float dtypes
Apr 2, 2024
2800569
Improved opening of GeoTIFFs to avoid mask & scale issues with dtype
Apr 16, 2024
6b6bb75
Merge branch 'develop' into SDAP-484-CoG
May 6, 2024
d1fb37c
Merge branch 'develop' into SDAP-484-CoG
Jun 10, 2024
18bbfea
Poetry re-lock
Jun 10, 2024
9a11bad
Fix bad merge
Jun 10, 2024
d44e530
Merge branch 'apache:develop' into SDAP-484-CoG
RKuttruff Jun 25, 2024
f41ebe6
Merge remote-tracking branch 'origin/develop' into SDAP-484-CoG
Jul 10, 2024
c7e6e82
poetry lock
Jul 10, 2024
606e13f
Merge remote-tracking branch 'origin/develop' into SDAP-484-CoG
Oct 8, 2024
f2e5cf1
Poetry re-lock
Oct 8, 2024
0cb9c83
Update backend for new NTS method
Oct 8, 2024
1e746f8
Merge branch 'apache:develop' into SDAP-484-CoG
RKuttruff Oct 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- SDAP-506:
- Added STAC Catalog endpoint for matchup outputs
- SDAP-508: Added spatial extents to the satellite dataset entries in `/list` and `/cdmslist`
- SDAP-472:
- Support for Zarr backend (gridded data only)
- Dataset management endpoints for Zarr datasets