Add support for input retrieval from CDSE #101
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi Martin, thanks so much for this through work, much appreciated. I think I understand what you are up to with the app and that's fine. Download from CDSP is also fine, I am just relying on ASF webpage for download, which in my opinion does a better job. Regarding the update of the xml files, which version of SNAP do you currently use? Is this then wroking with newer version, and has all functionality been tested against (I still use it for coherence, so that's definitely an important one). Using COG for export is also nice. I actually would like to get rid of GDAL, and only use rasterio, but I think rasterio still does not support vrts the way I am using it. Let me know, |
preprocessing.py now conforms to EO Application Package best practice by interpreting its input not as a direct path to a data file but as the path to a directory containing a STAC catalog containing an item containing an asset giving the path to the actual input. Similarly, after processing, preprocessing.py also writes a STAC catalog describing the output data.
|
Dear Andreas, we still use SNAP 8.0. We did not change that, hoping that this allows to reproduce what you had done in your notebooks. Do you have a hint what may be wrong when we get "products are not consecutive slices" . Is the assumption correct the toolbox makes, that the next granule starts at a time the previous granule stops, without overlap? What can we do to avoid that? Best regards, |
Logging is now done with the standard Python logging library, not with print statements.
preprocessing.py now writes its output to the CWD set on container start-up, as prescribed in OGC EOAP BP §7.2.
- Use custom headers when fetching OTB package to avoid download speed throttling. - Install OST package from version2 branch of bcdev repository. - Add constraints.txt and snap.varfile to build context. - Fetch last commit information via GitHub API to invalidate build cache if package has been updated.
- Add a dry-run parameter. - Update Docker requirement to reference public image on quay.io.
wget progress now uses the dot:giga setting to reduce the amount of noise in the build logs.
Version2
version2 branch has now been merged so we build images from main instead.
get_zip_from_stac has been renamed to get_input_path_from_stac and now also handles STAC catalogues describing unzipped SAFE directories. However, preprocessing.run() still needs to be adapted to deal with these SAFE directories as input.
- Update to version 1.2 to allow use of loadListing and networkAccess options. - Set "loadListing: no_listing" for directory input to prevent errors due to illegally named files in the input directory. - Set "neworkAccess: true" in CommandLineTool requirements to ensure that s1scene can make requests to external services.
This argument controls the branch or commit of the OpenSarToolkit repository which is fetched, making it easier to build locally from a different branch.
Add support for SAFE directory input
proposed changes to CWL - remove cdse and rename workflow ID
jq is not needed for OpenSarToolkit itself, but can be useful if the same Docker image is reused for JSON manipulation tasks in the EOAP CWL file.
For reasons that aren't clear, quay.io builds are failing at the apt install line since jq was added. This commit tries installing jq with an additional apt-get invocation instead.
- Update the geometry entry in the STAC output to be a dictionary including a type key rather than just a list of co-ordinates. - Improve source code formatting.
Output is now written to the result-item subdirectory of the output directory, along with the STAC item itself. The asset link in the STAC item record also points to this new location.
- Add the "title" and "gsd" keys to the STAC asset output. - Tile the dummy output for dry runs to make it COG-compliant, and increase the output image size to make sure that it's big enough to tile.
- Fix an error due to incorrect pystac usage when creating the STAC asset. - Improve the appearance of the dummy image produced by a dry run.
- Set HOME in Dockerfile before using it. - In the dockerfile, set the git ref for the OST repository to version5. - Add the --wipe-cwd argument to the CLI tool invocation in the CWL. - Update the Docker image version tag in the CWL to version5.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Dear developers,
in order to get OpenSarToolkit running I had to make a few updates to the Dockerfile and I had to implement download of input data from CDSE. Neither SciHub nor PEPS exist any more. I also have integrated the pull request by Aleksandr Tulenkov to let it survive a bit longer.
I still struggle with errors reporting "products are not consecutive slices" for more than half of the days of May 2020, the test month in your notebook. Hints how to solve that are welcome.
Best regards,
Martin