Update to docker image 0.0.2

README.md

@@ -62,7 +62,7 @@ Users should first ensure they have a functioning docker installation.
 ### Pull container
 
 The docker image can be downloaded from docker hub using the following command:
-`docker pull eawag/sencast:0.0.1`
+`docker pull eawag/sencast:0.0.2`
 
 ### Run Tests
 
@@ -72,7 +72,7 @@ functioning of the processors. **This must be run from inside the sencast reposi
 The option `-v /DIAS:/DIAS` maps the input/ output folders to a location outside the container. This should be updated to 
 the appropriate location, e.g. `-v /home/user/DIAS:/DIAS`
 
-`docker run -v /DIAS:/DIAS -v $(pwd):/sencast --rm -it eawag/sencast:0.0.1 -e docker.ini -t`
+`docker run -v /DIAS:/DIAS -v $(pwd):/sencast --rm -it eawag/sencast:0.0.2 -e docker.ini -t`
 
 `-e` name of the environment file in `sencast/environments`
 `-t` flag to indicate a test should be run 
@@ -81,16 +81,16 @@ the appropriate location, e.g. `-v /home/user/DIAS:/DIAS`
 
 In order to run a parameters file it can be passed to the command as follows using the `-p` flag.
 
-`docker run -v /DIAS:/DIAS -v $(pwd):/sencast --rm -it eawag/sencast:0.0.1 -e docker.ini -p example.ini`
+`docker run -v /DIAS:/DIAS -v $(pwd):/sencast --rm -it eawag/sencast:0.0.2 -e docker.ini -p example.ini`
 
 `-p` name of the parameter file in `sencast/parameters`
 
 ### Run Interactive Container
 
 Sometimes it is desirable to interact directly with the container, this can be achieved with the following command:
 
-`docker run -v /DIAS:/DIAS -v $(pwd):/sencast --rm -it --entrypoint /bin/bash eawag/sencast:0.0.1`
+`docker run -v /DIAS:/DIAS -v $(pwd):/sencast --rm -it --entrypoint /bin/bash eawag/sencast:0.0.2`
 
 ### Locally build container
 
-`docker build -t eawag/sencast:0.0.1 .`
+`docker build -t eawag/sencast:0.0.2 .`

docs/_build/html/_sources/cscs.rst.txt

@@ -72,7 +72,7 @@ Pull the image you want from dockerhub:
 
 .. code-block:: bash
 
-   srun -C mc -A em09 sarus pull --login eawag/sencast:0.0.1
+   srun -C mc -A em09 sarus pull --login eawag/sencast:0.0.2
 
 then enter your credentials for the repository (There is no prompt)
 
@@ -118,7 +118,7 @@ Create a submission script containing the following (adjust details to match you
    export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
    module load daint-mc
    module load sarus
-   image='eawag/sencast:0.0.1'
+   image='eawag/sencast:0.0.2'
    envvars='docker.ini'
    params='parameters.ini'
    filepath="${SCRATCH}/DIAS"

docs/_build/html/_sources/index.rst.txt

@@ -151,7 +151,7 @@ The docker image can be downloaded from docker hub using the following command:
 
 .. code-block:: bash
 
-   docker pull eawag/sencast:0.0.1
+   docker pull eawag/sencast:0.0.2
 
 Run Tests
 ~~~~~~~~~~~
@@ -164,7 +164,7 @@ the appropriate location, e.g. `-v /home/user/DIAS:/DIAS`
 
 .. code-block:: bash
 
-   docker run -v /DIAS:/DIAS -v $(pwd):/sencast --rm -it eawag/sencast:0.0.1 -e docker.ini -t
+   docker run -v /DIAS:/DIAS -v $(pwd):/sencast --rm -it eawag/sencast:0.0.2 -e docker.ini -t
 
 `-e` name of the environment file in `sencast/environments`
 `-t` flag to indicate a test should be run
@@ -176,7 +176,7 @@ In order to run a parameters file it can be passed to the command as follows usi
 
 .. code-block:: bash
 
-   docker run -v /DIAS:/DIAS -v $(pwd):/sencast --rm -it eawag/sencast:0.0.1 -e docker.ini -p example.ini
+   docker run -v /DIAS:/DIAS -v $(pwd):/sencast --rm -it eawag/sencast:0.0.2 -e docker.ini -p example.ini
 
 `-p` name of the parameter file in `sencast/parameters`
 
@@ -187,14 +187,14 @@ Sometimes it is desirable to interact directly with the container, this can be a
 
 .. code-block:: bash
 
-   docker run -v /DIAS:/DIAS -v $(pwd):/sencast --rm -it --entrypoint /bin/bash eawag/sencast:0.0.1
+   docker run -v /DIAS:/DIAS -v $(pwd):/sencast --rm -it --entrypoint /bin/bash eawag/sencast:0.0.2
 
 Locally build container
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. code-block:: bash
 
-   docker build -t eawag/sencast:0.0.1 .
+   docker build -t eawag/sencast:0.0.2 .
 
 
 .. toctree::

docs/_build/html/_sources/install/linux.rst.txt

@@ -41,36 +41,16 @@ First, uninstall all old versions of SNAP and remove associated data. You may ne
 
     $ cd ~
     $ curl -O http://step.esa.int/downloads/9.0/installers/esa-snap_all_unix_9_0_0.sh && chmod 755 esa-snap_all_unix_9_0_0.sh && bash esa-snap_all_unix_9_0_0.sh && rm esa-snap_all_unix_9_0_0.sh
-        [o, Enter]
-        [1, Enter]
-        [Enter]
-        [Enter]
-        [n, Enter]
-        [n, Enter]
-        [n, Enter]
-    $ echo "export SNAP_HOME=/home/username/snap" >> ~/.bashrc
-    $ export SNAP_HOME=/home/username/snap
-    $ $SNAP_HOME/bin/snap --nosplash --nogui --modules --update-all
-    $ $SNAP_HOME/bin/snap --nosplash --nogui --modules --install org.esa.snap.idepix.core org.esa.snap.idepix.probav org.esa.snap.idepix.modis org.esa.snap.idepix.spotvgt org.esa.snap.idepix.landsat8 org.esa.snap.idepix.viirs org.esa.snap.idepix.olci org.esa.snap.idepix.seawifs org.esa.snap.idepix.meris org.esa.snap.idepix.s2msi
-    $ echo "#SNAP configuration 's3tbx'" >> ~/.snap/etc/s3tbx.properties
-    $ echo "#Fri Mar 27 12:55:00 CET 2020" >> ~/.snap/etc/s3tbx.properties
-    $ echo "s3tbx.reader.olci.pixelGeoCoding=true" >> ~/.snap/etc/s3tbx.properties
-    $ echo "s3tbx.reader.meris.pixelGeoCoding=true" >> ~/.snap/etc/s3tbx.properties
-    $ echo "s3tbx.reader.slstrl1b.pixelGeoCodings=true" >> ~/.snap/etc/s3tbx.properties
 
-Note: if you encounter any strange error message with X11GraphicsEnvironment, try unsetting the DISPLAY variable (and don't question why)
+This will launch a interactive window where you can install the SNAP software.
 
-Note: there are many strange error messages, but it seems to work in the end when updating and installing plugins
+You need to add the path to gpt to the environment file it is typically something like /home/username/snap/bin/gpt
 
-To remove warning "WARNING: org.esa.snap.dataio.netcdf.util.MetadataUtils: Missing configuration property ‘snap.dataio.netcdf.metadataElementLimit’. Using default (100)."::
-
-    $ echo "" >> $SNAP_HOME/etc/snap.properties
-    $ echo "# NetCDF options" >> $SNAP_HOME/etc/snap.properties
-    $ echo "snap.dataio.netcdf.metadataElementLimit=10000" >> $SNAP_HOME/etc/snap.properties
-
-To remove warning "SEVERE: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: The environment variable LD_LIBRARY_PATH is not set. It must contain the current folder '.'."::
+IDEPIX
+--------
 
-    $ echo "export LD_LIBRARY_PATH=." >> ~/.bashrc
+Idepix is a plugin from SNAP that can be installed by opening SNAP and going Tools -> Plugins. From the available
+plugins download all the Idepix plugins. They will be available in Sencast once you restart SNAP.
 
 POLYMER
 --------
@@ -99,29 +79,6 @@ In shell do following::
 Configure path to l8_angles in your environment file.
 
 
-CDS API
-________
-
-Setup credentials for CDS API::
-
-	$ echo "url: https://cds.climate.copernicus.eu/api/v2" > ~/.cdsapirc
-	$ echo key: [uid]:[api-key] >> ~/.cdsapirc (Note: replace [uid] and [api-key] by your actual credentials, see https://cds.climate.copernicus.eu/api-how-to )
-	$ chmod 600 ~/.cdsapirc
-
-
-NASA Earthdata API
---------------------
-
-Have a NASA Earthdata account ready, otherwise create one: https://urs.earthdata.nasa.gov/
-
-In shell do following::
-
-    $ touch ~/.netrc
-    $ touch ~/.urs_cookies
-    $ echo "machine urs.earthdata.nasa.gov login <earthdata user> password <earthdata password>" >> ~/.netrc
-    $ chmod 0600 ~/.netrc
-
-
 Acolite
 --------
 
@@ -135,6 +92,13 @@ In shell do following::
 Configure your Acolite path in you environment file.
 
 
+OCSMART
+--------
+
+Download the OCSMART linux package from http://www.rtatmocn.com/oc-smart/
+
+Unzip the package somewhere and then add a path the the folder that contains OCSMART.py to your environment file
+
 FLUO
 -----
 

docs/_build/html/adapters/datalakes.html

@@ -91,11 +91,6 @@
 <ul class="current">
 <li class="toctree-l1 current"><a class="current reference internal" href="#">Datalakes</a><ul>
 <li class="toctree-l2"><a class="reference internal" href="#adapters.datalakes.datalakes.apply"><code class="docutils literal notranslate"><span class="pre">apply()</span></code></a></li>
-<li class="toctree-l2"><a class="reference internal" href="#adapters.datalakes.datalakes.convert_nc"><code class="docutils literal notranslate"><span class="pre">convert_nc()</span></code></a></li>
-<li class="toctree-l2"><a class="reference internal" href="#adapters.datalakes.datalakes.convert_valid_pixel_expression"><code class="docutils literal notranslate"><span class="pre">convert_valid_pixel_expression()</span></code></a></li>
-<li class="toctree-l2"><a class="reference internal" href="#adapters.datalakes.datalakes.get_mask_from_geojson"><code class="docutils literal notranslate"><span class="pre">get_mask_from_geojson()</span></code></a></li>
-<li class="toctree-l2"><a class="reference internal" href="#adapters.datalakes.datalakes.parse_bands"><code class="docutils literal notranslate"><span class="pre">parse_bands()</span></code></a></li>
-<li class="toctree-l2"><a class="reference internal" href="#adapters.datalakes.datalakes.upload_directory"><code class="docutils literal notranslate"><span class="pre">upload_directory()</span></code></a></li>
 </ul>
 </li>
 <li class="toctree-l1"><a class="reference internal" href="qlrgb.html">QLRGB</a></li>
@@ -134,15 +129,12 @@
 
   <section id="module-adapters.datalakes.datalakes">
 <span id="datalakes"></span><h1>Datalakes<a class="headerlink" href="#module-adapters.datalakes.datalakes" title="Link to this heading"></a></h1>
-<p>The Datalakes adapter is designed to output specified bands in order to facilitate web visualisation
-in the Datalakes data portal <a class="reference external" href="https://www.datalakes-eawag.ch/">https://www.datalakes-eawag.ch/</a>.</p>
+<p>The Datalakes adapter notifies Datalakes of new data</p>
 <dl class="py function">
 <dt class="sig sig-object py" id="adapters.datalakes.datalakes.apply">
 <span class="sig-prename descclassname"><span class="pre">adapters.datalakes.datalakes.</span></span><span class="sig-name descname"><span class="pre">apply</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">env</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">params</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">l2product_files</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">date</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#adapters.datalakes.datalakes.apply" title="Link to this definition"></a></dt>
-<dd><p>Apply datalakes adapter.
-1. Converts specified band in NetCDF to JSON format
-2. Save files to S3 storage
-3. Hits Datalakes endpoint to inform server of new data</p>
+<dd><p>Apply Datalakes adapter.
+1. Call Datalakes endpoint</p>
 <section id="parameters">
 <h2>Parameters<a class="headerlink" href="#parameters" title="Link to this heading"></a></h2>
 <dl class="simple">
@@ -158,32 +150,6 @@ <h2>Parameters<a class="headerlink" href="#parameters" title="Link to this headi
 </section>
 </dd></dl>
 
-<dl class="py function">
-<dt class="sig sig-object py" id="adapters.datalakes.datalakes.convert_nc">
-<span class="sig-prename descclassname"><span class="pre">adapters.datalakes.datalakes.</span></span><span class="sig-name descname"><span class="pre">convert_nc</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">output_type</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">input_file</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">output_file</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">band</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">decimals</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">band_min</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">band_max</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">satellite</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">date</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">env</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">projection</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">4326</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#adapters.datalakes.datalakes.convert_nc" title="Link to this definition"></a></dt>
-<dd></dd></dl>
-
-<dl class="py function">
-<dt class="sig sig-object py" id="adapters.datalakes.datalakes.convert_valid_pixel_expression">
-<span class="sig-prename descclassname"><span class="pre">adapters.datalakes.datalakes.</span></span><span class="sig-name descname"><span class="pre">convert_valid_pixel_expression</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">vpe</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">variables</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#adapters.datalakes.datalakes.convert_valid_pixel_expression" title="Link to this definition"></a></dt>
-<dd></dd></dl>
-
-<dl class="py function">
-<dt class="sig sig-object py" id="adapters.datalakes.datalakes.get_mask_from_geojson">
-<span class="sig-prename descclassname"><span class="pre">adapters.datalakes.datalakes.</span></span><span class="sig-name descname"><span class="pre">get_mask_from_geojson</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_file</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">geojson_path</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#adapters.datalakes.datalakes.get_mask_from_geojson" title="Link to this definition"></a></dt>
-<dd></dd></dl>
-
-<dl class="py function">
-<dt class="sig sig-object py" id="adapters.datalakes.datalakes.parse_bands">
-<span class="sig-prename descclassname"><span class="pre">adapters.datalakes.datalakes.</span></span><span class="sig-name descname"><span class="pre">parse_bands</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">bands</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#adapters.datalakes.datalakes.parse_bands" title="Link to this definition"></a></dt>
-<dd></dd></dl>
-
-<dl class="py function">
-<dt class="sig sig-object py" id="adapters.datalakes.datalakes.upload_directory">
-<span class="sig-prename descclassname"><span class="pre">adapters.datalakes.datalakes.</span></span><span class="sig-name descname"><span class="pre">upload_directory</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">bucket</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">aws_access_key_id</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">aws_secret_access_key</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">logger</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">failed</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">extension</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#adapters.datalakes.datalakes.upload_directory" title="Link to this definition"></a></dt>
-<dd><p>Upload a file to an S3 bucket</p>
-</dd></dl>
-
 </section>