CMR-7333 - support for DMR++ and DMR++ MISSING DATA URLs in UMM-G

* while testing, the revision_format_map.clj test was failing, it apparently is not stable for local builds. This change will more consistenly pass
* updating some comments to make it clear which document type the function is used for
* adding two enums to related URL for granules
* CMR-7333 - API documentation cleanup to render properly in markdown editors

Author: jceaser

ingest-app/docs/api.md

@@ -29,7 +29,7 @@
 (defconfig granule-umm-version
   "Defines the latest granule umm version accepted by ingest - it's the latest official version.
    This environment variable needs to be manually set when newer UMM version becomes official"
-  {:default "1.6.2"})
+  {:default "1.6.3"})
 
 (defconfig variable-umm-version
   "Defines the latest variable umm version accepted by ingest - it's the latest official version.

ingest-app/src/cmr/ingest/config.clj

@@ -1266,6 +1266,7 @@ __Example Query__
      curl "%CMR-ENDPOINT%/autocomplete?q=ice&type[]=platform&type[]=project"
 
 __Example Result with Type Filter__
+
 ```
 HTTP/1.1 200 OK
 Content-Type: application/json; charset=UTF-8
@@ -1832,17 +1833,17 @@ Shapefile upload is only supported using POST with `multipart/form-data` and the
 
 Examples:
 
-  **ESRI Shapefile**
+**ESRI Shapefile**
 
-  curl -XPOST "%CMR-ENDPOINT%/collections" -F "shapefile=@box.zip;type=application/shapefile+zip" -F "provider=PROV1"
+    curl -XPOST "%CMR-ENDPOINT%/collections" -F "shapefile=@box.zip;type=application/shapefile+zip" -F "provider=PROV1"
 
-  **GeoJSON**
+**GeoJSON**
 
-  curl -XPOST "%CMR-ENDPOINT%/collections" -F "shapefile=@box.geojson;type=application/geo+json" -F "provider=PROV1"
+    curl -XPOST "%CMR-ENDPOINT%/collections" -F "shapefile=@box.geojson;type=application/geo+json" -F "provider=PROV1"
 
-  **KML**
+**KML**
 
-  curl -XPOST "%CMR-ENDPOINT%/collections" -F "shapefile=@box.kml;type=application/vnd.google-earth.kml+xml" -F "provider=PROV1"
+    curl -XPOST "%CMR-ENDPOINT%/collections" -F "shapefile=@box.kml;type=application/vnd.google-earth.kml+xml" -F "provider=PROV1"
 
 Internally a WGS 84 Coordinate Reference System (CRS) is used. The system will attempt to transform shapefile geometry that uses a different CRS, but this is not guaranteed to work and the request will be rejected if a suitable transformation is not found.
 
@@ -1853,7 +1854,7 @@ Shapefiles are limited to 5000 points by default. A user using a shapefile with
 
 Example:
 
-  curl -XPOST "%CMR-ENDPOINT%/collections" -F "simplify-shapefile=true"  -F "shapefile=@africa.zip;type=application/shapefile+zip" -F "provider=PROV1"
+    curl -XPOST "%CMR-ENDPOINT%/collections" -F "simplify-shapefile=true"  -F "shapefile=@africa.zip;type=application/shapefile+zip" -F "provider=PROV1"
 
 Note that the simplification process attempts to preserve topology, i.e., the relationship between polygon outer boundaries and holes. The process uses the [Douglas-Peucker algorithm](https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm) and as such may result in geometries with less coverage than the original shapefile and potentially a loss of matching results.
 
@@ -2106,15 +2107,15 @@ The parameters used for searching granules by spatial are the same as the spatia
 
 As with collections, a shapefile can be uploaded to find granules that overlap the shapefile's geometry. See [Find collections by shapefile](#c-shapefile) for more details.
 
-  curl -XPOST "%CMR-ENDPOINT%/granules" -F "shapefile=@box.zip;type=application/shapefile+zip" -F "provider=PROV1"
+    curl -XPOST "%CMR-ENDPOINT%/granules" -F "shapefile=@box.zip;type=application/shapefile+zip" -F "provider=PROV1"
 
 **NOTE**: This is an experimental feature and may not be enabled in all environments.
 
 #### <a name="g-shapefile-simplification"></a> Simplifying shapefiles during granule search
 
 As with collections, an uplodaed shapefile can be simplified by setting the `simplfiy-shapefile` parameter to `true`. See [Simplifying shapefiles during collection search](#c-shapefile-simplification) for more details.
 
-  curl -XPOST "%CMR-ENDPOINT%/granules" -F "simplify-shapefile=true" -F "shapefile=@africa.zip;type=application/shapefile+zip" -F "provider=PROV1"
+    curl -XPOST "%CMR-ENDPOINT%/granules" -F "simplify-shapefile=true" -F "shapefile=@africa.zip;type=application/shapefile+zip" -F "provider=PROV1"
 
 #### <a name="g-orbit-number"></a> Find granules by orbit number
 
@@ -2169,7 +2170,7 @@ Find granules which have revision date within the ranges of datetimes. The datet
 
  Find granules which were created within the ranges of datetimes. The datetime has to be in yyyy-MM-ddTHH:mm:ssZ format. The default is inclusive on the range boundaries.
 
-   curl "%CMR-ENDPOINT%/granules?collection_concept_id=%CMR-EXAMPLE-COLLECTION-ID%&created_at\[\]=2000-01-01T10:00:00Z,2010-03-10T12:00:00Z&created_at\[\]=2015-01-01T10:00:00Z,"
+    curl "%CMR-ENDPOINT%/granules?collection_concept_id=%CMR-EXAMPLE-COLLECTION-ID%&created_at\[\]=2000-01-01T10:00:00Z,2010-03-10T12:00:00Z&created_at\[\]=2015-01-01T10:00:00Z,"
 
 #### <a name="g-production-date"></a> Find granules by production_date
 
@@ -2986,9 +2987,7 @@ Content-Type: application/json; charset=UTF-8
 
 The humanizers report provides a list of fields that have been humanized in CSV format. The report format is: provider, concept id, product short name, product version, original field value, humanized field value.
 
-```
-curl "%CMR-ENDPOINT%/humanizers/report"
-```
+    curl "%CMR-ENDPOINT%/humanizers/report"
 
 Note that this report is currently generated every 24 hours with the expectation that this more than satisfies weekly usage needs.
 
@@ -3567,7 +3566,10 @@ These parameters will match fields within a variable. They are case insensitive
   * options: ignore_case, or
 measurement_identifiers parameter is a nested parameter with subfields: contextmedium, object and quantity. Multiple measurement_identifiers can be specified via different indexes to search variables. The following example searches for variables that have at least one measurement_identifier with contextmedium of Med1, object of Object1 and quantity of Q1, and another measurement_identifier with contextmedium of Med2 and object of Obj2.
 
-    curl -g "%CMR-ENDPOINT%/variables?measurement_identifiers\[0\]\[contextmedium\]=Med1&measurement_identifiers\[0\]\[object\]=Object1&measurement_identifiers\[0\]\[quantity\]=Q1&measurement_identifiers\[1\]\[contextmedium\]=med2&measurement_identifiers\[2\]\[object\]=Obj2"
+    
+````
+curl -g "%CMR-ENDPOINT%/variables?measurement_identifiers\[0\]\[contextmedium\]=Med1&measurement_identifiers\[0\]\[object\]=Object1&measurement_identifiers\[0\]\[quantity\]=Q1&measurement_identifiers\[1\]\[contextmedium\]=med2&measurement_identifiers\[2\]\[object\]=Obj2"
+````
 
 The multiple measurement_identifiers are ANDed by default. User can specify `options[measurement-identifiers][or]=true` to make the measurement_identifiers ORed together.
 
@@ -3603,6 +3605,7 @@ The `references` field may contain multiple `reference` entries, each consisting
 | revision-id | the internal CMR version number for the result                                                                  |
 
 __Example__
+
 ```
 curl -i "%CMR-ENDPOINT%/variables?pretty=true&name=Variable1"
 
@@ -3638,6 +3641,7 @@ The JSON response includes the following fields.
   * long_name
 
 __Example__
+
 ```
 curl -g -i "%CMR-ENDPOINT%/variables.json?pretty=true&name=Var*&options[name][pattern]=true"
 
@@ -3669,6 +3673,7 @@ Content-Length: 292
 The UMM JSON response contains meta-metadata of the variable, the UMM fields and the associations field if applicable. The associations field only applies when there are collections associated with the variable and will list the collections that are associated with the variable.
 
 __Example__
+
 ```
 curl -g -i "%CMR-ENDPOINT%/variables.umm_json?name=Variable1234&pretty=true"
 HTTP/1.1 200 OK
@@ -3853,6 +3858,7 @@ The `references` field may contain multiple `reference` entries, each consisting
 | revision-id | the internal CMR version number for the result                                                                  |
 
 __Example__
+
 ```
 curl -i "%CMR-ENDPOINT%/services?name=Service1&pretty=true"
 
@@ -3888,6 +3894,7 @@ The JSON response includes the following fields.
   * long_name
 
 __Example__
+
 ```
 curl -g -i "%CMR-ENDPOINT%/services.json?pretty=true"
 
@@ -3933,6 +3940,7 @@ Content-Length: 944
 The UMM JSON response contains meta-metadata of the service and the UMM fields.
 
 __Example__
+
 ```
 curl -g -i "%CMR-ENDPOINT%/services.umm_json?name=NSIDC_AtlasNorth&pretty=true"
 HTTP/1.1 200 OK
@@ -4207,6 +4215,7 @@ The `references` field may contain multiple `reference` entries, each consisting
 | revision-id | the internal CMR version number for the result                                                                  |
 
 __Example__
+
 ```
 curl -i "%CMR-ENDPOINT%/tools?name=someTool1&pretty=true"
 
@@ -4242,6 +4251,7 @@ The JSON response includes the following fields.
   * long_name
 
 __Example__
+
 ```
 curl -g -i "%CMR-ENDPOINT%/tools.json?pretty=true"
 
@@ -4273,6 +4283,7 @@ Content-Length: 944
 The UMM JSON response contains meta-metadata of the tool and the UMM fields.
 
 __Example__
+
 ```
 curl -g -i "%CMR-ENDPOINT%/tools.umm_json?name=NSIDC_AtlasNorth&pretty=true"
 HTTP/1.1 200 OK
@@ -4558,6 +4569,7 @@ The `references` field may contain multiple `reference` entries, each consisting
 | revision-id | the internal CMR version number for the result                                                                  |
 
 __Example__
+
 ```
 curl -i "%CMR-ENDPOINT%/subscriptions?name=someSub1&pretty=true"
 
@@ -4594,6 +4606,7 @@ The JSON response includes the following fields.
   * collection_concept_id
 
 __Example__
+
 ```
 curl -g -i "%CMR-ENDPOINT%/subscriptions.json?pretty=true"
 
@@ -4627,6 +4640,7 @@ Content-Length: 944
 The UMM JSON response contains meta-metadata of the subscription and the UMM fields.
 
 __Example__
+
 ```
 curl -g -i "%CMR-ENDPOINT%/subscriptions.umm_json?name=NSIDC_AtlasNorth&pretty=true"
 HTTP/1.1 200 OK

search-app/docs/api.md

@@ -6,7 +6,8 @@
    [cmr.common.util :as util]
    [cmr.search.data.metadata-retrieval.revision-format-map :as r]
    [cmr.search.test.data.metadata-retrieval.test-metadata :as tm]
-   [cmr.umm-spec.test.expected-conversion :as expected-conversion]))
+   [cmr.umm-spec.test.expected-conversion :as expected-conversion]
+   [digest :as digest]))
 
 (use-fixtures :each tk/freeze-resume-time-fixture)
 
@@ -96,9 +97,20 @@
              (r/add-additional-format nil :echo10 rfm)))))
   (testing "Compressed"
     (let [rfm (r/compress
-                (r/concept->revision-format-map nil tm/dif10-concept #{:native}))]
-      (is (= (assoc rfm :echo10 (util/string->lz4-bytes (:metadata tm/echo10-concept)))
-             (r/add-additional-format nil :echo10 rfm))))))
+                (r/concept->revision-format-map nil tm/dif10-concept #{:native}))
+          expected (assoc rfm :echo10 (util/string->lz4-bytes (:metadata tm/echo10-concept)))
+          actual (r/add-additional-format nil :echo10 rfm)]
+
+      ; A test like (is(= expected actual)) does not work for everyone locally
+      ; as there seams to be some problem with testing byte arrays.
+      ; To get around this, a new test was come up with which drops the
+      ; compressed segment (byte array) which fails and tests that data sepratly
+      ; by doing a comparison on the md5 values
+
+      (is (= (util/dissoc-in expected [:echo10 :compressed])
+             (util/dissoc-in actual [:echo10 :compressed])))
+      (is (= (digest/md5 (String. (get-in expected [:echo10 :compressed]) "UTF-8"))
+             (digest/md5 (String. (get-in actual [:echo10 :compressed] )"UTF-8")))))))
 
 (defn test-rfm
   "Creates a revision format map with the specified formats."

search-app/test/cmr/search/test/data/metadata_retrieval/revision_format_map.clj

@@ -665,7 +665,7 @@
                                                                      :identifiers [:identifier "a very nice granule :)"
                                                                                    :identifier-type "a type of identifier"]})})
                         (data-core/item->concept {:format :umm-json
-                                                  :version "1.6.2"}))
+                                                  :version "1.6.3"}))
             {:keys [status] :as response} (ingest/ingest-concept granule)]
         (is (#{200 201} status) (pr-str response))))
 

system-int-test/test/cmr/system_int_test/ingest/granule_ingest_test.clj

@@ -217,7 +217,12 @@
 			<Type>METADATA</Type>
 			<MimeType>text/xml</MimeType>
 		</OnlineResource>
+		<OnlineResource>
+			<URL>https://some.s3.bucket.exammple.org</URL>
+			<Type>EXTENDED METADATA : DMR++</Type>
+			<MimeType>text/xml</MimeType>
+		</OnlineResource>
 	</OnlineResources>
 	<Orderable>true</Orderable>
 	<Visible>true</Visible>
-</Granule>
+</Granule>

umm-lib/resources/data/echo10/G1001765469-NSIDC_ECS.native.xml

@@ -9,7 +9,8 @@
 
 (def resource-type->related-url-types
   "A mapping of ECHO10 OnlineResource's type to UMM RelatedURL's type and sub-type.
-  This came from a list provided by Katie on ECHO10 collections, more may need to be added for granules."
+   This came from a list provided by Katie on ECHO10 collections, more may need to
+   be added for granules."
   {"STATIC URL" ["VIEW RELATED INFORMATION"]
    "GUIDE" ["VIEW RELATED INFORMATION" "USER'S GUIDE"]
    "HOMEPAGE" ["VIEW PROJECT HOME PAGE"]
@@ -51,7 +52,11 @@
    "ALGORITHM INFO" ["VIEW RELATED INFORMATION"]
    "GET DATA : OPENDAP DATA (DODS)" ["USE SERVICE API" "OPENDAP DATA"]
    "GET DATA : OPENDAP DATA" ["USE SERVICE API" "OPENDAP DATA"]
-   "VIEW PROJECT HOME PAGE" ["VIEW PROJECT HOME PAGE"]})
+   "VIEW PROJECT HOME PAGE" ["VIEW PROJECT HOME PAGE"]
+   "DMR++" ["EXTENDED METADATA" "DMR++"]
+   "EXTENDED METADATA : DMR++" ["EXTENDED METADATA" "DMR++"]
+   "DMR++ MISSING DATA" ["EXTENDED METADATA" "DMR++ MISSING DATA"]
+   "EXTENDED METADATA : DMR++ MISSING DATA" ["EXTENDED METADATA" "DMR++ MISSING DATA"]})
 
 (def related-url-types->resource-types
   "A mapping of UMM RelatedURL's type to ECHO10 OnlineResource's type.
@@ -165,6 +170,18 @@
                      (when description (x/element :URLDescription {} description))
                      (when mime-type (x/element :MimeType {} mime-type))))))))
 
+(defn- related-url->online-resource
+  "Related-url-types->resource-types does not handle the use case for matching a
+  subtype, if the type and subtype are extended metadata DMR values, then encode
+  them in colon format"
+  [related-url]
+  (let [url-type (:type related-url)
+        url-subtype (:sub-type related-url)]
+    (if (and (= "EXTENDED METADATA" url-type)
+             (some #{url-subtype} ["DMR++" "DMR++ MISSING DATA"]))
+      (str url-type " : " url-subtype)
+      (get related-url-types->resource-types url-type "USER SUPPORT"))))
+
 (defn generate-resource-urls
   "Generates the OnlineResources element of an ECHO10 XML from a UMM related urls entry."
   [related-urls]
@@ -178,7 +195,7 @@
                      (when description (x/element :Description {} description))
                      ;; There is not a well defined one to one mapping between related url type and resource type.
                      ;; This default value of "USER SUPPORT" is to get us by the xml schema validation.
-                     (x/element :Type {} (get related-url-types->resource-types type "USER SUPPORT"))
+                     (x/element :Type {} (related-url->online-resource related-url))
                      (when mime-type (x/element :MimeType {} mime-type))))))))
 
 (defn generate-browse-urls

umm-lib/src/cmr/umm/echo10/related_url.clj

@@ -114,7 +114,7 @@
   (filter resource-url? related-urls))
 
 (defn- related-url->link-type
-  "Returns the atom link type of the related url"
+  "Returns the atom link type of the related url - used for granules"
   [related-url]
   (cond
     (downloadable-url? related-url) "data"

umm-lib/src/cmr/umm/related_url_helper.clj

@@ -214,6 +214,11 @@
         <Description>Some description.</Description>
         <Type>Browse</Type>
       </OnlineResource>
+      <OnlineResource>
+        <URL>https://dmr.s3.bucket.example.org</URL>
+        <Description>Some description about DMR buckets</Description>
+        <Type>EXTENDED METADATA : DMR++</Type>
+      </OnlineResource>
     </OnlineResources>
     <CloudCover>0.8</CloudCover>
     <AssociatedBrowseImageUrls>
@@ -347,6 +352,12 @@
                        :url "ftp://camex.nsstc.nasa.gov/camex3/dc8capac/browse/"
                        :description "Some description."
                        :title "Some description. (Browse)"})
+                    (umm-c/map->RelatedURL
+                      {:type "EXTENDED METADATA"
+                       :sub-type "DMR++"
+                       :url "https://dmr.s3.bucket.example.org"
+                       :title "Some description about DMR buckets (EXTENDED METADATA : DMR++)"
+                       :description "Some description about DMR buckets"})
                     (umm-c/map->RelatedURL
                       {:type "GET RELATED VISUALIZATION"
                        :url "http://nasa.gov/1"

umm-lib/test/cmr/umm/test/echo10/granule.clj

@@ -0,0 +1,49 @@
+(ns cmr.umm.test.echo10.related-url
+  "Tests functions for granule related urls"
+  (:require
+   [clojure.test :refer :all]
+   [cmr.common.util :as util :refer [are3]]
+   [cmr.umm.echo10.related-url :as rurl]))
+
+(deftest validate-xml
+  (testing
+    "Validate that the related-url->online-resource function correctly returns
+    subtypes for the DMR use cases"
+    (are3 [expected actual-type actual-subtype]
+          (is (= expected
+                 (#'rurl/related-url->online-resource {:type actual-type
+                                                       :sub-type actual-subtype})))
+          "Extended metadata with DMR++"
+          "EXTENDED METADATA : DMR++"
+          "EXTENDED METADATA"
+          "DMR++"
+
+          "Extended metadata with DMR++ Missing Data"
+          "EXTENDED METADATA : DMR++ MISSING DATA"
+          "EXTENDED METADATA"
+          "DMR++ MISSING DATA"
+
+          "A future case of extended metadata which should not be changed"
+          "USER SUPPORT"
+          "EXTENDED METADATA"
+          "Future Kathryn Janeway"
+
+          "User support case for existing types"
+          "USER SUPPORT"
+          "DOWNLOAD SOFTWARE"
+          "APP"
+
+          "User support should be assumed for a nil"
+          "USER SUPPORT"
+          "type"
+          nil
+
+          "User support should be assumed for a nil"
+          "USER SUPPORT"
+          nil
+          "sub"
+
+          "User support should be assumed for a nil"
+          "USER SUPPORT"
+          nil
+          nil)))