Explore wildcard, nullability and negation checks in occurrence search

[timrobertson100]

The GraphQL API powering the hosted portals is able to support null, negation, and wildcard searches for occurrence search (currently enabled for a few fields only, but more will follow).

It would be a good addition to the REST API to support these allowing for URL hacking and URL consistency with the future of GBIF.org (see gbif/hosted-portals#209).

While the motivation applies to occurrence search, it may be of interest to other content types surfaced in REST APIs. These could be added progressively (e.g. returning some 4** code if detected and not supported in the meantime).

[MortenHofft]

So far gbif-web project has been using

• !issue=ZERO_COORDINATE for NOT.
• issue=* for NOT_NULL (syntax same as a wildcard search)
• !issue=* for NULL
• recordedBy=*humboldt for LIKE (could also be recordedBy~=*humboldt)

But a more explicit syntax might be preferable and more flexible.
issue_$exists, issue_$not=ZERO_COORDINATE, year_$gte=1900, recordedBy_$like=*humboldt

When negations are present I would interpret those as AND. unlike what we do normally where it is an OR
issue_$not=A&issue_$not=B&issue=C&issue=D
E.g.

{
  "type": "and",
    "predicates": [
      {
        "type": "not",
        "predicate": {
          "type": "in", // the occurrence must have neither A nor B. - (NOT A) and (NOT B)
          "key": "issue",
          "values": [
            "A",
            "B"
          ]
        }
      },
      {
        "type": "in", // the occurrence has either issue C or D - C or D
        "key": "issue",
        "values": [
          "C",
          "D"
        ]
      }
    ]
}

[abubelinha]

The GraphQL API powering the hosted portals is able to support null, negation, and wildcard searches for occurrence search (currently enabled for a few fields only, but more will follow).

@timrobertson100 is there a list of which fields support those searches as of today?

in https://www.gbif.org/developer/occurrence I can see this example of logical negation, but nothing else is said:

{
  "creator":"userName",
  "notificationAddresses": ["userName@example.org"],
  "predicate":
  {
    "type":"not",
    "predicate":
    {
      "type":"equals",
      "key":"DATASET_KEY",
      "value":"4fa7b334-ce0d-4e88-aaae-2e0c138d049e"
    }
  }
}

Thanks
@abubelinha

[timrobertson100]

@MortenHofft - can you answer that please so I don't give incorrect info?

[abubelinha]

BTW regarding wildcards I guess you mean the like predicate (I don't see the word wildcard in occurrence api reference)

like
search for a pattern, ? matches one character, * matches zero or more characters

{
  "creator":"userName",
  "notificationAddresses": ["userName@example.org"],
  "predicate":
  {
    "type":"like",
    "key":"CATALOG_NUMBER",
    "value":"PAPS5-560*"
  }
}
matchCase can be added if required.

[MortenHofft]

hi @abubelinha
I'm not sure I understand the question exactly, but...

Predicates
The API v1 predicates allow adding not around any other predicate. e.g.

{
  "creator":"userName",
  "notificationAddresses": ["userName@example.org"],
  "predicate":
  {
    "type":"not",
    "predicate":
    {
      "type":"or",
      "predicates": [
             {
              "type":"equals",
              "key":"TAXON_KEY",
              "value": 5
            },
            {
              "type":"like",
              "key":"CATALOG_NUMBER",
              "value":"PAPS5-560*"
            }
       ]
    }
  }
}

Search using predicates
The graphql API was created for multiple purposes, but one of them to allow search using predicates (the type the GBIF API has been using to specify downloads).

Since then that has been extended to the standard API albeit not yet documented.
gbif/portal16#1778
So it is now possible to search with a predicate as the filter.

[MortenHofft]

And yes with wildcard it is the predicate of type like with the characters * and ?. They work on text fields. So not on e.g. taxonKey

[MortenHofft]

a list of which fields support those searches as of today?

The fields that one can use for search is these: https://www.gbif.org/developer/occurrence#parameters

I do not believe we have any documentation about which fields that allow e.g. like queries (for example country (iso code) does not). But anything text that isn't a controlled vocabulary should support it

[MortenHofft]

Example usage in case someone arrives at this issue:

const url = 'https://api.gbif.org/v1/occurrence/search/predicate';

const data = {
    limit: 2,
    offset: 1,
    predicate: {
        type: "and",
        predicates: [
            {
                type: "equals",
                key: "TAXON_KEY",
                value: 212
            },
            {
                type: "equals",
                key: "COUNTRY",
                value: "DK"
            },
            {
                type: "equals",
                key: "YEAR",
                value: "1980"
            },
        ]
    }
};

fetch(url, {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify(data)
})
.then(response => response.json())
.then(result => {
    console.log(result);
})
.catch(error => {
    console.error('Error:', error);
});

[jloomisVCE]

Example v1 API /occurrence/predicate syntax snippet for GEOMETRY:

{
    "type": "within",
    "geometry": "POLYGON((-77.07713 38.9127,-77.07686 38.88456,-77.03829 38.88386,-77.03736 38.91195,-77.07713 38.9127))"
}