DOC-738 | Vector index reference docs

site/content/3.12/about-arangodb/features/community-edition.md

@@ -149,7 +149,7 @@ see [arangodb.com/community-server/](https://www.arangodb.com/community-server/)
 {{% /comment %}}
 
 {{% comment %}} Experimental feature
-- [**Vector search**](#TODO):
+- [**Vector search**](../../index-and-search/indexing/working-with-indexes/vector-indexes.md):
   Find items with similar properties by comparing vector embeddings generated by
   machine learning models.
 {{% /comment %}}

site/content/3.12/aql/functions/vector.md

@@ -0,0 +1,191 @@
+---
+title: Vector search functions in AQL
+menuTitle: Vector
+weight: 60
+description: >-
+  The functions for vector search let you quickly find semantically similar
+  documents utilizing indexed vector embeddings
+---
+To use vector search, you need to have vector embeddings stored in documents
+and the attribute that stores them needs to be indexed by a
+[vector index](../../index-and-search/indexing/working-with-indexes/vector-indexes.md).
+
+You can calculate vector embeddings using [ArangoDB's GraphML](../../data-science/arangographml/_index.md)
+capabilities (available in ArangoGraph) or using external tools.
+
+{{< warning >}}
+The vector index is an experimental feature that you need to enable for the
+ArangoDB server with the `--experimental-vector-index` startup option.
+Once enabled for a deployment, it cannot be disabled anymore because it
+permanently changes how the data is managed by the RocksDB storage engine
+(it adds an additional column family).
+
+To restore a dump that contains vector indexes, the `--experimental-vector-index`
+startup option needs to be enabled on the deployment you want to restore to.
+{{< /warning >}}
+
+## Vector similarity functions
+
+In order to utilize a vector index, you need to do the following in an AQL query:
+
+- Use one of the following vector similarity functions in a query.
+- `SORT` by the similarity so that the most similar documents come first.
+- Specify the maximum number of documents to retrieve with a `LIMIT` operation.
+
+As a result, you get up to the specified number of documents whose vector embeddings
+are the most similar to the reference vector embedding you provided in the query,
+as approximated by the vector index.
+
+Example:
+
+```aql
+FOR doc IN coll
+  SORT APPROX_NEAR_L2(doc.vector, @q)
+  LIMIT 5
+  RETURN doc
+```
+
+For this query, a vector index over the `vector` attribute and with the `l2`
+metric is required. The `@q` bind variable needs to be a vector (array of numbers)
+with the dimension as specified in the vector index. It defines the point at
+which to look for neighbors (up to `5` in this case). How many neighbors can be
+found depends on the data as well as the search effort (see the `nProbe` option).
+
+{{< info >}}
+- If there is more than one suitable vector index over the same attribute, it is
+  undefined which one is selected.
+- You cannot have any `FILTER` operation between `FOR` and `LIMIT` for
+  pre-filtering.
+{{< /info >}}
+
+### APPROX_NEAR_COSINE()
+
+`APPROX_NEAR_COSINE(vector1, vector2, options) → similarity`
+
+Retrieve the approximate angular similarity using the cosine metric, accelerated
+by a matching vector index.
+
+The closer the cosine similarity value is to 1, the more similar the two vectors
+are. The closer it is to 0, the more different they are. The value can also
+be up to -1, indicating that the vectors are not similar and point in opposite
+directions. You need to sort in descending order so that the most similar
+documents come first, which is what a vector index using the `cosine` metric
+can provide.
+
+- **vector1** (array of numbers): The first vector. Either this parameter or
+  `vector2` needs to reference a stored attribute holding the vector embedding.
+  attribute of a stored document that stores a vector, like `doc.vector`
+- **vector2** (array of numbers): The second vector. Either this parameter or
+  `vector1` needs to reference a stored attribute holding the vector embedding.
+- **options** (object, _optional_):
+  - **nProbe** (number, _optional_): How many neighboring centroids to consider
+    for the search results. The larger the number, the slower the search but the
+    better the search results. If not specified, the `defaultNProbe` value of
+    the vector index is used.
+- returns **similarity** (number): The approximate angular similarity between
+  both vectors.
+
+**Examples**
+
+Return the documents of up to `10` neighbors that are the closest to the vector
+`@q` according to the cosine metric:
+
+```aql
+FOR doc IN coll
+  SORT APPROX_NEAR_COSINE(doc.vector, @q) DESC
+  LIMIT 10
+  RETURN doc
+```
+
+Return the similarity value and the documents of up to `5` close neighbors,
+considering `20` neighboring centroids:
+
+```aql
+FOR doc IN coll
+  LET similarity = APPROX_NEAR_COSINE(doc.vector, @q, { nProbe: 20 })
+  SORT similarity DESC
+  LIMIT 5
+  RETURN MERGE( { similarity }, doc)
+```
+
+Return the similarity and the document keys of up to `3` neighbors for multiple
+vectors using a subquery, here taking from ten random documents of the same
+collection:
+
+```aql
+FOR docOuter IN coll
+  LIMIT 10
+  LET neighbors = (
+    FOR docInner IN coll
+      LET similarity = APPROX_NEAR_COSINE(docInner.vector, docOuter.vector)
+      SORT similarity DESC
+      LIMIT 3
+      RETURN { key: docInner._key, similarity }
+  )
+  RETURN { key: docOuter._key, neighbors }
+```
+
+### APPROX_NEAR_L2()
+
+`APPROX_NEAR_L2(vector1, vector2, options) → similarity`
+
+Retrieve the approximate distance using the L2 (Euclidean) metric, accelerated
+by a matching vector index.
+
+The closer the distance is to 0, the more similar the two vectors are. The higher
+the value, the more different the they are. You need to sort in ascending order
+so that the most similar documents come first, which is what a vector index using
+the `l2` metric can provide.
+
+- **vector1** (array of numbers): The first vector. Either this parameter or
+  `vector2` needs to reference a stored attribute holding the vector embedding.
+  attribute of a stored document that stores a vector, like `doc.vector`
+- **vector2** (array of numbers): The second vector. Either this parameter or
+  `vector1` needs to reference a stored attribute holding the vector embedding.
+- **options** (object, _optional_):
+  - **nProbe** (number, _optional_): How many neighboring centroids to consider
+    for the search results. The larger the number, the slower the search but the
+    better the search results. If not specified, the `defaultNProbe` value of
+    the vector index is used.
+- returns **similarity** (number): The approximate L2 (Euclidean) distance between
+  both vectors.
+
+**Examples**
+
+Return the documents of up to `10` neighbors that are the closest to the vector
+`@q` according to the L2 (Euclidean) metric:
+
+```aql
+FOR doc IN coll
+  SORT APPROX_NEAR_L2(doc.vector, @q)
+  LIMIT 10
+  RETURN doc
+```
+
+Return the similarity value and the documents of up to `5` close neighbors,
+considering `20` neighboring centroids:
+
+```aql
+FOR doc IN coll
+  LET similarity = APPROX_NEAR_L2(doc.vector, @q, { nProbe: 20 })
+  SORT similarity
+  LIMIT 5
+  RETURN MERGE( { similarity }, doc)
+```
+
+Return the similarity and the document keys of up to `3` close neighbors for
+multiple vectors using a subquery, here taking from ten random documents of the
+same collection:
+
+```aql
+FOR docOuter IN coll
+  LIMIT 10
+  LET neighbors = (
+    FOR docInner IN coll
+      LET similarity = APPROX_NEAR_L2(docInner.vector, docOuter.vector)
+      SORT similarity
+      LIMIT 3
+      RETURN { key: docInner._key, similarity }
+  )
+  RETURN { key: docOuter._key, neighbors }
+```

site/content/3.12/develop/http-api/indexes/fulltext.md

@@ -49,6 +49,7 @@ paths:
                   description: |
                     Must be equal to `"fulltext"`.
                   type: string
+                  example: fulltext
                 name:
                   description: |
                     An easy-to-remember name for the index to look it up or refer to it in index hints.
@@ -78,15 +79,13 @@ paths:
       responses:
         '200':
           description: |
-            If the index already exists, then a *HTTP 200* is
-            returned.
+            The index exists already.
         '201':
           description: |
-            If the index does not already exist and could be created, then a *HTTP 201*
-            is returned.
+            The index is created as there is no such existing index.
         '404':
           description: |
-            If the `collection-name` is unknown, then a *HTTP 404* is returned.
+            The collection is unknown.
       tags:
         - Indexes
 ```

site/content/3.12/develop/http-api/indexes/geo-spatial.md

@@ -13,7 +13,7 @@ paths:
       operationId: createIndexGeo
       description: |
         Creates a geo-spatial index in the collection `collection`, if
-        it does not already exist. Expects an object containing the index details.
+        it does not already exist.
 
         Geo indexes are always sparse, meaning that documents that do not contain
         the index attributes or have non-numeric values in the index attributes
@@ -47,6 +47,7 @@ paths:
                   description: |
                     Must be equal to `"geo"`.
                   type: string
+                  example: geo
                 name:
                   description: |
                     An easy-to-remember name for the index to look it up or refer to it in index hints.
@@ -105,14 +106,13 @@ paths:
       responses:
         '200':
           description: |
-            If the index already exists, then a *HTTP 200* is returned.
+            The index exists already.
         '201':
           description: |
-            If the index does not already exist and could be created, then a *HTTP 201*
-            is returned.
+            The index is created as there is no such existing index.
         '404':
           description: |
-            If the `collection` is unknown, then a *HTTP 404* is returned.
+            The collection is unknown.
       tags:
         - Indexes
 ```

site/content/3.12/develop/http-api/indexes/inverted.md

@@ -44,6 +44,7 @@ paths:
                   description: |
                     Must be equal to `"inverted"`.
                   type: string
+                  example: inverted
                 name:
                   description: |
                     An easy-to-remember name for the index to look it up or refer to it in index hints.
@@ -625,14 +626,13 @@ paths:
       responses:
         '200':
           description: |
-            If the index already exists, then a *HTTP 200* is returned.
+            The index exists already.
         '201':
           description: |
-            If the index does not already exist and can be created, then a *HTTP 201*
-            is returned.
+            The index is created as there is no such existing index.
         '404':
           description: |
-            If the `collection-name` is unknown, then a *HTTP 404* is returned.
+            The collection is unknown.
       tags:
         - Indexes
 ```

site/content/3.12/develop/http-api/indexes/multi-dimensional.md

@@ -13,8 +13,7 @@ paths:
       operationId: createIndexMdi
       description: |
         Creates a multi-dimensional index for the collection `collection-name`, if
-        it does not already exist. The call expects an object containing the index
-        details.
+        it does not already exist.
       parameters:
         - name: database-name
           in: path
@@ -45,6 +44,7 @@ paths:
                   description: |
                     Must be equal to `"mdi"` or `"mdi-prefixed"`.
                   type: string
+                  example: mdi
                 name:
                   description: |
                     An easy-to-remember name for the index to look it up or refer to it in index hints.
@@ -136,18 +136,16 @@ paths:
       responses:
         '200':
           description: |
-            If the index already exists, then a *HTTP 200* is
-            returned.
+            The index exists already.
         '201':
           description: |
-            If the index does not already exist and could be created, then a *HTTP 201*
-            is returned.
-        '404':
-          description: |
-            If the `collection-name` is unknown, then a *HTTP 404* is returned.
+            The index is created as there is no such existing index.
         '400':
           description: |
-            If the index definition is invalid, then a *HTTP 400* is returned.
+            The index definition is invalid.
+        '404':
+          description: |
+            The collection is unknown.
       tags:
         - Indexes
 ```

site/content/3.12/develop/http-api/indexes/persistent.md

@@ -19,8 +19,7 @@ paths:
       operationId: createIndexPersistent
       description: |
         Creates a persistent index for the collection `collection-name`, if
-        it does not already exist. The call expects an object containing the index
-        details.
+        it does not already exist.
 
         In a sparse index all documents will be excluded from the index that do not
         contain at least one of the specified index attributes (i.e. `fields`) or that
@@ -64,6 +63,7 @@ paths:
                   description: |
                     Must be equal to `"persistent"`.
                   type: string
+                  example: persistent
                 name:
                   description: |
                     An easy-to-remember name for the index to look it up or refer to it in index hints.
@@ -165,20 +165,17 @@ paths:
       responses:
         '200':
           description: |
-            If the index already exists, then a *HTTP 200* is
-            returned.
+            The index exists already.
         '201':
           description: |
-            If the index does not already exist and could be created, then a *HTTP 201*
-            is returned.
+            The index is created as there is no such existing index.
         '400':
           description: |
-            If the collection already contains documents and you try to create a unique
-            persistent index in such a way that there are documents violating the
-            uniqueness, then a *HTTP 400* is returned.
+            You try to create a unique persistent index but there are already
+            documents in the collection that violate the uniqueness requirement.
         '404':
           description: |
-            If the `collection-name` is unknown, then a *HTTP 404* is returned.
+            The collection is unknown.
       tags:
         - Indexes
 ```