ArangoDB Retriever (#3)

* Refactored Retreiver for Arango

added:
arango.py,README_arango.md
modified:
config.py, requirements.txt,compose.yaml

* refactored retriever

* commit after PR review

* Update comps/retrievers/src/README_arango.md

* Update comps/retrievers/src/integrations/arangodb.py

* Update comps/retrievers/src/requirements.txt

* update: docs

* misc: fix comment

* add comment

* cleanup: `arangodb.py`

format with black/isort, use env vars, remove custom `graph_name` functionality (no longer needed)

* rm: comment

---------

Co-authored-by: lasyasn <lasyan640@gmail.com>
Co-authored-by: Anthony Mahanna <43019056+aMahanna@users.noreply.github.com>
Co-authored-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

Author: 3 people

comps/retrievers/README.md

@@ -41,3 +41,7 @@ For details, please refer to this [readme](src/README_neo4j.md)
 ## Retriever Microservice with Pathway
 
 For details, please refer to this [readme](src/README_pathway.md)
+
+## Retriever Microservice with ArangoDB
+
+For details, please refer to this [readme](src/README_arangodb.md)

comps/retrievers/deployment/docker_compose/compose.yaml

@@ -12,6 +12,7 @@ include:
   - ../../../third_parties/tei/deployment/docker_compose/compose.yaml
   - ../../../third_parties/tgi/deployment/docker_compose/compose.yaml
   - ../../../third_parties/vdms/deployment/docker_compose/compose.yaml
+  - ../../../third_parties/arango/deployment/docker_compose/compose.yaml
 
 services:
   retriever:
@@ -170,6 +171,27 @@ services:
         condition: service_healthy
       tei-embedding-serving:
         condition: service_healthy
+  
+  retriever-arango:
+    extends: retriever
+    container_name: retriever-arango
+    depends_on:
+      - arango-vector-db
+      - tei-embedding-service
+    environment:
+      RETRIEVER_COMPONENT_NAME: ${RETRIEVER_COMPONENT_NAME:-OPEA_RETRIEVER_ARANGO}
+      no_proxy: ${no_proxy}
+      http_proxy: ${http_proxy}
+      https_proxy: ${https_proxy}
+      ARANGO_URL: ${ARANGO_URL}
+      ARANGO_USERNAME: ${ARANGO_USERNAME}
+      ARANGO_PASSWORD: ${ARANGO_PASSWORD}
+      ARANGO_DB_NAME: ${ARANGO_DB_NAME}
+      ARANGO_GRAPH_NAME: ${ARANGO_GRAPH_NAME}
+      TEI_EMBED_MODEL: ${TEI_EMBED_MODEL}
+      TEI_EMBEDDING_ENDPOINT: ${TEI_EMBEDDING_ENDPOINT}
+      HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
+      OPENAI_API_KEY: ${OPENAI_API_KEY}
 
 
 networks:

comps/retrievers/src/Dockerfile

@@ -10,7 +10,8 @@ RUN apt-get update -y && apt-get install -y --no-install-recommends --fix-missin
     libjemalloc-dev \
     libcairo2 \
     libglib2.0-0 \
-    vim
+    vim \
+    git
 
 RUN useradd -m -s /bin/bash user && \
     mkdir -p /home/user && \

comps/retrievers/src/README_arangodb.md

@@ -0,0 +1,110 @@
+# Retriever Microservice with ArangoDB (work-in-progress)
+
+## 🚀Start Microservice with Python
+
+### Install Requirements
+
+```bash
+pip install -r requirements.txt
+```
+
+### Start ArangoDB Server
+
+To launch ArangoDB locally, first ensure you have docker installed. Then, you can launch the database with the following docker command.
+
+
+```bash
+docker run -d   --name arango-vector-db  -p 8529:8529   -e ARANGO_ROOT_PASSWORD=password   arangodb/arangodb:3.12.4   --experimental-vector-index=true
+```
+
+### Setup Environment Variables
+
+```bash
+export no_proxy=${your_no_proxy}
+export http_proxy=${your_http_proxy}
+export https_proxy=${your_http_proxy}
+export ARANGO_URL=${your_arangodb_uri}
+export ARANGODB_USERNAME=${your_arangodb_username}
+export ARANGODB_PASSWORD=${your_arangodb_password}
+export ARANGO_DB_NAME=${your_arangodb_database}
+
+```
+
+
+## 🚀Start Microservice with Docker
+
+### Build Docker Image
+
+```bash
+cd ../../
+docker build -t opea/retriever:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/retrievers/src/Dockerfile .
+```
+
+### Run Docker with CLI
+
+```bash
+docker run -d --name="retriever-arango-server" -p 7000:7000 --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e ARANGODB_URL="http://localhost:8529"  opea/retriever:latest -e RETRIEVER_COMPONENT_NAME="OPEA_RETRIEVER_ARANGO"
+```
+
+
+
+## 🚀3. Consume Retriever Service
+
+### 3.1 Check Service Status
+
+```bash
+curl http://${your_ip}:7000/v1/health_check \
+  -X GET \
+  -H 'Content-Type: application/json'
+```
+
+### 3.2 Consume Embedding Service
+
+To consume the Retriever Microservice, you can generate a mock embedding vector of length 768 with Python.
+
+```bash
+export your_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(768)]; print(embedding)")
+curl http://${your_ip}:7000/v1/retrieval \
+  -X POST \
+  -d "{\"text\":\"What is the revenue of Nike in 2023?\",\"embedding\":${your_embedding}}" \
+  -H 'Content-Type: application/json'
+```
+
+```bash
+export your_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(768)]; print(embedding)")
+curl http://localhost:7000/v1/retrieval \
+  -X POST \
+  -d "{\"text\":\"What is the revenue of Nike in 2023?\",\"embedding\":${your_embedding},\"search_type\":\"similarity\", \"k\":4}" \
+  -H 'Content-Type: application/json'
+```
+
+
+
+```bash
+export your_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(768)]; print(embedding)")
+curl http://localhost:7000/v1/retrieval \
+  -X POST \
+  -d "{\"text\":\"What is the revenue of Nike in 2023?\",\"embedding\":${your_embedding},\"search_type\":\"similarity_distance_threshold\", \"k\":4, \"distance_threshold\":1.0}" \
+  -H 'Content-Type: application/json'
+```
+
+
+
+```bash
+export your_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(768)]; print(embedding)")
+curl http://localhost:7000/v1/retrieval \
+  -X POST \
+  -d "{\"text\":\"What is the revenue of Nike in 2023?\",\"embedding\":${your_embedding},\"search_type\":\"similarity_score_threshold\", \"k\":4, \"score_threshold\":0.2}" \
+  -H 'Content-Type: application/json'
+```
+
+
+```bash
+export your_embedding=$(python -c "import random; embedding = [random.uniform(-1, 1) for _ in range(768)]; print(embedding)")
+curl http://localhost:7000/v1/retrieval \
+  -X POST \
+  -d "{\"text\":\"What is the revenue of Nike in 2023?\",\"embedding\":${your_embedding},\"search_type\":\"mmr\", \"k\":4, \"fetch_k\":20, \"lambda_mult\":0.5}" \
+  -H 'Content-Type: application/json'
+```
+
+