Environment changes, error handling

Author: pamelafox

.gitignore

@@ -111,6 +111,7 @@ celerybeat.pid
 # Environments
 .env
 .venv
+.evalenv
 env/
 venv/
 ENV/

README.md

@@ -85,9 +85,8 @@ A related option is VS Code Dev Containers, which will open the project in your
 4. Install required Python packages and backend application:
 
     ```shell
-    pip install uv
-    uv pip install -r requirements-dev.txt
-    uv pip install -e src/backend
+    pip install -r requirements-dev.txt
+    pip install -e src/backend
     ```
 
 5. Continue with the [deployment steps](#deployment)

docs/safety_evaluation.md

@@ -25,25 +25,31 @@ In order to use the adversarial simulator and safety evaluators, you need an Azu
 
 ## Setup the evaluation environment
 
-1. Install all the dependencies for the safety evaluation script by running the following command:
+1. Create a new Python virtual environment in `.evalenv` by running the following command:
 
-  ```bash
-  uv pip install -r requirements-dev.txt
-  uv pip install -r evals/requirements.txt
-  ```
+    ```bash
+    python -m venv .evalenv
+    ```
 
-2. Activate the virtual environment:
+2. Activate the virtual environment by running the following command:
 
-  macOS and Linux:
+    MacOS/Linux:
 
-  ```bash
-  source .venv/bin/activate
-  ```
+    ```bash
+    source .evalenv/bin/activate
+    ```
+
+    Windows:
 
-  Windows:
+    ```bash
+    .evalenv\Scripts\activate
+    ```
+
+1. Install the dependencies for the safety evaluation script:
 
   ```bash
-  .venv\Scripts\activate
+  pip install uv
+  uv pip install -r evals/requirements.txt
   ```
 
 ## Simulate and evaluate adversarial users
@@ -54,7 +60,7 @@ Run the following command to simulate adversarial queries and evaluate the safet
 python evals/safety_evaluation.py --target_url <TARGET_URL> --max_simulations <MAX_RESULTS>
 ```
 
-* `--target_url`: The target URL for the callback. Default is `http://127.0.0.1:8000/chat`.
+* `--target_url`: The target URL for the callback. Default is `http://127.0.0.1:8000/chat`, so make sure that the RAG app is running locally. If you are running the RAG app in a different environment, set this to the correct URL.
 * `--max_simulations`: The maximum number of simulated user queries. Default is `200`. The higher the number, the longer the evaluation will take. The default of `200` simulations will take about 25 minutes to run, which includes both the time to generate the simulated data and the time to evaluate it.
 
 ## Review the safety evaluation results

evals/requirements.txt

@@ -1,2 +1,4 @@
-git+https://github.com/Azure-Samples/ai-rag-chat-evaluator/@installable
-rich
+git+https://github.com/Azure-Samples/ai-rag-chat-evaluator/@2025-02-06b
+azure-ai-evaluation
+rich
+dotenv-azd

scripts/load_python_env.sh

@@ -3,9 +3,9 @@
 python3 -m pip install uv
 
 echo 'Creating Python virtual environment in .venv...'
-uv venv
+python3 -m venv .venv
 
 echo 'Installing dependencies from "requirements.txt" into virtual environment (in quiet mode)...'
 
-uv pip install --quiet -e src/backend
-uv pip install --quiet -r requirements-dev.txt
+.venv/bin/python -m pip --quiet --disable-pip-version-check install -e src/backend
+.venv/bin/python -m pip --quiet --disable-pip-version-check install -r requirements-dev.txt

src/backend/fastapi_app/routes/api_routes.py

@@ -6,6 +6,7 @@
 import fastapi
 from fastapi import HTTPException
 from fastapi.responses import StreamingResponse
+from openai import APIError
 from sqlalchemy import select, text
 
 from fastapi_app.api_models import (
@@ -25,6 +26,9 @@
 router = fastapi.APIRouter()
 
 
+ERROR_FILTER = {"error": "Your message contains content that was flagged by the content filter."}
+
+
 async def format_as_ndjson(r: AsyncGenerator[RetrievalResponseDelta, None]) -> AsyncGenerator[str, None]:
     """
     Format the response as NDJSON
@@ -33,8 +37,11 @@ async def format_as_ndjson(r: AsyncGenerator[RetrievalResponseDelta, None]) -> A
         async for event in r:
             yield event.model_dump_json() + "\n"
     except Exception as error:
-        logging.exception("Exception while generating response stream: %s", error)
-        yield json.dumps({"error": str(error)}, ensure_ascii=False) + "\n"
+        if isinstance(error, APIError) and error.code == "content_filter":
+            yield json.dumps(ERROR_FILTER) + "\n"
+        else:
+            logging.exception("Exception while generating response stream: %s", error)
+            yield json.dumps({"error": str(error)}, ensure_ascii=False) + "\n"
 
 
 @router.get("/items/{id}", response_model=ItemPublic)
@@ -135,7 +142,10 @@ async def chat_handler(
         )
         return response
     except Exception as e:
-        return {"error": str(e)}
+        if isinstance(e, APIError) and e.code == "content_filter":
+            return ERROR_FILTER
+        else:
+            return {"error": str(e)}
 
 
 @router.post("/chat/stream")
@@ -175,9 +185,15 @@ async def chat_stream_handler(
 
     # Intentionally do this before we stream down a response, to avoid using database connections during stream
     # See https://github.com/tiangolo/fastapi/discussions/11321
-    contextual_messages, results, thoughts = await rag_flow.prepare_context(chat_params)
-
-    result = rag_flow.answer_stream(
-        chat_params=chat_params, contextual_messages=contextual_messages, results=results, earlier_thoughts=thoughts
-    )
-    return StreamingResponse(content=format_as_ndjson(result), media_type="application/x-ndjson")
+    try:
+        contextual_messages, results, thoughts = await rag_flow.prepare_context(chat_params)
+        result = rag_flow.answer_stream(
+            chat_params=chat_params, contextual_messages=contextual_messages, results=results, earlier_thoughts=thoughts
+        )
+        return StreamingResponse(content=format_as_ndjson(result), media_type="application/x-ndjson")
+    except Exception as e:
+        if isinstance(e, APIError) and e.code == "content_filter":
+            return StreamingResponse(
+                content=json.dumps(ERROR_FILTER) + "\n",
+                media_type="application/x-ndjson",
+            )