Skip to content

Commit 59c4828

Browse files
committed
Update documentation
1 parent 4d2977a commit 59c4828

File tree

1 file changed

+26
-9
lines changed

1 file changed

+26
-9
lines changed

docs/safety_evaluation.md

Lines changed: 26 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@ When deploying a RAG app to production, you should evaluate the safety of the an
44

55
* [Deploy an Azure AI project](#deploy-an-azure-ai-project)
66
* [Setup the evaluation environment](#setup-the-evaluation-environment)
7-
* [Run safety scan](#simulate-and-evaluate-adversarial-users)
8-
* [Review the safety evaluation results](#review-the-safety-evaluation-results)
7+
* [Run red teaming agent](#run-red-teaming-agent)
8+
* [Review the red teaming results](#review-the-red-team-results)
99

1010
## Deploy an Azure AI project
1111

@@ -45,25 +45,27 @@ In order to use the Red Teaming agent, you need an Azure AI project inside Azure
4545
.evalenv\Scripts\activate
4646
```
4747

48-
1. Install the dependencies for the safety evaluation script:
48+
3. Install the dependencies for the safety evaluation script:
4949

5050
```bash
5151
pip install uv
5252
uv pip install -r evals/requirements.txt
5353
```
5454

55-
## Simulate and evaluate adversarial users
55+
## Run the red teaming agent
5656

57-
Run the following command to simulate adversarial queries and evaluate the safety of the answers generated in response to those queries:
57+
To run the red teaming agent, you need to have the RAG app running at a deployed URL or local URL. Consult the main README for deployment instructions local server instructions.
58+
59+
In the same terminal where you activated `.evalenv`, run this command to perform a red teaming scan:
5860

5961
```shell
6062
python evals/safety_evaluation.py --target_url <TARGET_URL> --questions_per_category <NUM_QUESTIONS>
6163
```
6264

63-
* `--target_url`: The target URL for the callback. Default is `http://127.0.0.1:8000/chat`, so make sure that the RAG app is running locally. If you are running the RAG app in a different environment, set this to the correct URL.
65+
* `--target_url`: The target URL for the callback. Default is `http://127.0.0.1:8000/chat`, which assumes the RAG app is running locally. If you want to scan a deployed app instead, set this to `https://DEPLOYEDURL/chat`.
6466
* `--questions_per_category`: The number of questions per risk category to ask during the scan. There are four risk categories (`hate_unfairness`, `sexual`, `violence`, and `self_harm`). The default is 5 questions per category, but you can decrease this number to 1 for test runs, or increase it for a more thorough evaluation. Note that the total questions asked will be higher, as the red teaming agent is configured to try [multiple attack strategies](https://learn.microsoft.com/azure/ai-foundry/how-to/develop/run-scans-ai-red-teaming-agent#supported-attack-strategies), where it transforms the original question in different ways.
6567

66-
## Review the red team results
68+
## Review the red teaming results
6769

6870
That command will save results both in Azure AI Foundry and in the local `evals/redteams` directory.
6971

@@ -80,10 +82,25 @@ Then you can dig into the data and see the questions and answers received from y
8082
8183
Note that the questions are intentionally adversarial, so you should mentally prepare for that before looking at the results. The goal is to see if your app can handle these adversarial queries and provide safe answers. Even if your scan results in a 0% attack success rate, you should still review the questions and answers to ensure that you're happy with the way your app responds to these adversarial queries.
8284

85+
Learn more in the [Red Teaming Agent documentation](https://learn.microsoft.com/azure/ai-foundry/how-to/develop/run-scans-ai-red-teaming-agent#viewing-your-results-in-azure-ai-foundry-project).
86+
87+
## Review results locally
88+
89+
In addition to the results in Azure AI Foundry, you can also review the results locally in the `evals/redteams` directory. The results are saved in JSON format.
90+
91+
Each file in this directory corresponds to a single red teaming scan, and contains the following fields:
92+
93+
* `redteaming_scorecard`: A summary of the scan results, including the attack success rate and the number of questions asked.
94+
* `redteaming_parameters`: The parameters used for the scan, including the risk categories and attack strategies.
95+
* `redteaming_data`: A list of the questions asked during the scan, along with the answers received from your app.
96+
* `studio_url`: A link to the Azure AI Foundry studio where you can view the results in more detail.
97+
98+
Learn more in the [Red Teaming Agent documentation](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/run-scans-ai-red-teaming-agent#results-from-your-automated-scans).
99+
83100
## Resources
84101

85-
To learn more about the Azure AI services used in this project, look through the script, documentation, and videos below:
102+
To learn more about the red team scanning, look through the script, documentation, and videos below:
86103

87-
* [safety_evaluation.py](evals/safety_evaluation.py)
104+
* [safety_evaluation.py](/evals/safety_evaluation.py)
88105
* [Run automated safety scans with AI Red Teaming Agent](https://learn.microsoft.com/azure/ai-foundry/how-to/develop/run-scans-ai-red-teaming-agent)
89106
* [Build 2025: Red-teaming Demo](https://www.youtube.com/watch?v=sZzcSX7BFVA)

0 commit comments

Comments
 (0)