Skip to content

[pull] main from Azure-Samples:main #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,18 @@
"extensions": [
"ms-python.python",
"ms-python.vscode-pylance",
"ms-python.vscode-python-envs",
"charliermarsh.ruff",
"mtxr.sqltools",
"mtxr.sqltools-driver-pg",
"esbenp.prettier-vscode",
"mechatroner.rainbow-csv",
"ms-vscode.vscode-node-azure-pack",
"esbenp.prettier-vscode"
"esbenp.prettier-vscode",
"twixes.pypi-assistant",
"ms-python.vscode-python-envs",
"teamsdevapp.vscode-ai-foundry",
"ms-windows-ai-studio.windows-ai-studio"
],
// Set *default* container specific settings.json values on container create.
"settings": {
Expand Down
6 changes: 2 additions & 4 deletions .github/workflows/app-tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,8 @@ jobs:
fail-fast: false
matrix:
os: ["ubuntu-latest", "macos-latest-xlarge", "macos-13", "windows-latest"]
python_version: ["3.9", "3.10", "3.11", "3.12"]
python_version: ["3.10", "3.11", "3.12"]
exclude:
- os: macos-latest-xlarge
python_version: "3.9"
- os: macos-latest-xlarge
python_version: "3.10"
env:
Expand Down Expand Up @@ -123,7 +121,7 @@ jobs:
key: mypy${{ matrix.os }}-${{ matrix.python_version }}-${{ hashFiles('requirements-dev.txt', 'src/backend/requirements.txt', 'src/backend/pyproject.toml') }}

- name: Run MyPy
run: python3 -m mypy .
run: python3 -m mypy . --python-version ${{ matrix.python_version }}

- name: Run Pytest
run: python3 -m pytest -s -vv --cov --cov-fail-under=85
Expand Down
15 changes: 15 additions & 0 deletions .github/workflows/auto_approve_copilot.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
name: Auto-approve Copilot PRs

on:
pull_request:
types: [opened, synchronize]

jobs:
auto-approve:
runs-on: ubuntu-latest
if: github.actor == 'copilot-swe-agent'
steps:
- name: Auto-approve PR from Copilot agent
uses: hmarr/auto-approve-action@v3
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -149,3 +149,6 @@ static/

# Playwright test trace
test-results/

# Test virtual environments
test_venv*/
8 changes: 8 additions & 0 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,14 @@
"module": "uvicorn",
"args": ["fastapi_app:create_app", "--factory", "--reload"],
"justMyCode": false
},
{
"name": "Python: Current File",
"type": "debugpy",
"request": "launch",
"program": "${file}",
"console": "integratedTerminal",
"justMyCode": false
}
],
"compounds": [
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ A related option is VS Code Dev Containers, which will open the project in your

* [Azure Developer CLI (azd)](https://aka.ms/install-azd)
* [Node.js 18+](https://nodejs.org/download/)
* [Python 3.9+](https://www.python.org/downloads/)
* [Python 3.10+](https://www.python.org/downloads/)
* [PostgreSQL 14+](https://www.postgresql.org/download/)
* [pgvector](https://github.com/pgvector/pgvector)
* [Docker Desktop](https://www.docker.com/products/docker-desktop/)
Expand Down
Binary file added docs/images/redteam_dashboard.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/redteam_logs.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
97 changes: 48 additions & 49 deletions docs/safety_evaluation.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
# Evaluating RAG answer safety

When deploying a RAG app to production, you should evaluate the safety of the answers generated by the RAG flow. This is important to ensure that the answers are appropriate and do not contain any harmful or sensitive content. This project includes scripts that use Azure AI services to simulate an adversarial user and evaluate the safety of the answers generated in response to those adversarial queries.
When deploying a RAG app to production, you should evaluate the safety of the answers generated by the RAG flow. This is important to ensure that the answers are appropriate and do not contain any harmful or sensitive content. This project includes scripts that use the [azure-ai-evaluation SDK](https://pypi.org/project/azure-ai-evaluation/#history) to perform an [automated safety scan with an AI Red Teaming agent](https://learn.microsoft.com/azure/ai-foundry/how-to/develop/run-scans-ai-red-teaming-agent).

* [Deploy an Azure AI project](#deploy-an-azure-ai-project)
* [Setup the evaluation environment](#setup-the-evaluation-environment)
* [Simulate and evaluate adversarial users](#simulate-and-evaluate-adversarial-users)
* [Review the safety evaluation results](#review-the-safety-evaluation-results)
* [Run red teaming agent](#run-red-teaming-agent)
* [Review the red teaming results](#review-the-red-teaming-results)

## Deploy an Azure AI project

In order to use the adversarial simulator and safety evaluators, you need an Azure AI project inside an Azure AI Hub.
In order to use the Red Teaming agent, you need an Azure AI project inside Azure AI Foundry.

1. Run this command to tell `azd` to provision an Azure AI project and hub:
1. Run this command to tell `azd` to provision an Azure AI project:

```shell
azd env set USE_AI_PROJECT true
Expand Down Expand Up @@ -45,63 +45,62 @@ In order to use the adversarial simulator and safety evaluators, you need an Azu
.evalenv\Scripts\activate
```

1. Install the dependencies for the safety evaluation script:
3. Install the dependencies for the safety evaluation script:

```bash
pip install uv
uv pip install -r evals/requirements.txt
```

## Simulate and evaluate adversarial users
## Run the red teaming agent

Run the following command to simulate adversarial queries and evaluate the safety of the answers generated in response to those queries:
To run the red teaming agent, you need to have the RAG app running at a deployed URL or local URL. Consult the main README for deployment instructions local server instructions.

In the same terminal where you activated `.evalenv`, run this command to perform a red teaming scan:

```shell
python evals/safety_evaluation.py --target_url <TARGET_URL> --max_simulations <MAX_RESULTS>
python evals/safety_evaluation.py --target_url <TARGET_URL> --questions_per_category <NUM_QUESTIONS>
```

* `--target_url`: The target URL for the callback. Default is `http://127.0.0.1:8000/chat`, so make sure that the RAG app is running locally. If you are running the RAG app in a different environment, set this to the correct URL.
* `--max_simulations`: The maximum number of simulated user queries. Default is `200`. The higher the number, the longer the evaluation will take. The default of `200` simulations will take about 25 minutes to run, which includes both the time to generate the simulated data and the time to evaluate it.

## Review the safety evaluation results

That command will generate a report in `safety_results.json`.
Results will look like this:

```json
{
"hate_unfairness": {
"low_count": 20,
"score_total": 8,
"mean_score": 0.4,
"low_rate": 1.0
},
"sexual": {
"low_count": 20,
"score_total": 9,
"mean_score": 0.45,
"low_rate": 1.0
},
"violence": {
"low_count": 20,
"score_total": 9,
"mean_score": 0.45,
"low_rate": 1.0
},
"self_harm": {
"low_count": 20,
"score_total": 10,
"mean_score": 0.5,
"low_rate": 1.0
}
}
```
* `--target_url`: The target URL for the callback. Default is `http://127.0.0.1:8000/chat`, which assumes the RAG app is running locally. If you want to scan a deployed app instead, set this to `https://DEPLOYEDURL/chat`.
* `--questions_per_category`: The number of questions per risk category to ask during the scan. There are four risk categories (`hate_unfairness`, `sexual`, `violence`, and `self_harm`). The default is 5 questions per category, but you can decrease this number to 1 for test runs, or increase it for a more thorough evaluation. Note that the total questions asked will be higher, as the red teaming agent is configured to try [multiple attack strategies](https://learn.microsoft.com/azure/ai-foundry/how-to/develop/run-scans-ai-red-teaming-agent#supported-attack-strategies), where it transforms the original question in different ways.

## Review the red teaming results

That command will save results both in Azure AI Foundry and in the local `evals/redteams` directory.

### Review results in Azure AI Foundry

Navigate from the URL in the terminal output to the results in Azure AI Foundry.
First you'll see overall attack metrics:

![Red Teaming Agent Results](images/redteam_dashboard.png)

Then you can dig into the data and see the questions and answers received from your app:

![Red Teaming Agent Results](images/redteam_logs.png)

Note that the questions are intentionally adversarial, so you should mentally prepare for that before looking at the results. The goal is to see if your app can handle these adversarial queries and provide safe answers. Even if your scan results in a 0% attack success rate, you should still review the questions and answers to ensure that you're happy with the way your app responds to these adversarial queries.

Learn more in the [Red Teaming Agent documentation](https://learn.microsoft.com/azure/ai-foundry/how-to/develop/run-scans-ai-red-teaming-agent#viewing-your-results-in-azure-ai-foundry-project).

## Review results locally

In addition to the results in Azure AI Foundry, you can also review the results locally in the `evals/redteams` directory. The results are saved in JSON format.

Each file in this directory corresponds to a single red teaming scan, and contains the following fields:

* `scorecard`: A summary of the scan results, including the attack success rate and the number of questions asked.
* `parameters`: The parameters used for the scan, including the risk categories and attack strategies.
* `attack_details`: A list of the questions asked during the scan, along with the answers received from your app.
* `studio_url`: A link to the Azure AI Foundry studio where you can view the results in more detail.

The ideal score is `low_rate` of 1.0 and `mean_score` of 0.0. The `low_rate` indicates the fraction of answers that were reported as "Low" or "Very low" by an evaluator. The `mean_score` is the average score of all the answers, where 0 is a very safe answer and 7 is a very unsafe answer.
Learn more in the [Red Teaming Agent documentation](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/run-scans-ai-red-teaming-agent#results-from-your-automated-scans).

## Resources

To learn more about the Azure AI services used in this project, look through the script and reference the following documentation:
To learn more about the red team scanning, look through the script, documentation, and videos below:

* [Generate simulated data for evaluation](https://learn.microsoft.com/azure/ai-studio/how-to/develop/simulator-interaction-data)
* [Evaluate with the Azure AI Evaluation SDK](https://learn.microsoft.com/azure/ai-studio/how-to/develop/evaluate-sdk)
* [safety_evaluation.py](/evals/safety_evaluation.py)
* [Run automated safety scans with AI Red Teaming Agent](https://learn.microsoft.com/azure/ai-foundry/how-to/develop/run-scans-ai-red-teaming-agent)
* [Build 2025: Red-teaming Demo](https://www.youtube.com/watch?v=sZzcSX7BFVA)
Empty file added evals/redteams/.gitkeep
Empty file.
4 changes: 2 additions & 2 deletions evals/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
git+https://github.com/Azure-Samples/ai-rag-chat-evaluator/@2025-02-06b
azure-ai-evaluation
git+https://github.com/Azure-Samples/ai-rag-chat-evaluator/@2025-06-02
azure-ai-evaluation[redteam]>=1.8.0
rich
dotenv-azd
Loading