Skip to content

feat(evaluate): Add Azure AI Evaluation & Observability Workshop Notebook #200

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

thegovind
Copy link

Description

This PR adds a comprehensive workshop notebook demonstrating Azure AI's evaluation and observability capabilities, developed in collaboration with @changliu2 and Alex Sutton for TechConnect. This is the first iteration of the workshop material, which has been fully tested and provides hands-on examples of:

  • Safety evaluation using ContentSafety, Violence, and HateUnfairness evaluators
  • Security testing with DirectAttackSimulator and adversarial testing
  • Quality assessment using Relevance, Coherence, and Groundedness evaluators
  • Cloud-based evaluation workflows with azure-ai-projects
  • Observability setup with OpenTelemetry and Application Insights
  • Best practices for responsible AI governance

The workshop includes:

  • A Jupyter notebook with detailed explanations and working code
  • Sample evaluation test data in JSONL format
  • Environment configuration templates

This serves as a practical guide for developers to implement comprehensive evaluation and monitoring in their Azure AI applications.

Checklist

  • I have read the contribution guidelines
  • I have coordinated with the docs team (mldocs@microsoft.com) if this PR deletes files or changes any file names or file extensions.
  • This notebook or file is added to the CODEOWNERS file, pointing to the author or the author's team.

@thegovind thegovind requested a review from a team as a code owner February 10, 2025 05:01
@thegovind thegovind had a problem deploying to external-contribution February 10, 2025 05:02 — with GitHub Actions Failure
@thegovind thegovind had a problem deploying to external-contribution February 12, 2025 16:18 — with GitHub Actions Failure
@thegovind thegovind had a problem deploying to external-contribution February 12, 2025 19:16 — with GitHub Actions Failure
@thegovind thegovind had a problem deploying to external-contribution February 12, 2025 20:48 — with GitHub Actions Failure
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants