Skip to content

Commit

Permalink
XS ◾ Fixed image bug + other improvements (#9799)
Browse files Browse the repository at this point in the history
Fixed image bug + other improvements
  • Loading branch information
tiagov8 authored Jan 27, 2025
1 parent 726742e commit 6a89fa5
Showing 1 changed file with 13 additions and 14 deletions.
27 changes: 13 additions & 14 deletions rules/write-integration-tests-for-llm-prompts/rule.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,38 +5,37 @@ title: Do you write integration tests for your most common LLM prompts?
uri: write-integration-tests-for-llm-prompts
authors:
- title: Adam Cogan
url: https://www.ssw.com.au/people/adam-cogan/
url: https://www.ssw.com.au/people/adam-cogan
- title: Luke Mao
url: https://www.ssw.com.au/people/luke-mao/
url: https://www.ssw.com.au/people/luke-mao
created: 2024-10-03T15:12:00.000Z
guid: 3a9f5b3b-9f12-4b83-9f5e-850f9df31578
---

When integrating Azure AI's language models (LLMs) into your application, it’s important to ensure that the responses generated by the LLM are reliable and consistent. However, LLMs are non-deterministic, meaning the same prompt may not always generate the exact same response. This can introduce challenges in maintaining the quality of outputs in production environments. Writing integration tests for your most common LLM prompts helps you identify when model changes or updates could impact your application’s performance.
When integrating Azure AI's language models (LLMs) into your application, it’s important to ensure that the responses generated by the LLM are reliable and consistent. However, LLMs are non-deterministic, meaning the same prompt may not always generate the exact same response. This can introduce challenges in maintaining the quality of outputs in production environments. Writing integration tests for the most common LLM prompts helps you identify when model changes or updates could impact your application’s performance.

<!--endintro-->

:::
::: good
![Figure: Good example - EagleEye is doing Integration Testing for prompts](EagleEye.png)
:::

### Why you need integration tests for LLM prompts
## Why you need integration tests for LLM prompts

**Ensure consistency**: Integration tests allow you to check if the responses for your most critical prompts stay within an acceptable range of variation. Without these tests, you risk introducing variability that could negatively affect user experience or critical business logic
* **Ensure consistency** - Integration tests allow you to check if the responses for your most critical prompts stay within an acceptable range of variation. Without these tests, you risk introducing variability that could negatively affect user experience or critical business logic

**Detect regressions early**: As Azure AI models evolve and get updated, prompt behavior may change. By running tests regularly, you can catch regressions that result from model updates or changes in prompt design
* **Detect regressions early** - As Azure AI models evolve and get updated, prompt behavior may change. By running tests regularly, you can catch regressions that result from model updates or changes in prompt design

**Measure prompt quality**: Integration tests help you evaluate the quality of your prompts over time by establishing benchmarks for acceptable responses. You can track if the output still meets your defined criteria
* **Measure prompt quality** - Integration tests help you evaluate the quality of your prompts over time by establishing benchmarks for acceptable responses. You can track if the output still meets your defined criteria

**Test edge cases**: Prompts can behave unpredictably with edge case inputs. By testing common and edge case scenarios, you can ensure your AI model handles these situations gracefully
* **Test edge cases** - Prompts can behave unpredictably with edge case inputs. By testing common and edge case scenarios, you can ensure your AI model handles these situations gracefully

### Best practices for writing LLM integration tests
## Best practices for writing LLM integration tests

**Identify critical prompts**: Focus on writing tests for the most frequently used or mission-critical prompts in your application
* **Identify critical prompts** - Focus on writing tests for the most frequently used or mission-critical prompts in your application

**Set output expectations**: Define a range of acceptable output variations for your test cases. This might include specific keywords, response length, or adherence to format requirements
* **Set output expectations** - Define a range of acceptable output variations for your test cases. This might include specific keywords, response length, or adherence to format requirements

**Automate testing**: Use continuous integration (CI) pipelines to automatically run your LLM integration tests after each deployment or model update
* **Automate testing** - Use continuous integration (CI) pipelines to automatically run your LLM integration tests after each deployment or model update

**Log outputs**: Log the outputs from your tests to detect subtle changes over time. This can help identify patterns in model behavior and flag potential issues before they become problematic
* **Log outputs** - Log the outputs from your tests to detect subtle changes over time. This can help identify patterns in model behavior and flag potential issues before they become problematic

0 comments on commit 6a89fa5

Please sign in to comment.