Azure Annotation Message in API response breaks streaming

[jspv]

What happened?

Describe the bug
When streaming is enabled and using Azure hosted OpenAI models, we are often seeing a chunk at the end of the stream like the one below (actual chunk, not edited. Note the missing id value, the lack of delta in choices, etc.

Chunk:

{
  "id": "",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "content_filter_offsets": {
        "check_offset": 3329,
        "start_offset": 3329,
        "end_offset": 3438
      },
      "content_filter_results": {
        "hate": {
          "filtered": false,
          "severity": "safe"
        },
        "self_harm": {
          "filtered": false,
          "severity": "safe"
        },
        "sexual": {
          "filtered": false,
          "severity": "safe"
        },
        "violence": {
          "filtered": false,
          "severity": "safe"
        }
      }
    }
  ],
  "created": 0,
  "model": "",
  "object": ""
}

This is causing this block of code in _openai_client.py to raise AttributeError as there is no delta in choices.

            if choice.delta.model_extra is not None and "reasoning_content" in choice.delta.model_extra:
                # If there is a reasoning_content field, then we populate the thought field. This is for models such as R1.
                reasoning_content = choice.delta.model_extra.get("reasoning_content")

To Reproduce
We can reproduce this pretty consistently in our environment - content filtering annotations need to be enabled. The messages can be seen directly via curl (see my reply to this message)

Expected behavior
Exception should not be raised and streaming should continue.

Which packages was the bug in?

Python Extensions (autogen-ext)

AutoGen library version.

Python 0.5.6

Other library version.

No response

Model used

gpt-4.1

Model provider

Azure OpenAI

Other model provider

No response

Python version

3.11

.NET version

None

Operating system

Ubuntu

edit: more information

[jspv]

Additional information: This appears to be a content filtering annotation chunk as documented here: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter?tabs=warning%2Cuser-prompt%2Cpython-new

The message is in the form of:

data: { 
    "id": "", 
    "object": "", 
    "created": 0, 
    "model": "", 
    "choices": [ 
        { 
            "index": 0, 
            "finish_reason": null, 
            "content_filter_results": { ... }, 
            "content_filter_raw": [ ... ], 
            "content_filter_offsets": { 
                "check_offset": 44, 
                "start_offset": 44, 
                "end_offset": 198 
            } 
        } 
    ], 
    "usage": null 
}

It can be seen directly via curl if annotations are turned on. As the message does not have a delta attribute, an exception is raised and the stream breaks.

 curl "https://ENDPOINT/openai/deployments/gpt-4o/chat/completions?api-version=2024-10-21" \
  -H "Authorization: Bearer XXXXXX" \
  -H "Content-Type: application/json" \
  -d '{
      "model": "gpt-4o",
      "messages": [{"role": "user", "content": "Say this is a test!"}],
      "stream": true
  }'

data: {"choices":[],"created":0,"id":"","model":"","object":"","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}]}

data: {"choices":[{"delta":{"content":"","refusal":null,"role":"assistant"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":"This"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":" is"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":" a"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":" test"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":"!"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":" If"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":" you"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":" have"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":" any"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":" questions"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":" or"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":" need"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":" assistance"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":","},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":" feel"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":" free"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":" to"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":" ask"},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{"content":"."},"finish_reason":null,"index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"delta":{},"finish_reason":"stop","index":0,"logprobs":null}],"created":1746660649,"id":"chatcmpl-BUiVtPh8lynzTpEEOhyvc7uepB2nm","model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","system_fingerprint":"fp_ee1d74bde0"}

data: {"choices":[{"content_filter_offsets":{"check_offset":49,"start_offset":49,"end_offset":128},"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"finish_reason":null,"index":0}],"created":0,"id":"","model":"","object":""}

data: [DONE]