[CI] Make JSON output tests less likely to fail

[russellb]

We occasionally see the JSON format structured output tests fail in CI.
PR #17490 included a change to the prompts asking to make the response
as short as possible. This change includes a couple more things to help:

• Increase the output length limit. The failures occur when we cut off
  the output before a JSON object is properly terminated.

• Set additionalProperties to False in each JSON schema used. This
  should restrict the model from adding properties not specified in the
  schemas, unnecessarily increasing the size of the JSON object output
  and making it more likely to hit the length limit before it finishes.

Signed-off-by: Russell Bryant rbryant@redhat.com

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

LGTM thanks

[DarkLight1337]

Hmm the test is still failing

[russellb]

Hmm the test is still failing

The part that failed is a spot I missed in my changes. I pushed an update. Hopefully this time it's green ...

[DarkLight1337]

There seems to be yet another error

[russellb]

This error in the last failure is:

[2025-05-08T18:07:45Z] /usr/local/lib/python3.12/dist-packages/schemathesis/generation/coverage.py:305: DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.
  | [2025-05-08T18:07:45Z] ref_error: type[Exception] = jsonschema.RefResolutionError,

This was the test:

[2025-05-08T18:07:45Z] FAILED v1/entrypoints/llm/test_struct_output_generate.py::test_structured_output[meta-llama/Meta-Llama-3.1-8B-Instruct-xgrammar-auto-speculative_config6] - vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.  

I'll dig into it ...

[russellb]

I ran the failed test in a loop for about an hour and never hit the same failure. I did hit a different one, but it turned out to be a known issue in xgrammar: mlc-ai/xgrammar#286. It's allowing invalid characters inside of a JSON string, so that's probably causing CI failures on occasion, as well.

I think the current PR is probably a net improvement, though there is still work to do.

[Stranger6667]

@russellb

[2025-05-08T18:07:45Z] /usr/local/lib/python3.12/dist-packages/schemathesis/generation/coverage.py:305: DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.
| [2025-05-08T18:07:45Z] ref_error: type[Exception] = jsonschema.RefResolutionError,

FWIW, from the Schemathesis side, you can safely ignore that specific warning, the jsonschema version is pinned in Schemathesis, and the removal of RefResolutionError will only happen in jsonschema==5. Probably, I need to add this warning filter to Schemathesis to avoid irrelevant output