Caspar BroekhuizenandGitHub 77d98b426b test(prebuilt): standard integration tests for create_agent (#5988) ...

This PR improves standard integration tests in prebuilt to ensure
logical equivalence between the Python and JavaScript implementations of
`create_agent`.
* Cleans up `test_responses_int` test harness
* Adds new test utils to dynamically load JSON test specs
* Adds `test_return_direct_int` test harness to validate model behavior
when `return_direct` tool property is set
* Adds support for when the user instantiates `ToolOutput` with multiple
JSON schemas unified by the `oneOf` keyword

Failing tests:
* `test_inference_to_native_output`: there is some odd behavior where
the model makes a second call to `get_weather` despite having just
received the tool message, so there are 6 messages instead of the 4
expected.
* `test_responses_integration_matrix[asking for information that does
not fit into the response format]`: `XFAIL`, currently failing due to
undefined behavior when the model cannot conform to any of the
structured response formats.

TODO in future PRs:
* Add exception handling to pass `test_responses_integration_matrix`.

2025-08-22 13:55:09 -04:00

..

responses.json

test(prebuilt): standard integration tests for create_agent (#5988)

2025-08-22 13:55:09 -04:00

return_direct.json

test(prebuilt): standard integration tests for create_agent (#5988)

2025-08-22 13:55:09 -04:00