This PR improves standard integration tests in prebuilt to ensure
logical equivalence between the Python and JavaScript implementations of
`create_agent`.
* Cleans up `test_responses_int` test harness
* Adds new test utils to dynamically load JSON test specs
* Adds `test_return_direct_int` test harness to validate model behavior
when `return_direct` tool property is set
* Adds support for when the user instantiates `ToolOutput` with multiple
JSON schemas unified by the `oneOf` keyword
Failing tests:
* `test_inference_to_native_output`: there is some odd behavior where
the model makes a second call to `get_weather` despite having just
received the tool message, so there are 6 messages instead of the 4
expected.
* `test_responses_integration_matrix[asking for information that does
not fit into the response format]`: `XFAIL`, currently failing due to
undefined behavior when the model cannot conform to any of the
structured response formats.
TODO in future PRs:
* Add exception handling to pass `test_responses_integration_matrix`.
* Adds support for `NativeOutput` via a new `NativeOutput` dataclass
* Adds support for structured output specification via the following
(pydantic models already supported)
* dataclasses
* typed dicts
* json schemas
* Adds mocking support to support native strategies with
`FakeToolCallingModel`
* Add new default tool message when `tool_message_content` not provided
* Smart "selection" of native vs tool output based on provider support,
necessitates profiles down the line
Considered questions
* do we want to enforce docstrings? -- decided on no for now
* do we want to enforce names (titles) on json schemas? -- decided no
for now, defaulting to `structured_output`
* do we want to validate that json schemas coming in are valid? --
decided no for now
* do we want to validate model results against a given json schema? we
validate against all other types (typed dict, dataclass, etc) w/
pydantic -- decided no for now
TODO in future PRs:
* Figure out retry policy
* Add standard testing (handed off to @casparb)
* Further privatize certain structures (like the bindings) -- this is
low prio
---------
Co-authored-by: Sydney Runkle <sydneymarierunkle@gmail.com>
Co-authored-by: Sydney Runkle <54324534+sydney-runkle@users.noreply.github.com>
