This PR adds error handling and retry mechanisms for create_agent
structured output via a `handle_errors` parameter in `ToolOutput`.
Changes:
* Adds `MultipleStructuredOutputsError` exception for when models
incorrectly call multiple structured output tools simultaneously
* Adds `StructuredOutputParsingError` exception for when tool arguments
fail to parse according to the schema
* Implements automatic error handling logic that re-prompts the model
with a configurable error message when structured output failures occur
via `handle_errors` policy in `ToolOutput`:
```python
class ToolOutput:
...
handle_errors: Union[
bool, # True: retry all, False: no retry
str, # Custom static error message for all errors
type[Exception], # Retry only this exception type
tuple[type[Exception], ...], # Retry only these exception types
Callable[[Exception], str], # Custom callable returning error message
]
"""Error handling strategy. Default: True (retry on all error types with default error message)"""
```
Examples:
```python
# Retry all errors
ToolOutput(WeatherReport)
# No retry
ToolOutput(WeatherReport, handle_errors=False)
# Custom message for all errors
ToolOutput(WeatherReport, handle_errors="Please provide valid data")
# Only retry specific error type
ToolOutput(WeatherReport, handle_errors=StructuredOutputParsingError)
# Multiple error types
ToolOutput(WeatherReport, handle_errors=(MultipleStructuredOutputsError, StructuredOutputParsingError))
# Custom logic
ToolOutput(
Union[WeatherReport, LocationInfo],
handle_errors=lambda e: "Only one response please" if isinstance(e, MultipleStructuredOutputsError) else "Invalid format"
)
```
---------
Co-authored-by: Sydney Runkle <sydneymarierunkle@gmail.com>
This PR improves standard integration tests in prebuilt to ensure
logical equivalence between the Python and JavaScript implementations of
`create_agent`.
* Cleans up `test_responses_int` test harness
* Adds new test utils to dynamically load JSON test specs
* Adds `test_return_direct_int` test harness to validate model behavior
when `return_direct` tool property is set
* Adds support for when the user instantiates `ToolOutput` with multiple
JSON schemas unified by the `oneOf` keyword
Failing tests:
* `test_inference_to_native_output`: there is some odd behavior where
the model makes a second call to `get_weather` despite having just
received the tool message, so there are 6 messages instead of the 4
expected.
* `test_responses_integration_matrix[asking for information that does
not fit into the response format]`: `XFAIL`, currently failing due to
undefined behavior when the model cannot conform to any of the
structured response formats.
TODO in future PRs:
* Add exception handling to pass `test_responses_integration_matrix`.
* Adds support for `NativeOutput` via a new `NativeOutput` dataclass
* Adds support for structured output specification via the following
(pydantic models already supported)
* dataclasses
* typed dicts
* json schemas
* Adds mocking support to support native strategies with
`FakeToolCallingModel`
* Add new default tool message when `tool_message_content` not provided
* Smart "selection" of native vs tool output based on provider support,
necessitates profiles down the line
Considered questions
* do we want to enforce docstrings? -- decided on no for now
* do we want to enforce names (titles) on json schemas? -- decided no
for now, defaulting to `structured_output`
* do we want to validate that json schemas coming in are valid? --
decided no for now
* do we want to validate model results against a given json schema? we
validate against all other types (typed dict, dataclass, etc) w/
pydantic -- decided no for now
TODO in future PRs:
* Figure out retry policy
* Add standard testing (handed off to @casparb)
* Further privatize certain structures (like the bindings) -- this is
low prio
---------
Co-authored-by: Sydney Runkle <sydneymarierunkle@gmail.com>
Co-authored-by: Sydney Runkle <54324534+sydney-runkle@users.noreply.github.com>
