Files
rengine/shared/utils
Yogesh Ojha 2c23238db3 feat(url_discovery): pace requests per host and back off a struggling one
A source-map or schema sweep sends many requests to one host, and a fragile or
rate-limited host answers a burst with timeouts and 429s that read as no result.
Requests to one host now pass through an adaptive governor: the per-request delay
doubles when the host returns a timeout, 429 or 503 and eases back as it answers
cleanly, with a cap on how many requests are in flight at once. A healthy host
runs at full speed.
2026-09-25 08:55:47 +05:30
..