mirror of
https://github.com/langchain-ai/langgraph.git
synced 2026-09-16 14:47:55 +02:00
Lauren Hirata SinghandGitHub
230927fb3a
## Why `docs/llms.txt` is written by hand and was last touched in February, so the index served at <https://langchain-ai.github.io/langgraph/llms.txt> had drifted badly: - 12 pages listed, against the 43 the docs site publishes. - One entry, `/oss/python/langgraph/why-langgraph`, now 404s. The deploy workflow only fires on `docs/**` pushes, so nothing brought the file back in line as the docs site changed. It is served verbatim: `generate_redirects.py` copies it into `docs/_site` via a `static_files` list, and `deploy-redirects.yml` uploads that directory to Pages. The docs site already generates exactly this index, and regenerates it on every deploy: ``` https://docs.langchain.com/oss/python/langgraph/llms.txt ``` ## What changed `generate_redirects.py` now fetches that index at build time and writes it into `_site`, so the published file cannot drift from the docs site. The committed `docs/llms.txt` stays as a fallback only, refreshed here to current content. `deploy-redirects.yml` gains a weekly `schedule:` trigger, so docs changes that never touch this repo still reach the deployed file. The `permissions:` block is unchanged. ## Fetching remote content in CI The fetched body is published on a public Pages site, so it is validated before it is written: - The URL is a hardcoded module constant, never built from input. - Both that URL and the post-redirect `response.url` are checked against an HTTPS-plus-single-host allowlist. `urlopen` follows redirects, so checking only the request URL would not be enough. - 30 second timeout, response capped at 1MB, decoded as UTF-8. - The body must open with a markdown heading and contain a docs.langchain.com link, which rejects an error page or a truncated response. Any failure returns `None` and falls back to the committed copy, so a docs.langchain.com outage degrades to a stale file rather than a broken deploy or a published error page. ## Verification Ran `python docs/generate_redirects.py`: 294 redirect files, `llms.txt` fetched, 43 entries. Each rejection path was exercised against a stubbed `urlopen` and all fall back to the committed file: | Case | Result | |---|---| | Network error | falls back | | Timeout | falls back | | Redirect to another host | falls back | | Body over 1MB | falls back | | Invalid UTF-8 | falls back | | HTML error page | falls back | | Empty body | falls back | | Valid index | published | Allowlist rejects `http://docs.langchain.com/...`, `https://evil.com/...`, and `https://docs.langchain.com.evil.com/...`. ## Note on ordering The fallback committed here is labelled "LangGraph (Python)", which is what the docs site will serve once langchain-ai/docs#6032 deploys. Until then the fetched file reads "Open source (Python)". No action needed; it self-corrects. --- Written with Claude Code; I reviewed the diff and ran the verification above.