From 230927fb3a9ac9b2893a30322b4dfea7cdea9a8f Mon Sep 17 00:00:00 2001 From: Lauren Hirata Singh Date: Mon, 14 Sep 2026 15:45:19 -0400 Subject: [PATCH] docs: publish the generated llms.txt instead of a hand-written copy (#8922) ## Why `docs/llms.txt` is written by hand and was last touched in February, so the index served at had drifted badly: - 12 pages listed, against the 43 the docs site publishes. - One entry, `/oss/python/langgraph/why-langgraph`, now 404s. The deploy workflow only fires on `docs/**` pushes, so nothing brought the file back in line as the docs site changed. It is served verbatim: `generate_redirects.py` copies it into `docs/_site` via a `static_files` list, and `deploy-redirects.yml` uploads that directory to Pages. The docs site already generates exactly this index, and regenerates it on every deploy: ``` https://docs.langchain.com/oss/python/langgraph/llms.txt ``` ## What changed `generate_redirects.py` now fetches that index at build time and writes it into `_site`, so the published file cannot drift from the docs site. The committed `docs/llms.txt` stays as a fallback only, refreshed here to current content. `deploy-redirects.yml` gains a weekly `schedule:` trigger, so docs changes that never touch this repo still reach the deployed file. The `permissions:` block is unchanged. ## Fetching remote content in CI The fetched body is published on a public Pages site, so it is validated before it is written: - The URL is a hardcoded module constant, never built from input. - Both that URL and the post-redirect `response.url` are checked against an HTTPS-plus-single-host allowlist. `urlopen` follows redirects, so checking only the request URL would not be enough. - 30 second timeout, response capped at 1MB, decoded as UTF-8. - The body must open with a markdown heading and contain a docs.langchain.com link, which rejects an error page or a truncated response. Any failure returns `None` and falls back to the committed copy, so a docs.langchain.com outage degrades to a stale file rather than a broken deploy or a published error page. ## Verification Ran `python docs/generate_redirects.py`: 294 redirect files, `llms.txt` fetched, 43 entries. Each rejection path was exercised against a stubbed `urlopen` and all fall back to the committed file: | Case | Result | |---|---| | Network error | falls back | | Timeout | falls back | | Redirect to another host | falls back | | Body over 1MB | falls back | | Invalid UTF-8 | falls back | | HTML error page | falls back | | Empty body | falls back | | Valid index | published | Allowlist rejects `http://docs.langchain.com/...`, `https://evil.com/...`, and `https://docs.langchain.com.evil.com/...`. ## Note on ordering The fallback committed here is labelled "LangGraph (Python)", which is what the docs site will serve once langchain-ai/docs#6032 deploys. Until then the fetched file reads "Open source (Python)". No action needed; it self-corrects. --- Written with Claude Code; I reviewed the diff and ran the verification above. --- .github/workflows/deploy-redirects.yml | 4 ++ docs/generate_redirects.py | 90 ++++++++++++++++++++++++-- docs/llms.txt | 78 +++++++++++++--------- 3 files changed, 133 insertions(+), 39 deletions(-) diff --git a/.github/workflows/deploy-redirects.yml b/.github/workflows/deploy-redirects.yml index 81475ba2d..bed8aad74 100644 --- a/.github/workflows/deploy-redirects.yml +++ b/.github/workflows/deploy-redirects.yml @@ -7,6 +7,10 @@ on: paths: - 'docs/**' - '.github/workflows/deploy-redirects.yml' + # llms.txt is fetched from docs.langchain.com at build time, so redeploy on a + # schedule to pick up docs changes that never touch this repo. + schedule: + - cron: '17 6 * * 1' workflow_dispatch: permissions: diff --git a/docs/generate_redirects.py b/docs/generate_redirects.py index e8555ebdf..0afad1aad 100644 --- a/docs/generate_redirects.py +++ b/docs/generate_redirects.py @@ -12,13 +12,26 @@ which is SEO-friendly and treated similarly to 301 redirects by Google. To add new redirects, simply edit redirects.json and re-run this script. """ +import http.client import json import os +import urllib.error +import urllib.parse +import urllib.request from pathlib import Path # Default fallback URL for any path not in the redirect map DEFAULT_REDIRECT = "https://docs.langchain.com/oss/python/langgraph/overview" +# The docs site regenerates this index on every deploy, so fetching it here +# keeps the published llms.txt from drifting. The URL is a hardcoded constant, +# never built from input, and both it and the post-redirect URL are checked +# against ALLOWED_LLMS_HOST before anything is read. +CANONICAL_LLMS_URL = "https://docs.langchain.com/oss/python/langgraph/llms.txt" +ALLOWED_LLMS_HOST = "docs.langchain.com" +LLMS_FETCH_TIMEOUT = 30 +LLMS_MAX_BYTES = 1_000_000 + HTML_TEMPLATE = """ @@ -75,6 +88,64 @@ CATCHALL_404_TEMPLATE = """ """ +def is_allowed_llms_url(url): + """Return True if url is HTTPS on the one host we accept content from.""" + parsed = urllib.parse.urlsplit(url) + return parsed.scheme == "https" and parsed.hostname == ALLOWED_LLMS_HOST + + +def fetch_canonical_llms_txt(): + """Return the published LangGraph index, or None if it cannot be used. + + Returning None leaves the caller on the committed docs/llms.txt, so a + docs.langchain.com outage degrades to a stale file rather than a broken + deploy or a published error page. + """ + if not is_allowed_llms_url(CANONICAL_LLMS_URL): + print(f"Refusing to fetch {CANONICAL_LLMS_URL}: host not allowed") + return None + + try: + with urllib.request.urlopen( # noqa: S310 - constant, allowlisted URL + CANONICAL_LLMS_URL, timeout=LLMS_FETCH_TIMEOUT + ) as response: + # urlopen follows redirects, so re-check where it actually landed. + if not is_allowed_llms_url(response.url): + print(f"Refusing {CANONICAL_LLMS_URL}: redirected to {response.url}") + return None + body = response.read(LLMS_MAX_BYTES + 1) + # A connection dropped mid-body raises http.client.IncompleteRead, which + # descends from HTTPException rather than OSError, so catching only the + # urllib and OS errors would let it escape and fail the whole deploy. + except ( + urllib.error.URLError, + http.client.HTTPException, + TimeoutError, + OSError, + ) as exc: + print(f"Could not fetch {CANONICAL_LLMS_URL}: {type(exc).__name__}: {exc}") + return None + + if len(body) > LLMS_MAX_BYTES: + print(f"Refusing {CANONICAL_LLMS_URL}: larger than {LLMS_MAX_BYTES} bytes") + return None + + try: + text = body.decode("utf-8") + except UnicodeDecodeError as exc: + print(f"Refusing {CANONICAL_LLMS_URL}: not valid UTF-8: {exc}") + return None + + # An index opens with a markdown heading and links to the docs site. A + # body that does not is an error page or a truncated response, not content + # worth publishing. + if not text.startswith("# ") or f"https://{ALLOWED_LLMS_HOST}/" not in text: + print(f"Refusing {CANONICAL_LLMS_URL}: does not look like an llms.txt index") + return None + + return text + + def generate_redirects(): script_dir = Path(__file__).parent output_dir = script_dir / "_site" @@ -126,14 +197,19 @@ def generate_redirects(): catchall_404.write_text(CATCHALL_404_TEMPLATE.format(default_url=DEFAULT_REDIRECT)) print(f"Created: {catchall_404}") - # Copy static files (like llms.txt) that can't be redirected via HTML - static_files = ["llms.txt"] - for static_file in static_files: - src = script_dir / static_file + # llms.txt can't be redirected via HTML, so publish the docs site's own + # generated index. The committed copy is only a fallback. + llms_txt = fetch_canonical_llms_txt() + if llms_txt is not None: + (output_dir / "llms.txt").write_text(llms_txt) + print(f"Fetched: {output_dir / 'llms.txt'} (from {CANONICAL_LLMS_URL})") + else: + src = script_dir / "llms.txt" if src.exists(): - dst = output_dir / static_file - dst.write_text(src.read_text()) - print(f"Copied: {dst}") + (output_dir / "llms.txt").write_text(src.read_text()) + print(f"Copied: {output_dir / 'llms.txt'} (fallback, may be stale)") + else: + print("No llms.txt fetched and no committed fallback; skipping") print(f"\nGenerated {len(redirects)} redirect files in {output_dir}") diff --git a/docs/llms.txt b/docs/llms.txt index 75017c34f..3f498c6db 100644 --- a/docs/llms.txt +++ b/docs/llms.txt @@ -1,35 +1,49 @@ -# LangGraph +# Docs by LangChain: LangGraph (Python) -LangGraph documentation has moved to docs.langchain.com. +> Markdown index of the LangGraph (Python) documentation. -## Overview +## LangGraph (Python) -- [LangGraph Overview](https://docs.langchain.com/oss/python/langgraph/overview): Introduction to LangGraph, a library for building stateful, multi-actor applications with LLMs. -- [Why LangGraph?](https://docs.langchain.com/oss/python/langgraph/why-langgraph): Motivation for LangGraph and its key features. - -## Core Concepts - -- [Graph API](https://docs.langchain.com/oss/python/langgraph/graph-api): Learn how to define state, create nodes, and connect them with edges. -- [Streaming](https://docs.langchain.com/oss/python/langgraph/streaming): Stream outputs from your graph for better UX. -- [Persistence](https://docs.langchain.com/oss/python/langgraph/persistence): Add memory and checkpointing to your graphs. -- [Add Memory](https://docs.langchain.com/oss/python/langgraph/add-memory): Implement short-term and long-term memory. -- [Workflows & Agents](https://docs.langchain.com/oss/python/langgraph/workflows-agents): Build agents and workflows with LangGraph. - -## How-To Guides - -- [Use Subgraphs](https://docs.langchain.com/oss/python/langgraph/use-subgraphs): Compose graphs using subgraphs. -- [Observability](https://docs.langchain.com/oss/python/langgraph/observability): Add tracing and debugging to your graphs. -- [Common Errors](https://docs.langchain.com/oss/python/langgraph/common-errors): Troubleshoot common LangGraph errors. - -## Tutorials - -- [Agentic RAG](https://docs.langchain.com/oss/python/langgraph/agentic-rag): Build an agentic RAG system with LangGraph. -- [SQL Agent](https://docs.langchain.com/oss/python/langgraph/sql-agent): Create a SQL agent with LangGraph. - -## Reference - -- [API Reference](https://reference.langchain.com/python/langgraph/): Complete API documentation for LangGraph. - -## LangGraph Platform - -For deploying LangGraph applications in production, see the [LangSmith documentation](https://docs.langchain.com/langsmith/agent-server). +- [Memory](https://docs.langchain.com/oss/python/langgraph/add-memory.md) +- [Build a custom RAG agent with LangGraph](https://docs.langchain.com/oss/python/langgraph/agentic-rag.md) +- [Application structure](https://docs.langchain.com/oss/python/langgraph/application-structure.md) +- [Backward compatibility](https://docs.langchain.com/oss/python/langgraph/backward-compatibility.md) +- [Case studies](https://docs.langchain.com/oss/python/langgraph/case-studies.md) +- [Changelog](https://docs.langchain.com/oss/python/langgraph/changelog-js.md) +- [Changelog](https://docs.langchain.com/oss/python/langgraph/changelog-py.md) +- [Checkpointers](https://docs.langchain.com/oss/python/langgraph/checkpointers.md) +- [Choosing between Graph and Functional APIs](https://docs.langchain.com/oss/python/langgraph/choosing-apis.md) +- [Deployment](https://docs.langchain.com/oss/python/langgraph/deploy.md) +- [GRAPH_RECURSION_LIMIT](https://docs.langchain.com/oss/python/langgraph/errors/GRAPH_RECURSION_LIMIT.md) +- [INVALID_CHAT_HISTORY](https://docs.langchain.com/oss/python/langgraph/errors/INVALID_CHAT_HISTORY.md) +- [INVALID_CONCURRENT_GRAPH_UPDATE](https://docs.langchain.com/oss/python/langgraph/errors/INVALID_CONCURRENT_GRAPH_UPDATE.md) +- [INVALID_GRAPH_NODE_RETURN_VALUE](https://docs.langchain.com/oss/python/langgraph/errors/INVALID_GRAPH_NODE_RETURN_VALUE.md) +- [MISSING_CHECKPOINTER](https://docs.langchain.com/oss/python/langgraph/errors/MISSING_CHECKPOINTER.md) +- [MULTIPLE_SUBGRAPHS](https://docs.langchain.com/oss/python/langgraph/errors/MULTIPLE_SUBGRAPHS.md) +- [Event streaming](https://docs.langchain.com/oss/python/langgraph/event-streaming.md) +- [Fault tolerance](https://docs.langchain.com/oss/python/langgraph/fault-tolerance.md) +- [Custom stream channels](https://docs.langchain.com/oss/python/langgraph/frontend/custom-stream-channels.md) +- [Graph execution](https://docs.langchain.com/oss/python/langgraph/frontend/graph-execution.md) +- [Overview](https://docs.langchain.com/oss/python/langgraph/frontend/overview.md) +- [Functional API overview](https://docs.langchain.com/oss/python/langgraph/functional-api.md) +- [Graph API overview](https://docs.langchain.com/oss/python/langgraph/graph-api.md) +- [Install LangGraph](https://docs.langchain.com/oss/python/langgraph/install.md) +- [Interrupts](https://docs.langchain.com/oss/python/langgraph/interrupts.md) +- [Run a local server](https://docs.langchain.com/oss/python/langgraph/local-server.md) +- [LangSmith Observability](https://docs.langchain.com/oss/python/langgraph/observability.md) +- [LangGraph overview](https://docs.langchain.com/oss/python/langgraph/overview.md) +- [Persistence](https://docs.langchain.com/oss/python/langgraph/persistence.md) +- [LangGraph runtime](https://docs.langchain.com/oss/python/langgraph/pregel.md) +- [Quickstart](https://docs.langchain.com/oss/python/langgraph/quickstart.md) +- [Build a custom SQL agent](https://docs.langchain.com/oss/python/langgraph/sql-agent.md) +- [Stores](https://docs.langchain.com/oss/python/langgraph/stores.md) +- [Streaming](https://docs.langchain.com/oss/python/langgraph/streaming.md) +- [LangSmith Studio](https://docs.langchain.com/oss/python/langgraph/studio.md) +- [Test](https://docs.langchain.com/oss/python/langgraph/test.md) +- [Thinking in LangGraph](https://docs.langchain.com/oss/python/langgraph/thinking-in-langgraph.md) +- [Agent Chat UI](https://docs.langchain.com/oss/python/langgraph/ui.md) +- [Use the functional API](https://docs.langchain.com/oss/python/langgraph/use-functional-api.md) +- [Use the graph API](https://docs.langchain.com/oss/python/langgraph/use-graph-api.md) +- [Subgraphs](https://docs.langchain.com/oss/python/langgraph/use-subgraphs.md) +- [Use time-travel](https://docs.langchain.com/oss/python/langgraph/use-time-travel.md) +- [Workflows and agents](https://docs.langchain.com/oss/python/langgraph/workflows-agents.md)