{
 "cells": [
  {
   "attachments": {
    "15cba0ab-a549-4909-8373-fb761e384eff.png": {
     "image/png": "[encoded data, 141944 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "919fe33c-0149-4f7d-b200-544a18986c9a",
   "metadata": {},
   "source": [
    "# Self-RAG\n",
    "\n",
    "Self-RAG is a strategy for RAG that incorporates self-reflection / self-grading on retrieved documents and generations. \n",
    "\n",
    "In the [paper](https://arxiv.org/abs/2310.11511), a few decisions are made:\n",
    "\n",
    "1. Should I retrieve from retriever, `R` -\n",
    "\n",
    "* Input: `x (question)` OR `x (question)`, `y (generation)`\n",
    "* Decides when to retrieve `D` chunks with `R`\n",
    "* Output: `yes, no, continue`\n",
    "\n",
    "2. Are the retrieved passages `D` relevant to the question `x` -\n",
    "\n",
    "* * Input: (`x (question)`, `d (chunk)`) for `d` in `D`\n",
    "* `d` provides useful information to solve `x`\n",
    "* Output: `relevant, irrelevant`\n",
    "\n",
    "3. Are the LLM generation from each chunk in `D` is relevant to the chunk (hallucinations, etc)  -\n",
    "\n",
    "* Input: `x (question)`, `d (chunk)`,  `y (generation)` for `d` in `D`\n",
    "* All of the verification-worthy statements in `y (generation)` are supported by `d`\n",
    "* Output: `{fully supported, partially supported, no support`\n",
    "\n",
    "4. The LLM generation from each chunk in `D` is a useful response to `x (question)` -\n",
    "\n",
    "* Input: `x (question)`, `y (generation)` for `d` in `D`\n",
    "* `y (generation)` is a useful response to `x (question)`.\n",
    "* Output: `{5, 4, 3, 2, 1}`\n",
    "\n",
    "We will implement some of these ideas from scratch using [LangGraph](https://langchain-ai.github.io/langgraph/).\n",
    "\n",
    "![Screenshot 2024-04-01 at 12.41.50 PM.png](attachment:15cba0ab-a549-4909-8373-fb761e384eff.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72f3ee57-68ab-4040-bd36-4014e2a23d96",
   "metadata": {},
   "source": [
    "# Environment "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a384cc48-0425-4e8f-aafc-cfb8e56025c9",
   "metadata": {},
   "outputs": [],
   "source": ["! pip install -U langchain_community tiktoken langchain-openai langchainhub chromadb langchain langgraph"]
  },
  {
   "cell_type": "markdown",
   "id": "15569b93-3c68-4aac-838c-37112d33987a",
   "metadata": {},
   "source": [
    "### LLMs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f18b63c7-d0d3-41c1-ae6b-5a0f1b8ccf0f",
   "metadata": {},
   "outputs": [],
   "source": ["import os\n\nos.environ[\"OPENAI_API_KEY\"] = \"<your-api-key>\""]
  },
  {
   "cell_type": "markdown",
   "id": "532d91fb-381e-4e11-b3b1-254321351773",
   "metadata": {},
   "source": [
    "### Tracing\n",
    "\n",
    "Optionally, use [LangSmith](https://docs.smith.langchain.com/) for tracing (shown at bottom)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccc3dae5-1df6-48ca-af8a-50f0e6128876",
   "metadata": {},
   "outputs": [],
   "source": ["os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\nos.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.smith.langchain.com\"\nos.environ[\"LANGCHAIN_API_KEY\"] = \"<your-api-key>\""]
  },
  {
   "cell_type": "markdown",
   "id": "c27bebdc-be71-4130-ab9d-42f09f87658b",
   "metadata": {},
   "source": [
    "## Retriever\n",
    " \n",
    "Let's index 3 blog posts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "565a6d44-2c9f-4fff-b1ec-eea05df9350d",
   "metadata": {},
   "outputs": [],
   "source": ["from langchain.text_splitter import RecursiveCharacterTextSplitter\nfrom langchain_community.document_loaders import WebBaseLoader\nfrom langchain_community.vectorstores import Chroma\nfrom langchain_openai import OpenAIEmbeddings\n\nurls = [\n    \"https://lilianweng.github.io/posts/2023-06-23-agent/\",\n    \"https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/\",\n    \"https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm/\",\n]\n\ndocs = [WebBaseLoader(url).load() for url in urls]\ndocs_list = [item for sublist in docs for item in sublist]\n\ntext_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(\n    chunk_size=250, chunk_overlap=0\n)\ndoc_splits = text_splitter.split_documents(docs_list)\n\n# Add to vectorDB\nvectorstore = Chroma.from_documents(\n    documents=doc_splits,\n    collection_name=\"rag-chroma\",\n    embedding=OpenAIEmbeddings(),\n)\nretriever = vectorstore.as_retriever()"]
  },
  {
   "cell_type": "markdown",
   "id": "29c12f74-53e2-43cc-896f-875d1c5d9d93",
   "metadata": {},
   "source": [
    "## LLMs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1fafad21-60cc-483e-92a3-6a7edb1838e3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rlm/miniforge3/envs/llama2/lib/python3.11/site-packages/langchain_core/_api/deprecation.py:119: LangChainDeprecationWarning: The method `BaseRetriever.get_relevant_documents` was deprecated in langchain-core 0.1.46 and will be removed in 0.3.0. Use invoke instead.\n",
      "  warn_deprecated(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "binary_score='yes'\n"
     ]
    }
   ],
   "source": ["### Retrieval Grader\n\n\nfrom langchain_core.prompts import ChatPromptTemplate\nfrom langchain_core.pydantic_v1 import BaseModel, Field\nfrom langchain_openai import ChatOpenAI\n\n\n# Data model\nclass GradeDocuments(BaseModel):\n    \"\"\"Binary score for relevance check on retrieved documents.\"\"\"\n\n    binary_score: str = Field(\n        description=\"Documents are relevant to the question, 'yes' or 'no'\"\n    )\n\n\n# LLM with function call\nllm = ChatOpenAI(model=\"gpt-3.5-turbo-0125\", temperature=0)\nstructured_llm_grader = llm.with_structured_output(GradeDocuments)\n\n# Prompt\nsystem = \"\"\"You are a grader assessing relevance of a retrieved document to a user question. \\n \n    It does not need to be a stringent test. The goal is to filter out erroneous retrievals. \\n\n    If the document contains keyword(s) or semantic meaning related to the user question, grade it as relevant. \\n\n    Give a binary score 'yes' or 'no' score to indicate whether the document is relevant to the question.\"\"\"\ngrade_prompt = ChatPromptTemplate.from_messages(\n    [\n        (\"system\", system),\n        (\"human\", \"Retrieved document: \\n\\n {document} \\n\\n User question: {question}\"),\n    ]\n)\n\nretrieval_grader = grade_prompt | structured_llm_grader\nquestion = \"agent memory\"\ndocs = retriever.get_relevant_documents(question)\ndoc_txt = docs[1].page_content\nprint(retrieval_grader.invoke({\"question\": question, \"document\": doc_txt}))"]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "dcd77cc1-4587-40ec-b633-5364eab9e1ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The design of generative agents combines LLM with memory, planning, and reflection mechanisms to enable agents to behave conditioned on past experience and interact with other agents. Long-term memory provides the agent with the capability to retain and recall infinite information over extended periods. Short-term memory is utilized for in-context learning.\n"
     ]
    }
   ],
   "source": ["### Generate\n\nfrom langchain import hub\nfrom langchain_core.output_parsers import StrOutputParser\n\n# Prompt\nprompt = hub.pull(\"rlm/rag-prompt\")\n\n# LLM\nllm = ChatOpenAI(model_name=\"gpt-3.5-turbo\", temperature=0)\n\n\n# Post-processing\ndef format_docs(docs):\n    return \"\\n\\n\".join(doc.page_content for doc in docs)\n\n\n# Chain\nrag_chain = prompt | llm | StrOutputParser()\n\n# Run\ngeneration = rag_chain.invoke({\"context\": docs, \"question\": question})\nprint(generation)"]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e78931ec-940c-46ad-a0b2-f43f953f1fd7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GradeHallucinations(binary_score='yes')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": ["### Hallucination Grader\n\n\n# Data model\nclass GradeHallucinations(BaseModel):\n    \"\"\"Binary score for hallucination present in generation answer.\"\"\"\n\n    binary_score: str = Field(\n        description=\"Answer is grounded in the facts, 'yes' or 'no'\"\n    )\n\n\n# LLM with function call\nllm = ChatOpenAI(model=\"gpt-3.5-turbo-0125\", temperature=0)\nstructured_llm_grader = llm.with_structured_output(GradeHallucinations)\n\n# Prompt\nsystem = \"\"\"You are a grader assessing whether an LLM generation is grounded in / supported by a set of retrieved facts. \\n \n     Give a binary score 'yes' or 'no'. 'Yes' means that the answer is grounded in / supported by the set of facts.\"\"\"\nhallucination_prompt = ChatPromptTemplate.from_messages(\n    [\n        (\"system\", system),\n        (\"human\", \"Set of facts: \\n\\n {documents} \\n\\n LLM generation: {generation}\"),\n    ]\n)\n\nhallucination_grader = hallucination_prompt | structured_llm_grader\nhallucination_grader.invoke({\"documents\": docs, \"generation\": generation})"]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "bd62276f-bf26-40d0-8cff-e07b10e00321",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GradeAnswer(binary_score='yes')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": ["### Answer Grader\n\n\n# Data model\nclass GradeAnswer(BaseModel):\n    \"\"\"Binary score to assess answer addresses question.\"\"\"\n\n    binary_score: str = Field(\n        description=\"Answer addresses the question, 'yes' or 'no'\"\n    )\n\n\n# LLM with function call\nllm = ChatOpenAI(model=\"gpt-3.5-turbo-0125\", temperature=0)\nstructured_llm_grader = llm.with_structured_output(GradeAnswer)\n\n# Prompt\nsystem = \"\"\"You are a grader assessing whether an answer addresses / resolves a question \\n \n     Give a binary score 'yes' or 'no'. Yes' means that the answer resolves the question.\"\"\"\nanswer_prompt = ChatPromptTemplate.from_messages(\n    [\n        (\"system\", system),\n        (\"human\", \"User question: \\n\\n {question} \\n\\n LLM generation: {generation}\"),\n    ]\n)\n\nanswer_grader = answer_prompt | structured_llm_grader\nanswer_grader.invoke({\"question\": question, \"generation\": generation})"]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c6f4c70e-1660-4149-82c0-837f19fc9fb5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"What is the role of memory in an agent's functioning?\""
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": ["### Question Re-writer\n\n# LLM\nllm = ChatOpenAI(model=\"gpt-3.5-turbo-0125\", temperature=0)\n\n# Prompt\nsystem = \"\"\"You a question re-writer that converts an input question to a better version that is optimized \\n \n     for vectorstore retrieval. Look at the input and try to reason about the underlying semantic intent / meaning.\"\"\"\nre_write_prompt = ChatPromptTemplate.from_messages(\n    [\n        (\"system\", system),\n        (\n            \"human\",\n            \"Here is the initial question: \\n\\n {question} \\n Formulate an improved question.\",\n        ),\n    ]\n)\n\nquestion_rewriter = re_write_prompt | llm | StrOutputParser()\nquestion_rewriter.invoke({\"question\": question})"]
  },
  {
   "cell_type": "markdown",
   "id": "276001c5-c079-4e5b-9f42-81a06704d200",
   "metadata": {},
   "source": [
    "# Graph \n",
    "\n",
    "Capture the flow in as a graph.\n",
    "\n",
    "## Graph state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f1617e9e-66a8-4c1a-a1fe-cc936284c085",
   "metadata": {},
   "outputs": [],
   "source": ["from typing import List\n\nfrom typing_extensions import TypedDict\n\n\nclass GraphState(TypedDict):\n    \"\"\"\n    Represents the state of our graph.\n\n    Attributes:\n        question: question\n        generation: LLM generation\n        documents: list of documents\n    \"\"\"\n\n    question: str\n    generation: str\n    documents: List[str]"]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "add509d8-6682-4127-8d95-13dd37d79702",
   "metadata": {},
   "outputs": [],
   "source": ["### Nodes\n\n\ndef retrieve(state):\n    \"\"\"\n    Retrieve documents\n\n    Args:\n        state (dict): The current graph state\n\n    Returns:\n        state (dict): New key added to state, documents, that contains retrieved documents\n    \"\"\"\n    print(\"---RETRIEVE---\")\n    question = state[\"question\"]\n\n    # Retrieval\n    documents = retriever.get_relevant_documents(question)\n    return {\"documents\": documents, \"question\": question}\n\n\ndef generate(state):\n    \"\"\"\n    Generate answer\n\n    Args:\n        state (dict): The current graph state\n\n    Returns:\n        state (dict): New key added to state, generation, that contains LLM generation\n    \"\"\"\n    print(\"---GENERATE---\")\n    question = state[\"question\"]\n    documents = state[\"documents\"]\n\n    # RAG generation\n    generation = rag_chain.invoke({\"context\": documents, \"question\": question})\n    return {\"documents\": documents, \"question\": question, \"generation\": generation}\n\n\ndef grade_documents(state):\n    \"\"\"\n    Determines whether the retrieved documents are relevant to the question.\n\n    Args:\n        state (dict): The current graph state\n\n    Returns:\n        state (dict): Updates documents key with only filtered relevant documents\n    \"\"\"\n\n    print(\"---CHECK DOCUMENT RELEVANCE TO QUESTION---\")\n    question = state[\"question\"]\n    documents = state[\"documents\"]\n\n    # Score each doc\n    filtered_docs = []\n    for d in documents:\n        score = retrieval_grader.invoke(\n            {\"question\": question, \"document\": d.page_content}\n        )\n        grade = score.binary_score\n        if grade == \"yes\":\n            print(\"---GRADE: DOCUMENT RELEVANT---\")\n            filtered_docs.append(d)\n        else:\n            print(\"---GRADE: DOCUMENT NOT RELEVANT---\")\n            continue\n    return {\"documents\": filtered_docs, \"question\": question}\n\n\ndef transform_query(state):\n    \"\"\"\n    Transform the query to produce a better question.\n\n    Args:\n        state (dict): The current graph state\n\n    Returns:\n        state (dict): Updates question key with a re-phrased question\n    \"\"\"\n\n    print(\"---TRANSFORM QUERY---\")\n    question = state[\"question\"]\n    documents = state[\"documents\"]\n\n    # Re-write question\n    better_question = question_rewriter.invoke({\"question\": question})\n    return {\"documents\": documents, \"question\": better_question}\n\n\n### Edges\n\n\ndef decide_to_generate(state):\n    \"\"\"\n    Determines whether to generate an answer, or re-generate a question.\n\n    Args:\n        state (dict): The current graph state\n\n    Returns:\n        str: Binary decision for next node to call\n    \"\"\"\n\n    print(\"---ASSESS GRADED DOCUMENTS---\")\n    state[\"question\"]\n    filtered_documents = state[\"documents\"]\n\n    if not filtered_documents:\n        # All documents have been filtered check_relevance\n        # We will re-generate a new query\n        print(\n            \"---DECISION: ALL DOCUMENTS ARE NOT RELEVANT TO QUESTION, TRANSFORM QUERY---\"\n        )\n        return \"transform_query\"\n    else:\n        # We have relevant documents, so generate answer\n        print(\"---DECISION: GENERATE---\")\n        return \"generate\"\n\n\ndef grade_generation_v_documents_and_question(state):\n    \"\"\"\n    Determines whether the generation is grounded in the document and answers question.\n\n    Args:\n        state (dict): The current graph state\n\n    Returns:\n        str: Decision for next node to call\n    \"\"\"\n\n    print(\"---CHECK HALLUCINATIONS---\")\n    question = state[\"question\"]\n    documents = state[\"documents\"]\n    generation = state[\"generation\"]\n\n    score = hallucination_grader.invoke(\n        {\"documents\": documents, \"generation\": generation}\n    )\n    grade = score.binary_score\n\n    # Check hallucination\n    if grade == \"yes\":\n        print(\"---DECISION: GENERATION IS GROUNDED IN DOCUMENTS---\")\n        # Check question-answering\n        print(\"---GRADE GENERATION vs QUESTION---\")\n        score = answer_grader.invoke({\"question\": question, \"generation\": generation})\n        grade = score.binary_score\n        if grade == \"yes\":\n            print(\"---DECISION: GENERATION ADDRESSES QUESTION---\")\n            return \"useful\"\n        else:\n            print(\"---DECISION: GENERATION DOES NOT ADDRESS QUESTION---\")\n            return \"not useful\"\n    else:\n        pprint(\"---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---\")\n        return \"not supported\""]
  },
  {
   "cell_type": "markdown",
   "id": "61cd5797-1782-4d78-a277-8196d13f3e1b",
   "metadata": {},
   "source": [
    "## Build Graph\n",
    "\n",
    "The just follows the flow we outlined in the figure above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0e09ca9f-e36d-4ef4-a0d5-79fdbada9fe0",
   "metadata": {},
   "outputs": [],
   "source": ["from langgraph.graph import END, StateGraph, START\n\nworkflow = StateGraph(GraphState)\n\n# Define the nodes\nworkflow.add_node(\"retrieve\", retrieve)  # retrieve\nworkflow.add_node(\"grade_documents\", grade_documents)  # grade documents\nworkflow.add_node(\"generate\", generate)  # generatae\nworkflow.add_node(\"transform_query\", transform_query)  # transform_query\n\n# Build graph\nworkflow.add_edge(START, \"retrieve\")\nworkflow.add_edge(\"retrieve\", \"grade_documents\")\nworkflow.add_conditional_edges(\n    \"grade_documents\",\n    decide_to_generate,\n    {\n        \"transform_query\": \"transform_query\",\n        \"generate\": \"generate\",\n    },\n)\nworkflow.add_edge(\"transform_query\", \"retrieve\")\nworkflow.add_conditional_edges(\n    \"generate\",\n    grade_generation_v_documents_and_question,\n    {\n        \"not supported\": \"generate\",\n        \"useful\": END,\n        \"not useful\": \"transform_query\",\n    },\n)\n\n# Compile\napp = workflow.compile()"]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fb69dbb9-91ee-4868-8c3c-93af3cd885be",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---RETRIEVE---\n",
      "\"Node 'retrieve':\"\n",
      "'\\n---\\n'\n",
      "---CHECK DOCUMENT RELEVANCE TO QUESTION---\n",
      "---GRADE: DOCUMENT NOT RELEVANT---\n",
      "---GRADE: DOCUMENT RELEVANT---\n",
      "---GRADE: DOCUMENT NOT RELEVANT---\n",
      "---GRADE: DOCUMENT RELEVANT---\n",
      "---ASSESS GRADED DOCUMENTS---\n",
      "---DECISION: GENERATE---\n",
      "\"Node 'grade_documents':\"\n",
      "'\\n---\\n'\n",
      "---GENERATE---\n",
      "---CHECK HALLUCINATIONS---\n",
      "---DECISION: GENERATION IS GROUNDED IN DOCUMENTS---\n",
      "---GRADE GENERATION vs QUESTION---\n",
      "---DECISION: GENERATION ADDRESSES QUESTION---\n",
      "\"Node 'generate':\"\n",
      "'\\n---\\n'\n",
      "('Short-term memory is used for in-context learning in agents, allowing them '\n",
      " 'to learn quickly. Long-term memory enables agents to retain and recall vast '\n",
      " 'amounts of information over extended periods. Agents can also utilize '\n",
      " 'external tools like APIs to access additional information beyond what is '\n",
      " 'stored in their memory.')\n"
     ]
    }
   ],
   "source": ["from pprint import pprint\n\n# Run\ninputs = {\"question\": \"Explain how the different types of agent memory work?\"}\nfor output in app.stream(inputs):\n    for key, value in output.items():\n        # Node\n        pprint(f\"Node '{key}':\")\n        # Optional: print full state at each node\n        # pprint.pprint(value[\"keys\"], indent=2, width=80, depth=None)\n    pprint(\"\\n---\\n\")\n\n# Final generation\npprint(value[\"generation\"])"]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4138bc51-8c84-4b8a-8d24-f7f470721f6f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---RETRIEVE---\n",
      "\"Node 'retrieve':\"\n",
      "'\\n---\\n'\n",
      "---CHECK DOCUMENT RELEVANCE TO QUESTION---\n",
      "---GRADE: DOCUMENT RELEVANT---\n",
      "---GRADE: DOCUMENT NOT RELEVANT---\n",
      "---GRADE: DOCUMENT RELEVANT---\n",
      "---GRADE: DOCUMENT RELEVANT---\n",
      "---ASSESS GRADED DOCUMENTS---\n",
      "---DECISION: GENERATE---\n",
      "\"Node 'grade_documents':\"\n",
      "'\\n---\\n'\n",
      "---GENERATE---\n",
      "---CHECK HALLUCINATIONS---\n",
      "---DECISION: GENERATION IS GROUNDED IN DOCUMENTS---\n",
      "---GRADE GENERATION vs QUESTION---\n",
      "---DECISION: GENERATION ADDRESSES QUESTION---\n",
      "\"Node 'generate':\"\n",
      "'\\n---\\n'\n",
      "('Chain of thought prompting works by repeatedly prompting the model to ask '\n",
      " 'follow-up questions to construct the thought process iteratively. This '\n",
      " 'method can be combined with queries to search for relevant entities and '\n",
      " 'content to add back into the context. It extends the thought process by '\n",
      " 'exploring multiple reasoning possibilities at each step, creating a tree '\n",
      " 'structure of thoughts.')\n"
     ]
    }
   ],
   "source": ["inputs = {\"question\": \"Explain how chain of thought prompting works?\"}\nfor output in app.stream(inputs):\n    for key, value in output.items():\n        # Node\n        pprint(f\"Node '{key}':\")\n        # Optional: print full state at each node\n        # pprint.pprint(value[\"keys\"], indent=2, width=80, depth=None)\n    pprint(\"\\n---\\n\")\n\n# Final generation\npprint(value[\"generation\"])"]
  },
  {
   "cell_type": "markdown",
   "id": "548f1c5b-4108-4aae-8abb-ec171b511b92",
   "metadata": {},
   "source": [
    "LangSmith Traces - \n",
    " \n",
    "* https://smith.langchain.com/public/55d6180f-aab8-42bc-8799-dadce6247d9b/r\n",
    "\n",
    "* https://smith.langchain.com/public/1c6bf654-61b2-4fc5-9889-054b020c78aa/r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42369ab8-322d-434a-b5dd-2266e4cb2903",
   "metadata": {},
   "outputs": [],
   "source": [""]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
