diff --git a/examples/storm/storm.ipynb b/examples/storm/storm.ipynb index b4191e6bd..8c91b47d2 100644 --- a/examples/storm/storm.ipynb +++ b/examples/storm/storm.ipynb @@ -10,10 +10,12 @@ "\n", "It is tasked with generating Wikipedia-like ariticles on a user-provided topic. It has a few main stages:\n", "\n", - "1. Survey related subjects\n", - "2. Identify perspectives\n", - "3. \"Expert Interviews\" (between the writer and an agent role-playing as a perspective)\n", - "4. Refine article \n", + "1. Generate initial outline + Survey related subjects\n", + "2. Identify distinct perspectives\n", + "3. \"Interview subject matter experts\" (role-playing LLMs)\n", + "4. Refine outline\n", + "5. Write article\n", + "\n", "\n", "The expert interviews stage ocurrs between the article writer and each role-playing agent and itself is a loop, where the \"expert\" is able to query external knowledge and respond to pointed questions.\n", "\n", @@ -27,11 +29,20 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 68, "metadata": {}, "outputs": [], "source": [ - "# %pip install langchain_community langchain_openai langgraph wikipedia tavily-python scikit-learn" + "# %pip install langchain_community langchain_openai langchain_fireworks langgraph wikipedia tavily-python scikit-learn" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Select LLMs\n", + "\n", + "We will have a faster LLM do most of the work, but a slower, long-context model distill the conversations and write the final report." ] }, { @@ -41,11 +52,22 @@ "outputs": [], "source": [ "from langchain_openai import ChatOpenAI\n", + "from langchain_fireworks\n", "\n", "fast_llm = ChatOpenAI(model=\"gpt-3.5-turbo\")\n", "long_context_llm = ChatOpenAI(model=\"gpt-4-turbo-preview\")" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Generate Initial Outline\n", + "\n", + "For many topics, your LLM may have an initial idea of the important and related topics. We can generate an initial\n", + "outline to be refined after our research. Below, we will use our \"fast\" llm to generate the outline." + ] + }, { "cell_type": "code", "execution_count": 2, @@ -169,7 +191,7 @@ "\n", "While language models do store some Wikipedia-like knowledge in their parameters, you will get better results by incorporating relevant and recent information using a search engine.\n", "\n", - "We will start our search by generating a list of related topics." + "We will start our search by generating a list of related topics, sourced from Wikipedia." ] }, { @@ -220,6 +242,16 @@ "related_subjects" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Generate Perspectives\n", + "\n", + "From these related subjects, we can select representative Wikipedia editors as \"subject matter experts\" with distinct\n", + "backgrounds and affiliations. These will help distribute the search process to encourage a more well-rounded final report." + ] + }, { "cell_type": "code", "execution_count": 6, @@ -371,7 +403,12 @@ "source": [ "## Expert Dialog\n", "\n", - "Now the true fun begins, the wikipedia writer will \"talk\" with expert agents primed to role-play using the perspectives presented above." + "Now the true fun begins, each wikipedia writer is primed to role-play using the perspectives presented above. It will ask a series of questions of a second \"domain expert\" with access to a search engine. This generate content to generate a refined outline as well as an updated index of reference documents.\n", + "\n", + "\n", + "### Interview State\n", + "\n", + "The conversation is cyclic, so we will construct it within its own graph. The State will contain messages, the reference docs, and the editor (with its own \"persona\") to make it easy to parallelize these conversations." ] }, { @@ -414,6 +451,15 @@ " editor: Annotated[Optional[Editor], update_editor]" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Dialog Roles\n", + "\n", + "The graph will have two participants: the wikipedia editor (`generate_question`), who asks questions based on its assigned role, and a domain expert (`gen_answer_chain), who uses a search engine to answer the questions as accurately as possible." + ] + }, { "cell_type": "code", "execution_count": 11, @@ -503,6 +549,15 @@ "question[\"messages\"][0].content" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Answer questions\n", + "\n", + "The `gen_answer_chain` first generates queries (query expansion) to answer the editor's question, then responds with citations." + ] + }, { "cell_type": "code", "execution_count": 13, @@ -694,6 +749,16 @@ "example_answer[\"messages\"][-1].content" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Construct the Interview Graph\n", + "\n", + "\n", + "Now that we've defined the editor and domain expert, we can compose them in a graph." + ] + }, { "cell_type": "code", "execution_count": 33, @@ -792,7 +857,7 @@ "source": [ "## Refine Outline\n", "\n", - "Now that we have all this cool stuff, let's distill it into a refined outline." + "At this point in STORM, we've conducted a large amount of research from different perspectives. It's time to refine the original outline based on these investigations. Below, create a chain using the LLM with a long context window to update the original outline." ] }, { @@ -911,12 +976,76 @@ "source": [ "## Generate Article\n", "\n", - "Now it's time to generate the full article. We will divide-and-conquer, so that each section can be tackled by an individual llm." + "Now it's time to generate the full article. We will first divide-and-conquer, so that each section can be tackled by an individual llm. Then we will prompt the long-form LLM to refine the finished article (since each section may use an inconsistent voice).\n", + "\n", + "#### Create Retriever\n", + "\n", + "When completing a given section of the outline, the LLM can retrieve reference documents unearthed during the conversations above.\n", + "\n", + "First, create the retriever:" ] }, { "cell_type": "code", - "execution_count": 39, + "execution_count": 69, + "metadata": {}, + "outputs": [], + "source": [ + "from langchain_core.documents import Document\n", + "\n", + "from langchain_community.vectorstores import SKLearnVectorStore\n", + "from langchain_openai import OpenAIEmbeddings\n", + "\n", + "embeddings = OpenAIEmbeddings(model=\"text-embedding-3-small\")\n", + "reference_docs = [\n", + " Document(page_content=v, metadata={\"source\": k})\n", + " for k, v in final_state[\"references\"].items()\n", + "]\n", + "# This really doesn't need to be a vectorstore for this size of data.\n", + "# It could just be a numpy matrix. Or you could store documents\n", + "# across requests if you want.\n", + "vectorstore = SKLearnVectorStore.from_documents(\n", + " reference_docs,\n", + " embedding=embeddings,\n", + ")\n", + "retriever = vectorstore.as_retriever(k=10)" + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[Document(page_content='Large Language Models (LLMs) have achieved remarkable success across various tasks. However, they often grapple with a limited context window size due to the high costs of fine-tuning, scarcity of lengthy texts, and the introduction of catastrophic values by new token positions. To address this issue, in a new paper LongRoPE: Extending LLM Context Window', metadata={'id': '20dbbce3-ae12-4a05-94e9-df4a97676098', 'source': 'https://syncedreview.com/2024/02/25/microsofts-longrope-breaks-the-limit-of-context-window-of-llms-extents-it-to-2-million-tokens/'}),\n", + " Document(page_content='Large context window is a desirable feature in large language models (LLMs). However, due to high fine-tuning costs, scarcity of long texts, and catastrophic values introduced by new token positions, current extended context windows are limited to around 128k tokens. This paper introduces LongRoPE that, for the first time, extends the context window of pre-trained LLMs to an impressive 2048k ...', metadata={'id': 'f7881b24-697b-46fa-a31c-8e476ae40f3f', 'source': 'https://arxiv.org/abs/2402.13753'}),\n", + " Document(page_content='Large language models (LLMs) have witnessed significant advancements, aiming to enhance their capabilities for interpreting and processing extensive textual data. LLMs like GPT-3 have revolutionized our interactions with AI, offering insights and analyses across various domains, from writing assistance to complex data interpretation. However, a key limitation has been their context window size ...', metadata={'id': '354ceab3-03ae-4141-94a1-4b0109182aab', 'source': 'https://www.marktechpost.com/2024/02/23/breaking-barriers-in-language-understanding-how-microsoft-ais-longrope-extends-large-language-models-to-a-2048k-token-context-window/'}),\n", + " Document(page_content='Large Language Models (LLMs) demonstrate significant capabilities but face challenges such as hallucination, outdated knowledge, and non-transparent, untraceable reasoning processes. Retrieval-Augmented Generation (RAG) has emerged as a promising solution by incorporating knowledge from external databases. This enhances the accuracy and credibility of the models, particularly for knowledge ...', metadata={'id': '85f7d7a1-2820-466c-ac80-975f4e2d65af', 'source': 'https://arxiv.org/abs/2312.10997'})]" + ] + }, + "execution_count": 41, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "retriever.invoke(\"What's a long context LLM anyway?\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Generate Sections\n", + "\n", + "Now you can generate the sections using the indexed docs." + ] + }, + { + "cell_type": "code", + "execution_count": 50, "metadata": {}, "outputs": [], "source": [ @@ -950,63 +1079,9 @@ " return (\n", " f\"## {self.section_title}\\n\\n{self.content}\\n\\n{subsections}\".strip()\n", " + f\"\\n\\n{citations}\".strip()\n", - " )" - ] - }, - { - "cell_type": "code", - "execution_count": 40, - "metadata": {}, - "outputs": [], - "source": [ - "from langchain_core.documents import Document\n", + " )\n", "\n", - "from langchain_community.vectorstores import SKLearnVectorStore\n", - "from langchain_openai import OpenAIEmbeddings\n", "\n", - "embeddings = OpenAIEmbeddings(model=\"text-embedding-3-small\")\n", - "reference_docs = [\n", - " Document(page_content=v, metadata={\"source\": k})\n", - " for k, v in final_state[\"references\"].items()\n", - "]\n", - "# This really doesn't need to be a vectorstore.\n", - "# could just be a numpy matrix\n", - "vectorstore = SKLearnVectorStore.from_documents(\n", - " reference_docs,\n", - " embedding=embeddings,\n", - ")\n", - "retriever = vectorstore.as_retriever(k=10)" - ] - }, - { - "cell_type": "code", - "execution_count": 41, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "[Document(page_content='Large Language Models (LLMs) have achieved remarkable success across various tasks. However, they often grapple with a limited context window size due to the high costs of fine-tuning, scarcity of lengthy texts, and the introduction of catastrophic values by new token positions. To address this issue, in a new paper LongRoPE: Extending LLM Context Window', metadata={'id': '20dbbce3-ae12-4a05-94e9-df4a97676098', 'source': 'https://syncedreview.com/2024/02/25/microsofts-longrope-breaks-the-limit-of-context-window-of-llms-extents-it-to-2-million-tokens/'}),\n", - " Document(page_content='Large context window is a desirable feature in large language models (LLMs). However, due to high fine-tuning costs, scarcity of long texts, and catastrophic values introduced by new token positions, current extended context windows are limited to around 128k tokens. This paper introduces LongRoPE that, for the first time, extends the context window of pre-trained LLMs to an impressive 2048k ...', metadata={'id': 'f7881b24-697b-46fa-a31c-8e476ae40f3f', 'source': 'https://arxiv.org/abs/2402.13753'}),\n", - " Document(page_content='Large language models (LLMs) have witnessed significant advancements, aiming to enhance their capabilities for interpreting and processing extensive textual data. LLMs like GPT-3 have revolutionized our interactions with AI, offering insights and analyses across various domains, from writing assistance to complex data interpretation. However, a key limitation has been their context window size ...', metadata={'id': '354ceab3-03ae-4141-94a1-4b0109182aab', 'source': 'https://www.marktechpost.com/2024/02/23/breaking-barriers-in-language-understanding-how-microsoft-ais-longrope-extends-large-language-models-to-a-2048k-token-context-window/'}),\n", - " Document(page_content='Large Language Models (LLMs) demonstrate significant capabilities but face challenges such as hallucination, outdated knowledge, and non-transparent, untraceable reasoning processes. Retrieval-Augmented Generation (RAG) has emerged as a promising solution by incorporating knowledge from external databases. This enhances the accuracy and credibility of the models, particularly for knowledge ...', metadata={'id': '85f7d7a1-2820-466c-ac80-975f4e2d65af', 'source': 'https://arxiv.org/abs/2312.10997'})]" - ] - }, - "execution_count": 41, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "retriever.invoke(\"What's a long context LLM anyway?\")" - ] - }, - { - "cell_type": "code", - "execution_count": 50, - "metadata": {}, - "outputs": [], - "source": [ "section_writer_prompt = ChatPromptTemplate.from_messages(\n", " [\n", " (\n", @@ -1082,48 +1157,13 @@ "print(section.as_str)" ] }, - { - "cell_type": "code", - "execution_count": 53, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "## The Evolution of Large Context Windows in Language Models\n", - "\n", - "The evolution of large context windows in language models (LLMs) has been a critical factor in the advancement of natural language processing (NLP) technologies. Initially, LLMs were constrained by smaller context windows, limiting their understanding and generation capabilities. However, the demand for models capable of processing and integrating more extensive sequences of text has led to significant research and development efforts aimed at expanding these context windows.\n", - "\n", - "Over time, this push for larger context windows has seen the emergence of several key milestones that have progressively increased the amount of text LLMs can consider when generating responses or analyses. These milestones include models like Gemini 1.5, Mixtral, GPT-3.5-Turbo-16k, Llama2-7B-chat-4k, and LongRoPE, each contributing to the landscape of large context window LLMs in unique ways.\n", - "\n", - "Despite the benefits, expanding the context window size brings several challenges, including the high fine-tuning costs associated with processing longer sequences of text, the scarcity of long texts suitable for training these models, and the potential for catastrophic values introduced by new token positions in expanded contexts. Addressing these challenges has been central to the continued development and effectiveness of LLMs with large context windows.\n", - "\n", - "### Key Milestones\n", - "\n", - "The journey towards expanding the context window sizes of language models has been marked by several significant milestones. Gemini 1.5, Mixtral, GPT-3.5-Turbo-16k, Llama2-7B-chat-4k, and LongRoPE represent some of the most notable advancements in this area. Each of these models has pushed the boundaries of what was previously possible, setting new standards for the amount of text that can be processed and understood by LLMs.\n", - "\n", - "For instance, LongRoPE has made a groundbreaking contribution by extending the context window of pre-trained LLMs to an impressive 2048k tokens, far surpassing previous limits and opening up new possibilities for NLP applications.\n", - "\n", - "### Challenges Overcome\n", - "\n", - "Expanding the context window sizes of language models has not been without its challenges. High fine-tuning costs, the scarcity of suitable long texts for training, and the introduction of catastrophic values by new token positions have all posed significant hurdles.\n", - "\n", - "Technological and methodological advancements have been crucial in overcoming these challenges, allowing for the successful expansion of context windows beyond previous limitations. Innovations in model architecture, training methodologies, and data processing techniques have all played a role in addressing these issues and enabling the development of more capable and efficient LLMs.[0] https://arxiv.org/abs/2402.13753\n", - " [1] https://syncedreview.com/2024/02/25/microsofts-longrope-breaks-the-limit-of-context-window-of-llms-extents-it-to-2-million-tokens/\n", - " [2] https://www.marktechpost.com/2024/02/23/breaking-barriers-in-language-understanding-how-microsoft-ais-longrope-extends-large-language-models-to-a-2048k-token-context-window/\n" - ] - } - ], - "source": [ - "print(section.as_str)" - ] - }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### Generate final article" + "#### Generate final article\n", + "\n", + "Now we can rewrite the draft to appropriately group all the citations and maintain a consistent voice." ] }, { @@ -1229,12 +1269,14 @@ "source": [ "## Final Flow\n", "\n", - "Now it's time to string everything together. We will have 3 main stages in sequence:\n", + "Now it's time to string everything together. We will have 6 main stages in sequence:\n", ".\n", "1. Generate the initial outline + perspectives\n", - "2. Batch converse with each perspective to expand the content for the article.\n", + "2. Batch converse with each perspective to expand the content for the article\n", "3. Refine the outline based on the conversations\n", - "4. Write the final wiki." + "4. Index the reference docs from the conversations\n", + "5. Write the individual sections of the article\n", + "6. Write the final wiki" ] }, {