{ "cells": [ { "cell_type": "markdown", "id": "9fd54a32", "metadata": {}, "source": [ "\"Open" ] }, { "cell_type": "markdown", "id": "9e3a8796-edc8-43f2-94ad-fe4fb20d70ed", "metadata": {}, "source": [ "# OpenAI\n", "\n", "This notebook shows how to use the OpenAI LLM.\n", "\n", "If you are looking to integrate with an OpenAI-Compatible API that is not the official OpenAI API, please see the [OpenAI-Compatible LLMs](https://docs.llamaindex.ai/en/stable/api_reference/llms/openai_like/#llama_index.llms.openai_like.OpenAILike) integration." ] }, { "cell_type": "markdown", "id": "081d07d2", "metadata": {}, "source": [ "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙." ] }, { "cell_type": "code", "execution_count": null, "id": "3f6f8702", "metadata": {}, "outputs": [], "source": [ "%pip install llama-index llama-index-llms-openai" ] }, { "cell_type": "markdown", "id": "b007403c-6b7a-420c-92f1-4171d05ed9bb", "metadata": {}, "source": [ "## Basic Usage" ] }, { "cell_type": "code", "execution_count": null, "id": "e0a6c491", "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "os.environ[\"OPENAI_API_KEY\"] = \"sk-...\"" ] }, { "cell_type": "code", "execution_count": null, "id": "0b79b0d9", "metadata": {}, "outputs": [], "source": [ "from llama_index.llms.openai import OpenAI\n", "\n", "llm = OpenAI(\n", " model=\"gpt-4o-mini\",\n", " # api_key=\"some key\", # uses OPENAI_API_KEY env var by default\n", ")" ] }, { "cell_type": "markdown", "id": "8ead155e-b8bd-46f9-ab9b-28fc009361dd", "metadata": {}, "source": [ "#### Call `complete` with a prompt" ] }, { "cell_type": "code", "execution_count": null, "id": "60be18ae-c957-4ac2-a58a-0652e18ee6d6", "metadata": {}, "outputs": [], "source": [ "from llama_index.llms.openai import OpenAI\n", "\n", "resp = llm.complete(\"Paul Graham is \")" ] }, { "cell_type": "code", "execution_count": null, "id": "ac2cbebb-a444-4a46-9d85-b265a3483d68", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "a computer scientist, entrepreneur, and venture capitalist. He is best known for co-founding the startup accelerator Y Combinator and for his work on Lisp, a programming language. Graham has also written several influential essays on startups, technology, and entrepreneurship.\n" ] } ], "source": [ "print(resp)" ] }, { "cell_type": "markdown", "id": "14831268-f90f-499d-9d86-925dbc88292b", "metadata": {}, "source": [ "#### Call `chat` with a list of messages" ] }, { "cell_type": "code", "execution_count": null, "id": "bbe29574-4af1-48d5-9739-f60652b6ce6c", "metadata": {}, "outputs": [], "source": [ "from llama_index.core.llms import ChatMessage\n", "\n", "messages = [\n", " ChatMessage(\n", " role=\"system\", content=\"You are a pirate with a colorful personality\"\n", " ),\n", " ChatMessage(role=\"user\", content=\"What is your name\"),\n", "]\n", "resp = llm.chat(messages)" ] }, { "cell_type": "code", "execution_count": null, "id": "9cbd550a-0264-4a11-9b2c-a08d8723a5ae", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "assistant: Ahoy matey! The name's Rainbow Roger, the most colorful pirate on the seven seas! What can I do for ye today?\n" ] } ], "source": [ "print(resp)" ] }, { "cell_type": "markdown", "id": "2ed5e894-4597-4911-a623-591560f72b82", "metadata": {}, "source": [ "## Streaming" ] }, { "cell_type": "markdown", "id": "7311566b", "metadata": {}, "source": [ "Using `stream_complete` endpoint" ] }, { "cell_type": "code", "execution_count": null, "id": "0c896dbd", "metadata": {}, "outputs": [], "source": [ "resp = llm.stream_complete(\"Paul Graham is \")" ] }, { "cell_type": "code", "execution_count": null, "id": "a79deaf3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "a computer scientist, entrepreneur, and venture capitalist. He is best known for co-founding the startup accelerator Y Combinator and for his work on programming languages and web development. Graham is also a prolific writer and has published several influential essays on technology, startups, and entrepreneurship." ] } ], "source": [ "for r in resp:\n", " print(r.delta, end=\"\")" ] }, { "cell_type": "markdown", "id": "15ae9c4a", "metadata": {}, "source": [ "Using `stream_chat` endpoint" ] }, { "cell_type": "code", "execution_count": null, "id": "173d8412", "metadata": {}, "outputs": [], "source": [ "from llama_index.core.llms import ChatMessage\n", "\n", "messages = [\n", " ChatMessage(\n", " role=\"system\", content=\"You are a pirate with a colorful personality\"\n", " ),\n", " ChatMessage(role=\"user\", content=\"What is your name\"),\n", "]\n", "resp = llm.stream_chat(messages)" ] }, { "cell_type": "code", "execution_count": null, "id": "9cc4cb7d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Ahoy matey! The name's Captain Rainbowbeard! Aye, I be a pirate with a love for all things colorful and bright. Me beard be as vibrant as a rainbow, and me ship be the most colorful vessel on the seven seas! What can I do for ye today, me hearty?" ] } ], "source": [ "for r in resp:\n", " print(r.delta, end=\"\")" ] }, { "cell_type": "markdown", "id": "87356c8d", "metadata": {}, "source": [ "## Configure Model" ] }, { "cell_type": "code", "execution_count": null, "id": "53e9ae5b", "metadata": {}, "outputs": [], "source": [ "from llama_index.llms.openai import OpenAI\n", "\n", "llm = OpenAI(model=\"gpt-4o\")" ] }, { "cell_type": "code", "execution_count": null, "id": "ff7a3aaa", "metadata": {}, "outputs": [], "source": [ "resp = llm.complete(\"Paul Graham is \")" ] }, { "cell_type": "code", "execution_count": null, "id": "6b73f350", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "a computer scientist, entrepreneur, and venture capitalist. He is best known for co-founding the startup accelerator Y Combinator and for his work on Lisp, a programming language. Graham has also written several influential essays on startups, technology, and entrepreneurship.\n" ] } ], "source": [ "print(resp)" ] }, { "cell_type": "code", "execution_count": null, "id": "4fc4c7c6", "metadata": {}, "outputs": [], "source": [ "messages = [\n", " ChatMessage(\n", " role=\"system\", content=\"You are a pirate with a colorful personality\"\n", " ),\n", " ChatMessage(role=\"user\", content=\"What is your name\"),\n", "]\n", "resp = llm.chat(messages)" ] }, { "cell_type": "code", "execution_count": null, "id": "0efdf231", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "assistant: Ahoy matey! The name's Captain Rainbowbeard, the most colorful pirate on the seven seas! What can I do for ye today? Arrr!\n" ] } ], "source": [ "print(resp)" ] }, { "cell_type": "markdown", "id": "5170f908", "metadata": {}, "source": [ "## Image Support\n", "\n", "OpenAI has support for images in the input of chat messages for many models.\n", "\n", "Using the content blocks feature of chat messages, you can easily combone text and images in a single LLM prompt." ] }, { "cell_type": "code", "execution_count": null, "id": "d9ac4a61", "metadata": {}, "outputs": [], "source": [ "!wget https://cdn.pixabay.com/photo/2016/07/07/16/46/dice-1502706_640.jpg -O image.png" ] }, { "cell_type": "code", "execution_count": null, "id": "4da96618", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The image is a black and white photograph featuring three dice in mid-air above a checkered surface, likely a chessboard. The dice are captured in motion, with one showing the number five prominently. The background is dark, with a subtle light beam creating a triangular shape above the dice, adding a dramatic effect to the composition.\n" ] } ], "source": [ "from llama_index.core.llms import ChatMessage, TextBlock, ImageBlock\n", "from llama_index.llms.openai import OpenAI\n", "\n", "llm = OpenAI(model=\"gpt-4o\")\n", "\n", "messages = [\n", " ChatMessage(\n", " role=\"user\",\n", " blocks=[\n", " ImageBlock(path=\"image.png\"),\n", " TextBlock(text=\"Describe the image in a few sentences.\"),\n", " ],\n", " )\n", "]\n", "\n", "resp = llm.chat(messages)\n", "print(resp.message.content)" ] }, { "cell_type": "markdown", "id": "3e83c40c", "metadata": {}, "source": [ "## Audio Support\n", "\n", "OpenAI has beta support for audio inputs and outputs, using their audio-preview models.\n", "\n", "When using these models, you can configure the output modality (text or audio) using the `modalities` parameter. The output audio configuration can also be set using the `audio_config` parameter. See the [OpenAI docs](https://platform.openai.com/docs/guides/audio) for more information." ] }, { "cell_type": "code", "execution_count": null, "id": "989fc819", "metadata": {}, "outputs": [], "source": [ "from llama_index.core.llms import ChatMessage\n", "from llama_index.llms.openai import OpenAI\n", "\n", "\n", "llm = OpenAI(\n", " model=\"gpt-4o-audio-preview\",\n", " modalities=[\"text\", \"audio\"],\n", " audio_config={\"voice\": \"alloy\", \"format\": \"wav\"},\n", ")\n", "\n", "messages = [\n", " ChatMessage(role=\"user\", content=\"Hello! My name is Logan.\"),\n", "]\n", "\n", "resp = llm.chat(messages)" ] }, { "cell_type": "code", "execution_count": null, "id": "477587a7", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import base64\n", "from IPython.display import Audio\n", "\n", "Audio(base64.b64decode(resp.message.blocks[0].audio), rate=16000)" ] }, { "cell_type": "code", "execution_count": null, "id": "90b20a7d", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Add the response to the chat history and ask for the user's name\n", "messages.append(resp.message)\n", "messages.append(ChatMessage(role=\"user\", content=\"What is my name?\"))\n", "\n", "resp = llm.chat(messages)\n", "Audio(base64.b64decode(resp.message.blocks[0].audio), rate=16000)" ] }, { "cell_type": "markdown", "id": "a5a769a8", "metadata": {}, "source": [ "We can also use audio as input and get descriptions or transcriptions of the audio." ] }, { "cell_type": "code", "execution_count": null, "id": "6b6527df", "metadata": {}, "outputs": [], "source": [ "!wget AUDIO_URL = \"https://science.nasa.gov/wp-content/uploads/2024/04/sounds-of-mars-one-small-step-earth.wav\" -O audio.wav" ] }, { "cell_type": "code", "execution_count": null, "id": "45a6d3c2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "assistant: The audio is a famous quote from astronaut Neil Armstrong during the Apollo 11 moon landing in 1969. As he became the first human to step onto the moon's surface, he said, \"That's one small step for man, one giant leap for mankind.\" This moment marked a significant achievement in space exploration and human history.\n" ] } ], "source": [ "from llama_index.core.llms import ChatMessage, AudioBlock, TextBlock\n", "\n", "messages = [\n", " ChatMessage(\n", " role=\"user\",\n", " blocks=[\n", " AudioBlock(path=\"audio.wav\", format=\"wav\"),\n", " TextBlock(\n", " text=\"Describe the audio in a few sentences. What is it from?\"\n", " ),\n", " ],\n", " )\n", "]\n", "\n", "llm = OpenAI(\n", " model=\"gpt-4o-audio-preview\",\n", " modalities=[\"text\"],\n", ")\n", "\n", "resp = llm.chat(messages)\n", "print(resp)" ] }, { "cell_type": "markdown", "id": "90f07f7e-927f-47a2-9797-de5a86d61e1f", "metadata": {}, "source": [ "## Using Function/Tool Calling\n", "\n", "OpenAI models have native support for function calling. This conveniently integrates with LlamaIndex tool abstractions, letting you plug in any arbitrary Python function to the LLM.\n", "\n", "In the example below, we define a function to generate a Song object." ] }, { "cell_type": "code", "execution_count": null, "id": "212bb2d2-2bed-4188-85ad-3cd497d4b864", "metadata": {}, "outputs": [], "source": [ "from pydantic import BaseModel\n", "from llama_index.core.tools import FunctionTool\n", "\n", "\n", "class Song(BaseModel):\n", " \"\"\"A song with name and artist\"\"\"\n", "\n", " name: str\n", " artist: str\n", "\n", "\n", "def generate_song(name: str, artist: str) -> Song:\n", " \"\"\"Generates a song with provided name and artist.\"\"\"\n", " return Song(name=name, artist=artist)\n", "\n", "\n", "tool = FunctionTool.from_defaults(fn=generate_song)" ] }, { "cell_type": "markdown", "id": "daec99fe", "metadata": {}, "source": [ "The `strict` parameter tells OpenAI whether or not to use constrained sampling when generating tool calls/structured outputs. This means that the generated tool call schema will always contain the expected fields.\n", "\n", "Since this seems to increase latency, it defaults to false." ] }, { "cell_type": "code", "execution_count": null, "id": "ba29ea39-39cd-42f4-934e-78fba4935a1c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "name='Random Vibes' artist='DJ Chill'\n" ] } ], "source": [ "from llama_index.llms.openai import OpenAI\n", "\n", "llm = OpenAI(model=\"gpt-4o-mini\", strict=True)\n", "response = llm.predict_and_call(\n", " [tool],\n", " \"Pick a random song for me\",\n", " # strict=True # can also be set at the function level to override the class\n", ")\n", "print(str(response))" ] }, { "cell_type": "markdown", "id": "629bdb64-e5bd-4539-9537-54352dd8dbb4", "metadata": {}, "source": [ "We can also do multiple function calling." ] }, { "cell_type": "code", "execution_count": null, "id": "c098cb38-bfed-4907-8109-6810756f2ae5", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Name: generate_song, Input: {'args': (), 'kwargs': {'name': 'Hey Jude', 'artist': 'The Beatles'}}, Output: name='Hey Jude' artist='The Beatles'\n", "Name: generate_song, Input: {'args': (), 'kwargs': {'name': 'Let It Be', 'artist': 'The Beatles'}}, Output: name='Let It Be' artist='The Beatles'\n", "Name: generate_song, Input: {'args': (), 'kwargs': {'name': 'Yesterday', 'artist': 'The Beatles'}}, Output: name='Yesterday' artist='The Beatles'\n", "Name: generate_song, Input: {'args': (), 'kwargs': {'name': 'Come Together', 'artist': 'The Beatles'}}, Output: name='Come Together' artist='The Beatles'\n", "Name: generate_song, Input: {'args': (), 'kwargs': {'name': 'Help!', 'artist': 'The Beatles'}}, Output: name='Help!' artist='The Beatles'\n" ] } ], "source": [ "llm = OpenAI(model=\"gpt-3.5-turbo\")\n", "response = llm.predict_and_call(\n", " [tool],\n", " \"Generate five songs from the Beatles\",\n", " allow_parallel_tool_calls=True,\n", ")\n", "for s in response.sources:\n", " print(f\"Name: {s.tool_name}, Input: {s.raw_input}, Output: {str(s)}\")" ] }, { "cell_type": "markdown", "id": "7552118f", "metadata": {}, "source": [ "### Manual Tool Calling" ] }, { "cell_type": "markdown", "id": "dade92c4", "metadata": {}, "source": [ "If you want to control how a tool is called, you can also split the tool calling and tool selection into their own steps.\n", "\n", "First, lets select a tool." ] }, { "cell_type": "code", "execution_count": null, "id": "1a76226a", "metadata": {}, "outputs": [], "source": [ "from llama_index.core.llms import ChatMessage\n", "\n", "chat_history = [ChatMessage(role=\"user\", content=\"Pick a random song for me\")]\n", "\n", "resp = llm.chat_with_tools([tool], chat_history=chat_history)" ] }, { "cell_type": "markdown", "id": "b9eb38e2", "metadata": {}, "source": [ "Now, lets call the tool the LLM selected (if any).\n", "\n", "If there was a tool call, we should send the results to the LLM to generate the final response (or another tool call!)." ] }, { "cell_type": "code", "execution_count": null, "id": "43163a59", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Calling generate_song with {'name': 'Random Vibes', 'artist': 'DJ Chill'}\n" ] } ], "source": [ "tools_by_name = {t.metadata.name: t for t in [tool]}\n", "tool_calls = llm.get_tool_calls_from_response(\n", " resp, error_on_no_tool_call=False\n", ")\n", "\n", "while tool_calls:\n", " # add the LLM's response to the chat history\n", " chat_history.append(resp.message)\n", "\n", " for tool_call in tool_calls:\n", " tool_name = tool_call.tool_name\n", " tool_kwargs = tool_call.tool_kwargs\n", "\n", " print(f\"Calling {tool_name} with {tool_kwargs}\")\n", " tool_output = tool(**tool_kwargs)\n", " chat_history.append(\n", " ChatMessage(\n", " role=\"tool\",\n", " content=str(tool_output),\n", " # most LLMs like OpenAI need to know the tool call id\n", " additional_kwargs={\"tool_call_id\": tool_call.tool_id},\n", " )\n", " )\n", "\n", " resp = llm.chat_with_tools([tool], chat_history=chat_history)\n", " tool_calls = llm.get_tool_calls_from_response(\n", " resp, error_on_no_tool_call=False\n", " )" ] }, { "cell_type": "markdown", "id": "78d611c5", "metadata": {}, "source": [ "Now, we should have a final response!" ] }, { "cell_type": "code", "execution_count": null, "id": "2c5864e3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Here's a random song for you: **\"Random Vibes\"** by **DJ Chill**. Enjoy!\n" ] } ], "source": [ "print(resp.message.content)" ] }, { "cell_type": "markdown", "id": "7ede8d94-524b-4a51-8150-552df952f1bf", "metadata": {}, "source": [ "## Structured Prediction\n", "\n", "An important use case for function calling is extracting structured objects. LlamaIndex provides an intuitive interface for converting any LLM into a structured LLM - simply define the target Pydantic class (can be nested), and given a prompt, we extract out the desired object." ] }, { "cell_type": "code", "execution_count": null, "id": "04312089-bf9a-48d0-918f-ca1b3808439b", "metadata": {}, "outputs": [], "source": [ "from llama_index.llms.openai import OpenAI\n", "from llama_index.core.prompts import PromptTemplate\n", "from pydantic import BaseModel\n", "from typing import List\n", "\n", "\n", "class MenuItem(BaseModel):\n", " \"\"\"A menu item in a restaurant.\"\"\"\n", "\n", " course_name: str\n", " is_vegetarian: bool\n", "\n", "\n", "class Restaurant(BaseModel):\n", " \"\"\"A restaurant with name, city, and cuisine.\"\"\"\n", "\n", " name: str\n", " city: str\n", " cuisine: str\n", " menu_items: List[MenuItem]\n", "\n", "\n", "llm = OpenAI(model=\"gpt-3.5-turbo\")\n", "prompt_tmpl = PromptTemplate(\n", " \"Generate a restaurant in a given city {city_name}\"\n", ")\n", "# Option 1: Use `as_structured_llm`\n", "restaurant_obj = (\n", " llm.as_structured_llm(Restaurant)\n", " .complete(prompt_tmpl.format(city_name=\"Dallas\"))\n", " .raw\n", ")\n", "# Option 2: Use `structured_predict`\n", "# restaurant_obj = llm.structured_predict(Restaurant, prompt_tmpl, city_name=\"Miami\")" ] }, { "cell_type": "code", "execution_count": null, "id": "4b6fffcd-ff1e-4755-a851-1c6757a8075e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Restaurant(name='Tasty Bites', city='Dallas', cuisine='Italian', menu_items=[MenuItem(course_name='Appetizer', is_vegetarian=True), MenuItem(course_name='Main Course', is_vegetarian=False), MenuItem(course_name='Dessert', is_vegetarian=True)])" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "restaurant_obj" ] }, { "cell_type": "markdown", "id": "0c46b9ea-f6e0-4c4b-aaeb-9d77fcbb9f63", "metadata": {}, "source": [ "#### Structured Prediction with Streaming\n", "\n", "Any LLM wrapped with `as_structured_llm` supports streaming through `stream_chat`." ] }, { "cell_type": "code", "execution_count": null, "id": "8d55877e-9313-4e94-b448-b8ef68ed0a6d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'city': 'Boston',\n", " 'cuisine': 'American',\n", " 'menu_items': [{'course_name': 'Appetizer', 'is_vegetarian': True},\n", " {'course_name': 'Main Course', 'is_vegetarian': False},\n", " {'course_name': 'Dessert', 'is_vegetarian': True}],\n", " 'name': 'Boston Bites'}\n" ] }, { "data": { "text/plain": [ "Restaurant(name='Boston Bites', city='Boston', cuisine='American', menu_items=[MenuItem(course_name='Appetizer', is_vegetarian=True), MenuItem(course_name='Main Course', is_vegetarian=False), MenuItem(course_name='Dessert', is_vegetarian=True)])" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from llama_index.core.llms import ChatMessage\n", "from IPython.display import clear_output\n", "from pprint import pprint\n", "\n", "input_msg = ChatMessage.from_str(\"Generate a restaurant in Boston\")\n", "\n", "sllm = llm.as_structured_llm(Restaurant)\n", "stream_output = sllm.stream_chat([input_msg])\n", "for partial_output in stream_output:\n", " clear_output(wait=True)\n", " pprint(partial_output.raw.dict())\n", " restaurant_obj = partial_output.raw\n", "\n", "restaurant_obj" ] }, { "cell_type": "markdown", "id": "df5fa1ab-f598-46da-80f3-f6af5dd2fe83", "metadata": {}, "source": [ "## Async" ] }, { "cell_type": "code", "execution_count": null, "id": "3e088b90-12b6-4211-a9ca-696e9897e9ae", "metadata": {}, "outputs": [], "source": [ "from llama_index.llms.openai import OpenAI\n", "\n", "llm = OpenAI(model=\"gpt-3.5-turbo\")" ] }, { "cell_type": "code", "execution_count": null, "id": "8f649683-896a-439e-b14b-e5df5d803b82", "metadata": {}, "outputs": [], "source": [ "resp = await llm.acomplete(\"Paul Graham is \")" ] }, { "cell_type": "code", "execution_count": null, "id": "d05d8b7c-07b3-4ce8-9a8c-fcd1e830316c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "a computer scientist, entrepreneur, and venture capitalist. He is best known for co-founding the startup accelerator Y Combinator and for his work as an essayist and author on topics related to technology, startups, and entrepreneurship. Graham is also the co-founder of Viaweb, one of the first web-based applications, which was acquired by Yahoo in 1998. He has been a prominent figure in the tech industry for many years and is known for his insightful and thought-provoking writings on a wide range of subjects.\n" ] } ], "source": [ "print(resp)" ] }, { "cell_type": "code", "execution_count": null, "id": "abc72d09-3bcd-4d48-9bb7-0920c56310c1", "metadata": {}, "outputs": [], "source": [ "resp = await llm.astream_complete(\"Paul Graham is \")" ] }, { "cell_type": "code", "execution_count": null, "id": "b28d7a06-1518-4ec0-b3cc-6364395b3561", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "Paul Graham is an entrepreneur, venture capitalist, and computer scientist. He is best known for his work in the startup world, having co-founded the accelerator Y Combinator and investing in many successful startups such as Airbnb, Dropbox, and Stripe. He is also a prolific writer, having authored several books on topics such as startups, programming, and technology." ] } ], "source": [ "async for delta in resp:\n", " print(delta.delta, end=\"\")" ] }, { "cell_type": "markdown", "id": "3782e752-3c65-4b11-b2c6-8efe55fc33b1", "metadata": {}, "source": [ "Async function calling is also supported." ] }, { "cell_type": "code", "execution_count": null, "id": "7dcfaa77-edfe-42f9-8138-884c99dd3e62", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "name='Sunshine' artist='John Smith'\n" ] } ], "source": [ "llm = OpenAI(model=\"gpt-3.5-turbo\")\n", "response = await llm.apredict_and_call([tool], \"Generate a song\")\n", "print(str(response))" ] }, { "cell_type": "markdown", "id": "a2782f06", "metadata": {}, "source": [ "## Set API Key at a per-instance level\n", "If desired, you can have separate LLM instances use separate API keys." ] }, { "cell_type": "code", "execution_count": null, "id": "015c2d39", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "a computer scientist, entrepreneur, and venture capitalist. He is best known as the co-founder of the startup accelerator Y Combinator. Graham has also written several influential essays on startups and entrepreneurship, which have gained a wide following in the tech industry. He has been involved in the founding and funding of numerous successful startups, including Reddit, Dropbox, and Airbnb. Graham is known for his insightful and often controversial opinions on various topics, including education, inequality, and the future of technology.\n" ] } ], "source": [ "from llama_index.llms.openai import OpenAI\n", "\n", "llm = OpenAI(model=\"gpt-3.5-turbo\", api_key=\"BAD_KEY\")\n", "resp = llm.complete(\"Paul Graham is \")\n", "print(resp)" ] }, { "cell_type": "markdown", "id": "2d3d07bd", "metadata": {}, "source": [ "## Additional kwargs\n", "Rather than adding same parameters to each chat or completion call, you can set them at a per-instance level with `additional_kwargs`." ] }, { "cell_type": "code", "execution_count": null, "id": "b2da534d", "metadata": {}, "outputs": [], "source": [ "from llama_index.llms.openai import OpenAI\n", "\n", "llm = OpenAI(model=\"gpt-3.5-turbo\", additional_kwargs={\"user\": \"your_user_id\"})\n", "resp = llm.complete(\"Paul Graham is \")\n", "print(resp)" ] }, { "cell_type": "code", "execution_count": null, "id": "890efd05", "metadata": {}, "outputs": [], "source": [ "from llama_index.llms.openai import OpenAI\n", "\n", "llm = OpenAI(model=\"gpt-3.5-turbo\", additional_kwargs={\"user\": \"your_user_id\"})\n", "messages = [\n", " ChatMessage(\n", " role=\"system\", content=\"You are a pirate with a colorful personality\"\n", " ),\n", " ChatMessage(role=\"user\", content=\"What is your name\"),\n", "]\n", "resp = llm.chat(messages)" ] }, { "cell_type": "markdown", "id": "80fe5482", "metadata": {}, "source": [ "## RAG with LlamaCloud\n", "\n", "LlamaCloud is our cloud-based service that allows you to upload, parse, and index documents, and then search them using LlamaIndex. LlamaCloud is currently in a private alpha; please [get in touch](https://docs.google.com/forms/d/e/1FAIpQLSdehUJJB4NIYfrPIKoFdF4j8kyfnLhMSH_qYJI_WGQbDWD25A/viewform) if you'd like to be considered as a design partner." ] }, { "cell_type": "markdown", "id": "8fcf590a", "metadata": {}, "source": [ "### Installation" ] }, { "cell_type": "code", "execution_count": null, "id": "f345dd74", "metadata": {}, "outputs": [], "source": [ "%pip install llama-cloud-services" ] }, { "cell_type": "markdown", "id": "1f452fa6", "metadata": {}, "source": [ "### Setup OpenAI and LlamaCloud API Keys" ] }, { "cell_type": "code", "execution_count": null, "id": "2896e8dc", "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "os.environ[\"OPENAI_API_KEY\"] = \"sk-...\"\n", "\n", "os.environ[\"LLAMA_CLOUD_API_KEY\"] = \"llx-...\"" ] }, { "cell_type": "code", "execution_count": null, "id": "12feb97b", "metadata": {}, "outputs": [], "source": [ "from llama_cloud.client import LlamaCloud\n", "\n", "client = LlamaCloud(token=os.environ[\"LLAMA_CLOUD_API_KEY\"])" ] }, { "cell_type": "markdown", "id": "fe879565", "metadata": {}, "source": [ "### Create a Pipeline.\n", "\n", "Pipeline is an empty index on which you can ingest data.\n", "\n", "\n", "You need to Setup transformation and embedding config which will be used while ingesting the data." ] }, { "cell_type": "code", "execution_count": null, "id": "c9f2b3f1", "metadata": {}, "outputs": [], "source": [ "# Embedding config\n", "embedding_config = {\n", " \"type\": \"OPENAI_EMBEDDING\",\n", " \"component\": {\n", " \"api_key\": os.environ[\"OPENAI_API_KEY\"],\n", " \"model_name\": \"text-embedding-ada-002\", # You can choose any OpenAI Embedding model\n", " },\n", "}\n", "\n", "# Transformation auto config\n", "transform_config = {\n", " \"mode\": \"auto\",\n", " \"config\": {\n", " \"chunk_size\": 1024, # editable\n", " \"chunk_overlap\": 20, # editable\n", " },\n", "}\n", "\n", "pipeline = {\n", " \"name\": \"openai-rag-pipeline\", # Change the name if needed\n", " \"embedding_config\": embedding_config,\n", " \"transform_config\": transform_config,\n", " \"data_sink_id\": None,\n", "}\n", "\n", "pipeline = client.pipelines.upsert_pipeline(request=pipeline)" ] }, { "cell_type": "markdown", "id": "89a646cc", "metadata": {}, "source": [ "### File Upload\n", "\n", "We will upload files and add them to the index." ] }, { "cell_type": "code", "execution_count": null, "id": "9b994ca8", "metadata": {}, "outputs": [], "source": [ "with open(\"../data/10k/uber_2021.pdf\", \"rb\") as f:\n", " file = client.files.upload_file(upload_file=f)" ] }, { "cell_type": "code", "execution_count": null, "id": "27b256e0", "metadata": {}, "outputs": [], "source": [ "files = [{\"file_id\": file.id}]\n", "\n", "pipeline_files = client.pipelines.add_files_to_pipeline(\n", " pipeline.id, request=files\n", ")" ] }, { "cell_type": "markdown", "id": "bbcf7e4f", "metadata": {}, "source": [ "### Check the Ingestion job status" ] }, { "cell_type": "code", "execution_count": null, "id": "a8947b06", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "jobs = client.pipelines.list_pipeline_jobs(pipeline.id)\n", "\n", "jobs[0].status" ] }, { "cell_type": "markdown", "id": "e027b28e", "metadata": {}, "source": [ "### Connect to Index.\n", "\n", "Once the ingestion job is done, head over to your index on the [platform](https://cloud.llamaindex.ai/) and get the necessary details to connect to the index." ] }, { "cell_type": "code", "execution_count": null, "id": "d17c769a", "metadata": {}, "outputs": [], "source": [ "from llama_cloud_services import LlamaCloudIndex\n", "\n", "index = LlamaCloudIndex(\n", " name=\"openai-rag-pipeline\",\n", " project_name=\"Default\",\n", " organization_id=\"YOUR ORG ID\",\n", " api_key=os.environ[\"LLAMA_CLOUD_API_KEY\"],\n", ")" ] }, { "cell_type": "markdown", "id": "adc6cbb2", "metadata": {}, "source": [ "### Test on Sample Query" ] }, { "cell_type": "code", "execution_count": null, "id": "1efceb5d", "metadata": {}, "outputs": [], "source": [ "query = \"What is the revenue of Uber in 2021?\"" ] }, { "cell_type": "markdown", "id": "408da2bb", "metadata": {}, "source": [ "### Retriever \n", "\n", "Here we use hybrid search and re-ranker (cohere re-ranker by default)." ] }, { "cell_type": "code", "execution_count": null, "id": "f1204749", "metadata": {}, "outputs": [], "source": [ "retriever = index.as_retriever(\n", " dense_similarity_top_k=3,\n", " sparse_similarity_top_k=3,\n", " alpha=0.5,\n", " enable_reranking=True,\n", ")\n", "\n", "retrieved_nodes = retriever.retrieve(query)" ] }, { "cell_type": "markdown", "id": "d1470409", "metadata": {}, "source": [ "#### Display the retrieved nodes" ] }, { "cell_type": "code", "execution_count": null, "id": "cde7b9fc", "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "**Node ID:** 6341cc9c-1d81-46d6-afa3-9c2490f79514
**Similarity:** 0.99879813
**Text:** 2021 Compared to 2020\n", "\n", "Revenue increased $6.3 billion, or 57%, primarily attributable to an increase in Gross Bookings of 56%, or 53% on a constant currency basis. The increase in Gross Bookings was primarily driven by an increase in Delivery Gross Bookings of 71%, or 66% on a constant currency basis, due to an increase in food delivery orders and higher basket sizes as a result of stay-at-home order demand related to COVID-19, as well as continued expansion across U.S. and international markets. The increase was also driven by Mobility Gross Bookings growth of 38%, or 36% on a constant currency basis, due to increases in Trip volumes as the business recovers from the impacts of COVID-19. Additionally, we saw an increase in Delivery revenue resulting from an increase in certain Courier payments and incentives that are recorded in cost of revenue, where we are primarily responsible for delivery services and pay Couriers for services provided.
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "**Node ID:** e022d492-0fe0-4988-979e-dc5de9eeaf2d
**Similarity:** 0.996597
**Text:** Highlights for 2021\n", "\n", "Overall Gross Bookings increased by $32.5 billion in 2021, up 56%, or 53% on a constant currency basis, compared to 2020. Delivery Gross Bookings grew 66% from 2020, on a constant currency basis, due to an increase in food delivery orders and higher basket sizes as a result of stay-at-home order demand related to COVID-19, as well as continued expansion across U.S. and international markets. Additionally, we saw an increase in Delivery revenue resulting from an increase in certain Courier payments and incentives that are recorded in cost of revenue, where we are primarily responsible for delivery services and pay Couriers for services provided. Mobility Gross Bookings grew 36%, on a constant currency basis, from 2020, due to increases in Trip volumes as the business recovers from the impacts of COVID-19.\n", "\n", "Revenue was $17.5 billion, or up 57% year-over-year, reflecting the overall growth in our Delivery business and an increase in Freight revenue attributable to ...
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "**Node ID:** 00d31b26-b734-4475-b47a-8cb839ff65e0
**Similarity:** 0.9962638
**Text:** 2021 Compared to 2020\n", "\n", "Cost of revenue, exclusive of depreciation and amortization, increased $4.2 billion, or 81%, mainly due to a $2.1 billion increase in Courier payments and incentives in certain markets, a $660 million increase in insurance expense primarily due to an increase in miles driven in our Delivery business, and a $873 million increase in Freight carrier payments.\n", "---
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from llama_index.core.response.notebook_utils import display_source_node\n", "\n", "for retrieved_node in retrieved_nodes:\n", " display_source_node(retrieved_node, source_length=1000)" ] }, { "cell_type": "markdown", "id": "34c01d6f", "metadata": {}, "source": [ "#### Query Engine\n", "\n", "QueryEngine to setup entire RAG workflow." ] }, { "cell_type": "code", "execution_count": null, "id": "b8b8ab1c", "metadata": {}, "outputs": [], "source": [ "query_engine = index.as_query_engine(\n", " dense_similarity_top_k=3,\n", " sparse_similarity_top_k=3,\n", " alpha=0.5,\n", " enable_reranking=True,\n", ")" ] }, { "cell_type": "markdown", "id": "f4b9c590", "metadata": {}, "source": [ "#### Response" ] }, { "cell_type": "code", "execution_count": null, "id": "b5b1ed5d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The revenue of Uber in 2021 was $17.5 billion.\n" ] } ], "source": [ "response = query_engine.query(query)\n", "\n", "print(response)" ] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "llama-index-caVs7DDe-py3.10", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3" } }, "nbformat": 4, "nbformat_minor": 5 }