/examples/flows/chat/chat-with-pdf/chat-with-pdf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chat with PDF - test, evaluation and experimentation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "We will walk you through how to use prompt flow Python SDK to test, evaluate and experiment with the \"Chat with PDF\" flow.\n",
    "\n",
    "## 0. Install dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -r requirements.txt"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Create connections\n",
    "Connection in prompt flow is for managing settings of your application behaviors incl. how to talk to different services (Azure OpenAI for example)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import promptflow\n",
    "\n",
    "pf = promptflow.PFClient()\n",
    "\n",
    "# List all the available connections\n",
    "for c in pf.connections.list():\n",
    "    print(c.name + \" (\" + c.type + \")\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need to have a connection named \"open_ai_connection\" to run the chat_with_pdf flow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create needed connection\n",
    "from promptflow.entities import AzureOpenAIConnection, OpenAIConnection\n",
    "\n",
    "try:\n",
    "    conn_name = \"open_ai_connection\"\n",
    "    conn = pf.connections.get(name=conn_name)\n",
    "    print(\"using existing connection\")\n",
    "except:\n",
    "    # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
    "    connection = AzureOpenAIConnection(\n",
    "        name=conn_name,\n",
    "        api_key=\"<user-input>\",\n",
    "        api_base=\"<test_base>\",\n",
    "        api_type=\"azure\",\n",
    "        api_version=\"<test_version>\",\n",
    "    )\n",
    "\n",
    "    # use this if you have an existing OpenAI account\n",
    "    # connection = OpenAIConnection(\n",
    "    #     name=conn_name,\n",
    "    #     api_key=\"<user-input>\",\n",
    "    # )\n",
    "    conn = pf.connections.create_or_update(connection)\n",
    "    print(\"successfully created connection\")\n",
    "\n",
    "print(conn)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Test the flow\n",
    "\n",
    "**Note**: this sample uses `predownloaded PDFs` and `prebuilt FAISS Index` to speed up execution time.\n",
    "You can remove the folders to start a fresh run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ./chat_with_pdf/.pdfs/ stores predownloaded PDFs\n",
    "# ./chat_with_pdf/.index/ stores prebuilt index files\n",
    "\n",
    "output = pf.flows.test(\n",
    "    \".\",\n",
    "    inputs={\n",
    "        \"chat_history\": [],\n",
    "        \"pdf_url\": \"https://arxiv.org/pdf/1810.04805.pdf\",\n",
    "        \"question\": \"what is BERT?\",\n",
    "    },\n",
    ")\n",
    "print(output)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Run the flow with a data file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flow_path = \".\"\n",
    "data_path = \"./data/bert-paper-qna-3-line.jsonl\"\n",
    "\n",
    "config_2k_context = {\n",
    "    \"EMBEDDING_MODEL_DEPLOYMENT_NAME\": \"text-embedding-ada-002\",\n",
    "    \"CHAT_MODEL_DEPLOYMENT_NAME\": \"gpt-4\",  # change this to the name of your deployment if you're using Azure OpenAI\n",
    "    \"PROMPT_TOKEN_LIMIT\": 2000,\n",
    "    \"MAX_COMPLETION_TOKENS\": 256,\n",
    "    \"VERBOSE\": True,\n",
    "    \"CHUNK_SIZE\": 1024,\n",
    "    \"CHUNK_OVERLAP\": 64,\n",
    "}\n",
    "\n",
    "column_mapping = {\n",
    "    \"question\": \"${data.question}\",\n",
    "    \"pdf_url\": \"${data.pdf_url}\",\n",
    "    \"chat_history\": \"${data.chat_history}\",\n",
    "    \"config\": config_2k_context,\n",
    "}\n",
    "run_2k_context = pf.run(flow=flow_path, data=data_path, column_mapping=column_mapping)\n",
    "pf.stream(run_2k_context)\n",
    "\n",
    "print(run_2k_context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pf.get_details(run_2k_context)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Evaluate the \"groundedness\"\n",
    "The `eval-groundedness flow` is using ChatGPT/GPT4 model to grade the answers generated by chat-with-pdf flow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_groundedness_flow_path = \"../../evaluation/eval-groundedness/\"\n",
    "eval_groundedness_2k_context = pf.run(\n",
    "    flow=eval_groundedness_flow_path,\n",
    "    run=run_2k_context,\n",
    "    column_mapping={\n",
    "        \"question\": \"${run.inputs.question}\",\n",
    "        \"answer\": \"${run.outputs.answer}\",\n",
    "        \"context\": \"${run.outputs.context}\",\n",
    "    },\n",
    "    display_name=\"eval_groundedness_2k_context\",\n",
    ")\n",
    "pf.stream(eval_groundedness_2k_context)\n",
    "\n",
    "print(eval_groundedness_2k_context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pf.get_details(eval_groundedness_2k_context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pf.get_metrics(eval_groundedness_2k_context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pf.visualize(eval_groundedness_2k_context)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will see a web page like this. It gives you detail about how each row is graded and even the details how the evaluation run executes:\n",
    "![pf-visualize-screenshot](./media/chat-with-pdf/pf-visualize-screenshot.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Try a different configuration and evaluate again - experimentation\n",
    "\n",
    "NOTE: since we only use 3 lines of test data in this example, and because of the non-deterministic nature of LLMs, don't be surprised if you see exact same metrics when you run this process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "config_3k_context = {\n",
    "    \"EMBEDDING_MODEL_DEPLOYMENT_NAME\": \"text-embedding-ada-002\",\n",
    "    \"CHAT_MODEL_DEPLOYMENT_NAME\": \"gpt-4\",  # change this to the name of your deployment if you're using Azure OpenAI\n",
    "    \"PROMPT_TOKEN_LIMIT\": 3000,\n",
    "    \"MAX_COMPLETION_TOKENS\": 256,\n",
    "    \"VERBOSE\": True,\n",
    "    \"CHUNK_SIZE\": 1024,\n",
    "    \"CHUNK_OVERLAP\": 64,\n",
    "}\n",
    "\n",
    "run_3k_context = pf.run(flow=flow_path, data=data_path, column_mapping=column_mapping)\n",
    "pf.stream(run_3k_context)\n",
    "\n",
    "print(run_3k_context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_groundedness_3k_context = pf.run(\n",
    "    flow=eval_groundedness_flow_path,\n",
    "    run=run_3k_context,\n",
    "    column_mapping={\n",
    "        \"question\": \"${run.inputs.question}\",\n",
    "        \"answer\": \"${run.outputs.answer}\",\n",
    "        \"context\": \"${run.outputs.context}\",\n",
    "    },\n",
    "    display_name=\"eval_groundedness_3k_context\",\n",
    ")\n",
    "pf.stream(eval_groundedness_3k_context)\n",
    "\n",
    "print(eval_groundedness_3k_context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pf.get_details(eval_groundedness_3k_context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pf.visualize([eval_groundedness_2k_context, eval_groundedness_3k_context])"
   ]
  }
 ],
 "metadata": {
  "build_doc": {
   "author": [
    "wangchao1230@github.com",
    "ttthree@github.com"
   ],
   "category": "local",
   "section": "Rag",
   "weight": 10
  },
  "description": "A tutorial of chat-with-pdf flow that allows user ask questions about the content of a PDF file and get answers",
  "kernelspec": {
   "display_name": "prompt-flow",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}