{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1079e444-91e1-4b81-a28a-2ce4763f4bc4",
   "metadata": {},
   "source": [
    "# Langfuse\n",
    "\n",
    "Ragas and Langfuse is a powerful combination that can help you evaluate and monitor your Retrieval-Augmented Generation (RAG) pipelines.\n",
    "\n",
    "## What is Langfuse?\n",
    "\n",
    "Langfuse ([GitHub](https://github.com/langfuse/langfuse)) is an open-source platform for LLM [tracing](https://langfuse.com/docs/tracing), [prompt management](https://langfuse.com/docs/prompts/get-started), and [evaluation](https://langfuse.com/docs/scores/overview). It allows you to score your traces and spans, providing insights into the performance of your RAG pipelines. Langfuse supports various integrations, including [OpenAI](https://langfuse.com/docs/integrations/openai/python/get-started), [Langchain](https://langfuse.com/docs/integrations/langchain/tracing), and [more](https://langfuse.com/docs/integrations/overview).\n",
    "\n",
    "## Key Benefits of using Langfuse with Ragas\n",
    "\n",
    "- **Score Traces**: [Score](https://langfuse.com/docs/scores/overview) your traces and spans, providing insights into the performance of your RAG pipelines.\n",
    "- **Detailed Analytics**: Segment and [analyze](https://langfuse.com/docs/analytics/overview) traces to identify low-quality scores and improve your system's performance.\n",
    "- **Score Reporting**: Drill down into detailed reports for specific use cases and user segments.\n",
    "\n",
    "Ragas ([GitHub](https://github.com/vibrantlabsai/ragas)) is an open-source tool that can help you run [Model-Based Evaluation](https://langfuse.com/docs/scores/model-based-evals) on your traces/spans, especially for RAG pipelines. Ragas can perform reference-free evaluations of various aspects of your RAG pipeline. Because it is reference-free you don't need ground-truths when running the evaluations and can run it on production traces that you've collected with Langfuse.\n",
    "\n",
    "## Getting Started\n",
    "\n",
    "This guide will walk you through and end-to-end example of RAG evaluations with Ragas and Langfuse.\n",
    "\n",
    "### The Environment\n",
    "\n",
    "[Sign up](https://cloud.langfuse.com) for Langfuse to get your API keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "017dc09a-c59c-4e5f-a632-d8a5110f931d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# get keys for your project from https://cloud.langfuse.com\n",
    "os.environ[\"LANGFUSE_SECRET_KEY\"] = \"sk-...\"\n",
    "os.environ[\"LANGFUSE_PUBLIC_KEY\"] = \"pk-...\"\n",
    "\n",
    "# your openai key\n",
    "# os.environ[\"OPENAI_API_KEY\"] = \"sk-...\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90a9536a-4997-47a4-82a7-3970c1145dab",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install datasets ragas llama_index python-dotenv --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "580b6d2a-06e2-4682-8e03-47d054d7f240",
   "metadata": {},
   "source": [
    "### The Data\n",
    "\n",
    "For this example, we are going to use a dataset that has already been prepared by querying a RAG system and gathering its outputs. See below for instruction on how to fetch your production data from Langfuse.\n",
    "\n",
    "The dataset contains the following columns:\n",
    "- `question`: *list[str]* - These are the questions your RAG pipeline will be evaluated on.\n",
    "- `answer`: *list[str]* - The answer generated from the RAG pipeline and given to the user.\n",
    "- `contexts`: *list[list[str]]* - The contexts which were passed into the LLM to answer the question.\n",
    "- `ground_truth`: list[list[str]] - The ground truth answer to the questions. However, this can be ignored for online evaluations since we will not have access to ground-truth data in our case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ebfb8207-8ddc-4b61-bcbc-f257820bf671",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset amnesty_qa (/home/jjmachan/.cache/huggingface/datasets/vibrantlabsai___amnesty_qa/english_v2/2.0.0/d0ed9800191a31943ee52a5c22ee4305e28a33f5edcd9a323802112cff07cc24)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "77e7ed90dd244b5c93865eb284f31f6d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['question', 'ground_truth', 'answer', 'contexts'],\n",
       "    num_rows: 20\n",
       "})"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "amnesty_qa = load_dataset(\"vibrantlabsai/amnesty_qa\", \"english_v2\")[\"eval\"]\n",
    "amnesty_qa"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b889cb2a-f718-4104-a62b-7357f76742d5",
   "metadata": {},
   "source": [
    "### The Metrics\n",
    "In this example, we will use the following metrics from the Ragas library:\n",
    "\n",
    "- [`faithfulness`](https://docs.ragas.io/en/latest/concepts/metrics/faithfulness.html): This measures the factual consistency of the generated answer against the given context.\n",
    "- [`answer_relevancy`](https://docs.ragas.io/en/latest/concepts/metrics/answer_relevance.html): Answer Relevancy, focuses on assessing how to-the-point and relevant the generated answer is to the given prompt.\n",
    "- [`context precision`](https://docs.ragas.io/en/latest/concepts/metrics/context_precision.html): Context Precision is a metric that evaluates whether all of the ground-truth relevant items present in the contexts are ranked higher or not. Ideally, all the relevant chunks must appear at the top ranks. This metric is computed using the question and the contexts, with values ranging between 0 and 1, where higher scores indicate better precision.\n",
    "- [`aspect_critique`](https://docs.ragas.io/en/latest/concepts/metrics/critique.html): This is designed to assess submissions based on predefined aspects such as harmlessness and correctness. Additionally, users have the flexibility to define their own aspects for evaluating submissions according to their specific criteria.\n",
    "\n",
    "Have a look at the [documentation](https://docs.ragas.io/en/latest/concepts/metrics/index.html) to learn more about these metrics and how they work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "33c49997-a491-4aae-bc7f-01adb61071f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import metrics\n",
    "from ragas.metrics import answer_relevancy, context_precision, faithfulness\n",
    "from ragas.metrics.critique import harmfulness\n",
    "\n",
    "# metrics you chose\n",
    "metrics = [faithfulness, answer_relevancy, context_precision, harmfulness]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8da36b85",
   "metadata": {},
   "source": [
    "Next, initialize the metrics using the LLMs and Embeddings of your choice. In this example, we are using OpenAI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d51a0580",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.metrics.base import MetricWithEmbeddings, MetricWithLLM\n",
    "from ragas.run_config import RunConfig\n",
    "\n",
    "\n",
    "# util function to init Ragas Metrics\n",
    "def init_ragas_metrics(metrics, llm, embedding):\n",
    "    for metric in metrics:\n",
    "        if isinstance(metric, MetricWithLLM):\n",
    "            metric.llm = llm\n",
    "        if isinstance(metric, MetricWithEmbeddings):\n",
    "            metric.embeddings = embedding\n",
    "        run_config = RunConfig()\n",
    "        metric.init(run_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5c41d94d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_openai.chat_models import ChatOpenAI\n",
    "from langchain_openai.embeddings import OpenAIEmbeddings\n",
    "\n",
    "from ragas.embeddings import LangchainEmbeddingsWrapper\n",
    "\n",
    "# wrappers\n",
    "from ragas.llms import LangchainLLMWrapper\n",
    "\n",
    "llm = ChatOpenAI()\n",
    "emb = OpenAIEmbeddings()\n",
    "\n",
    "init_ragas_metrics(\n",
    "    metrics,\n",
    "    llm=LangchainLLMWrapper(llm),\n",
    "    embedding=LangchainEmbeddingsWrapper(emb),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23e21d8b-f72a-4043-b42e-4cf4c72d7dc8",
   "metadata": {},
   "source": [
    "### The Setup\n",
    "You can use model-based evaluation with Ragas in 2 ways:\n",
    "\n",
    "1. **Score each Trace**: This means you will run the evaluations for each trace item. This gives you much better idea since of how each call to your RAG pipelines is performing but can be expensive\n",
    "2. **Score as Batch**: In this method we will take a random sample of traces on a periodic basis and score them. This brings down cost and gives you a rough estimate the performance of your app but can miss out on important samples.\n",
    "\n",
    "In this cookbook, we'll show you how to setup both."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e482dec-f02c-4fa1-bd07-d292a863cde5",
   "metadata": {},
   "source": [
    "### Score the Trace\n",
    "\n",
    "Lets take a small example of a single trace and see how you can score that with Ragas. First lets load the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "184b901f-9f08-4ab1-96c4-1c50586753a3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "question:  What are the global implications of the USA Supreme Court ruling on abortion?\n",
      "answer:  The global implications of the USA Supreme Court ruling on abortion can be significant, as it sets a precedent for other countries and influences the global discourse on reproductive rights. Here are some potential implications:\n",
      "\n",
      "1. Influence on other countries: The Supreme Court's ruling can serve as a reference point for other countries grappling with their own abortion laws. It can provide legal arguments and reasoning that advocates for reproductive rights can use to challenge restrictive abortion laws in their respective jurisdictions.\n",
      "\n",
      "2. Strengthening of global reproductive rights movements: A favorable ruling by the Supreme Court can energize and empower reproductive rights movements worldwide. It can serve as a rallying point for activists and organizations advocating for women's rights, leading to increased mobilization and advocacy efforts globally.\n",
      "\n",
      "3. Counteracting anti-abortion movements: Conversely, a ruling that restricts abortion rights can embolden anti-abortion movements globally. It can provide legitimacy to their arguments and encourage similar restrictive measures in other countries, potentially leading to a rollback of existing reproductive rights.\n",
      "\n",
      "4. Impact on international aid and policies: The Supreme Court's ruling can influence international aid and policies related to reproductive health. It can shape the priorities and funding decisions of donor countries and organizations, potentially leading to increased support for reproductive rights initiatives or conversely, restrictions on funding for abortion-related services.\n",
      "\n",
      "5. Shaping international human rights standards: The ruling can contribute to the development of international human rights standards regarding reproductive rights. It can influence the interpretation and application of existing human rights treaties and conventions, potentially strengthening the recognition of reproductive rights as fundamental human rights globally.\n",
      "\n",
      "6. Global health implications: The Supreme Court's ruling can have implications for global health outcomes, particularly in countries with restrictive abortion laws. It can impact the availability and accessibility of safe and legal abortion services, potentially leading to an increase in unsafe abortions and related health complications.\n",
      "\n",
      "It is important to note that the specific implications will depend on the nature of the Supreme Court ruling and the subsequent actions taken by governments, activists, and organizations both within and outside the United States.\n"
     ]
    }
   ],
   "source": [
    "row = amnesty_qa[0]\n",
    "print(\"question: \", row[\"question\"])\n",
    "print(\"answer: \", row[\"answer\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f6138d0-2ace-4b0b-beb1-da9c20bd14a0",
   "metadata": {},
   "source": [
    "Now lets init a Langfuse client SDK to instrument you app."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fb0e3d09-fdd6-4093-8dfe-917bc58129e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langfuse import Langfuse\n",
    "\n",
    "langfuse = Langfuse()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65cf5034-c2f2-4e5e-b0bc-af34dd414020",
   "metadata": {},
   "source": [
    "Here we are defining a utility function to score your trace with the metrics you chose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d9b0af86-f152-4e63-a7db-264e7f0ccb18",
   "metadata": {},
   "outputs": [],
   "source": [
    "async def score_with_ragas(query, chunks, answer):\n",
    "    scores = {}\n",
    "    for m in metrics:\n",
    "        print(f\"calculating {m.name}\")\n",
    "        scores[m.name] = await m.ascore(\n",
    "            row={\"question\": query, \"contexts\": chunks, \"answer\": answer}\n",
    "        )\n",
    "    return scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "0afd699c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "calculating faithfulness\n",
      "calculating answer_relevancy\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using 'context_precision' without ground truth will be soon depreciated. Use 'context_utilization' instead\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "calculating context_precision\n",
      "calculating harmfulness\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'faithfulness': 0.0,\n",
       " 'answer_relevancy': 0.9999999999999996,\n",
       " 'context_precision': 0.9999999999,\n",
       " 'harmfulness': 0}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question, contexts, answer = row[\"question\"], row[\"contexts\"], row[\"answer\"]\n",
    "await score_with_ragas(question, contexts, answer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3b1e219-8319-4dfb-997b-0449a927e113",
   "metadata": {},
   "source": [
    "You compute the score with each request. Below we've outlined a dummy application that does the following steps:\n",
    "\n",
    "1. Gets a question from the user\n",
    "2. Fetch context from the database or vector store that can be used to answer the question from the user\n",
    "3. Pass the question and the contexts to the LLM to generate the answer\n",
    "\n",
    "All these step are logged as spans in a single trace in Langfuse. You can read more about traces and spans from the [Langfuse documentation](https://langfuse.com/docs/tracing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "91e87b6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# the logic of the dummy application is\n",
    "# given a question fetch the correspoinding contexts and answers from a dict\n",
    "\n",
    "import hashlib\n",
    "\n",
    "\n",
    "def hash_string(input_string):\n",
    "    return hashlib.sha256(input_string.encode()).hexdigest()\n",
    "\n",
    "\n",
    "q_to_c = {}  # map between question and context\n",
    "q_to_a = {}  # map between question and answer\n",
    "for row in amnesty_qa:\n",
    "    q_hash = hash_string(row[\"question\"])\n",
    "    q_to_c[q_hash] = row[\"contexts\"]\n",
    "    q_to_a[q_hash] = row[\"answer\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "c65284ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# if your running this in a notebook - please run this cell\n",
    "# to manage asyncio event loops\n",
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "f21a1623-5b47-4425-a6ed-e71a7d5bc25d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from asyncio import run\n",
    "\n",
    "from langfuse.decorators import langfuse_context, observe\n",
    "\n",
    "\n",
    "@observe()\n",
    "def retriver(question: str):\n",
    "    return q_to_c[question]\n",
    "\n",
    "\n",
    "@observe()\n",
    "def generator(question):\n",
    "    return q_to_a[question]\n",
    "\n",
    "\n",
    "@observe()\n",
    "def rag_pipeline(question):\n",
    "    q_hash = hash_string(question)\n",
    "    contexts = retriver(q_hash)\n",
    "    generated_answer = generator(q_hash)\n",
    "\n",
    "    # score the runs\n",
    "    score = run(score_with_ragas(question, contexts, answer=generated_answer))\n",
    "    for s in score:\n",
    "        langfuse_context.score_current_trace(name=s, value=score[s])\n",
    "    return generated_answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "29b2c7c6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "calculating faithfulness\n",
      "calculating answer_relevancy\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using 'context_precision' without ground truth will be soon depreciated. Use 'context_utilization' instead\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "calculating context_precision\n",
      "calculating harmfulness\n"
     ]
    }
   ],
   "source": [
    "question, contexts, answer = row[\"question\"], row[\"contexts\"], row[\"answer\"]\n",
    "generated_answer = rag_pipeline(amnesty_qa[0][\"question\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6403f6ac-807c-441c-9fe2-a6bf2f6771f3",
   "metadata": {},
   "source": [
    "### Analyze the Scores in Langfuse\n",
    "\n",
    "You can [analyze](https://langfuse.com/docs/analytics/overview) the scores in the Langfuse UI and drill down into the scores for each question or user.\n",
    "\n",
    "→ Not using Langfuse yet? Explore the dashboard in our [interactive demo](https://langfuse.com/docs/demo)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19a0e5d0-3a67-46e0-b5c6-baf272dbea3b",
   "metadata": {},
   "source": [
    "![Trace with RAGAS scores](https://langfuse.com/images/docs/ragas-trace-score.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fd68b13-9743-424f-830a-c6d32e3d09c6",
   "metadata": {},
   "source": [
    "Note that the scoring is blocking so make sure that you sent the generated answer before waiting for the scores to get computed. Alternatively you can run `score_with_ragas()` in a separate thread and pass in the `trace_id` to log the scores.\n",
    "\n",
    "## Resources\n",
    "\n",
    "- Have a look at our guide on [Model-Based Evaluation](https://langfuse.com/docs/scores/model-based-evals) to learn more about how to run model-based evaluations with Ragas.\n",
    "- Learn more about analyzing and improving your LLM application [here](https://langfuse.com/faq/all/llm-analytics-101).\n",
    "\n",
    "## Feedback\n",
    "\n",
    "If you have any feedback or requests, please create a GitHub [Issue](https://langfuse.com/issue) or share your work with the community on [Discord](https://discord.langfuse.com/).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b212e72c",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}