{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluate with langchain's evaluator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
    "\n",
    "- Convert LangChain criteria evaluator applications to `flex flow`.\n",
    "- Use `CustomConnection` to store secrets.\n",
    "\n",
    "## 0. Install dependent packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture --no-stderr\n",
    "%pip install -r ./requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Trace your langchain evaluator with prompt flow\n",
    "### Initialize a pf client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from promptflow.client import PFClient\n",
    "\n",
    "pf = PFClient()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a custom connection to protect your API key\n",
    "\n",
    "You can protect your API key in custom connection's secrets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "from promptflow.entities import CustomConnection\n",
    "\n",
    "conn_name = \"my_llm_connection\"\n",
    "\n",
    "try:\n",
    "    conn = pf.connections.get(name=conn_name)\n",
    "    print(\"using existing connection\")\n",
    "except:\n",
    "    if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
    "        # load environment variables from .env file\n",
    "        load_dotenv()\n",
    "\n",
    "    # put API key in secrets\n",
    "    connection = CustomConnection(\n",
    "        name=conn_name,\n",
    "        configs={\n",
    "            \"azure_endpoint\": os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
    "        },\n",
    "        secrets={\n",
    "            # store API key\n",
    "            # \"anthropic_api_key\": \"<your-api-key>\",\n",
    "            \"openai_api_key\": os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
    "        },\n",
    "    )\n",
    "    # Create the connection, note that all secret values will be scrubbed in the returned result\n",
    "    conn = pf.connections.create_or_update(connection)\n",
    "    print(\"successfully created connection\")\n",
    "print(conn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the evaluator with trace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from eval_conciseness import LangChainEvaluator\n",
    "\n",
    "\n",
    "evaluator = LangChainEvaluator(custom_connection=conn)\n",
    "result = evaluator(\n",
    "    prediction=\"What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four.\",\n",
    "    input=\"What's 2+2?\",\n",
    ")\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Batch run the evaluator with flow yaml\n",
    "Create a [flow.flex.yaml](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/eval-criteria-with-langchain/flow.flex.yaml) file to define a flow which entry pointing to the python function we defined.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = \"./data.jsonl\"  # path to the data file\n",
    "# create run with the flow function and data\n",
    "base_run = pf.run(\n",
    "    flow=\"./flow.flex.yaml\",\n",
    "    # reference custom connection by name\n",
    "    init={\n",
    "        \"custom_connection\": \"my_llm_connection\",\n",
    "    },\n",
    "    data=data,\n",
    "    column_mapping={\n",
    "        \"prediction\": \"${data.prediction}\",\n",
    "        \"input\": \"${data.input}\",\n",
    "    },\n",
    "    stream=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "details = pf.get_details(base_run)\n",
    "details.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pf.visualize([base_run])"
   ]
  }
 ],
 "metadata": {
  "build_doc": {
   "author": [
    "D-W-@github.com",
    "wangchao1230@github.com"
   ],
   "category": "local",
   "section": "Flow",
   "weight": 60
  },
  "description": "A tutorial to converting LangChain criteria evaluator application to flex flow.",
  "kernelspec": {
   "display_name": "prompt_flow",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  },
  "resources": "examples/flex-flows/eval-criteria-with-langchain"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}