/09-metacognition/code_samples/09-python-agent-framework.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a1b2c3d4",
   "metadata": {},
   "source": [
    "# Lesson 09 - Metacognition Design Pattern"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2c3d4e5",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "This notebook demonstrates the Metacognition design pattern using the Microsoft Agent Framework.\n",
    "\n",
    "**Prerequisites:**\n",
    "- Azure OpenAI deployment configured via environment variables\n",
    "- Azure CLI authenticated (`az login`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3d4e5f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install agent-framework azure-ai-projects azure-identity python-dotenv -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4e5f6a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "logging.getLogger(\"agent_framework.foundry\").setLevel(logging.ERROR)\n",
    "\n",
    "import os\n",
    "import asyncio\n",
    "import dotenv\n",
    "from typing import Annotated\n",
    "\n",
    "from agent_framework import tool\n",
    "from agent_framework.foundry import FoundryChatClient\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "dotenv.load_dotenv()\n",
    "\n",
    "endpoint = os.getenv(\"AZURE_AI_PROJECT_ENDPOINT\")\n",
    "deployment_name = os.getenv(\"AZURE_AI_MODEL_DEPLOYMENT_NAME\")\n",
    "\n",
    "missing = [k for k, v in {\n",
    "    \"AZURE_AI_PROJECT_ENDPOINT\": endpoint,\n",
    "    \"AZURE_AI_MODEL_DEPLOYMENT_NAME\": deployment_name\n",
    "}.items() if not v]\n",
    "\n",
    "if missing:\n",
    "    raise ValueError(\n",
    "        f\"Missing required environment variables: {', '.join(missing)}. \"\n",
    "        \"Please set them as environment variables (e.g., in your .env file or shell environment).\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5f6a7b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the Microsoft Foundry client\n",
    "client = FoundryChatClient(\n",
    "    project_endpoint=endpoint,\n",
    "    model=deployment_name,\n",
    "    credential=DefaultAzureCredential()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6a7b8c9",
   "metadata": {},
   "source": [
    "## What is Metacognition?\n",
    "\n",
    "Metacognition is **thinking about thinking**. In the context of AI agents, it means building agents that can:\n",
    "\n",
    "- **Self-reflect** on their own outputs and reasoning process\n",
    "- **Detect errors** and recover gracefully instead of failing silently\n",
    "- **Evaluate** whether their responses are complete and helpful\n",
    "- **Adapt** their strategy when an initial approach doesn't work (e.g., falling back to a backup system)\n",
    "\n",
    "A metacognitive agent doesn't just answer questions — it monitors its own performance and adjusts on the fly."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7b8c9d0",
   "metadata": {},
   "source": [
    "## Primary and Backup Tools\n",
    "\n",
    "A common metacognition pattern is the **fallback strategy**. The agent tries a primary tool first; if it fails (e.g., a 404 error), the agent recognizes the failure and transparently switches to a backup tool.\n",
    "\n",
    "This mirrors real-world systems where primary services may be unavailable and agents must self-diagnose the issue before choosing an alternative path.\n",
    "\n",
    "Below we define two flight lookup tools:\n",
    "- **Primary** — covers Paris, Tokyo, and Barcelona\n",
    "- **Backup** — covers Berlin, Sydney, and New York City"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8c9d0e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "@tool(approval_mode=\"never_require\")\n",
    "def get_flight_times(\n",
    "    destination: Annotated[str, \"The destination city\"]\n",
    ") -> str:\n",
    "    \"\"\"Get available flight times for a destination (primary source).\"\"\"\n",
    "    flights = {\n",
    "        \"Paris\": \"Departures: 08:00, 12:30, 17:45 — from $350\",\n",
    "        \"Tokyo\": \"Departures: 11:00, 23:30 — from $890\",\n",
    "        \"Barcelona\": \"Departures: 07:15, 14:00, 19:30 — from $280\",\n",
    "    }\n",
    "    if destination in flights:\n",
    "        return flights[destination]\n",
    "    raise Exception(f\"404: No flights found for {destination} in primary system\")\n",
    "\n",
    "\n",
    "@tool(approval_mode=\"never_require\")\n",
    "def get_flight_times_backup(\n",
    "    destination: Annotated[str, \"The destination city\"]\n",
    ") -> str:\n",
    "    \"\"\"Get available flight times from backup system (used when primary fails).\"\"\"\n",
    "    backup_flights = {\n",
    "        \"Berlin\": \"Departures: 09:00, 16:00 — from $220\",\n",
    "        \"Sydney\": \"Departures: 22:00 — from $1200\",\n",
    "        \"New York City\": \"Departures: 06:00, 10:30, 15:00, 20:00 — from $450\",\n",
    "    }\n",
    "    return backup_flights.get(\n",
    "        destination,\n",
    "        f\"No flights found for {destination} in any system. Please try again later.\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9d0e1f2",
   "metadata": {},
   "source": [
    "## Self-Reflecting Agent with Error Recovery\n",
    "\n",
    "The agent below is instructed to try the primary flight system first, recognize failures, and transparently fall back to the backup system. After each response it briefly self-evaluates whether it fully answered the user's question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0e1f2a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = client.as_agent(\n",
    "    tools=[get_flight_times, get_flight_times_backup],\n",
    "    name=\"FlightBookingAgent\",\n",
    "    instructions=\"\"\"You are a flight booking agent with self-reflection capabilities.\n",
    "\n",
    "When looking up flights:\n",
    "1. Try the primary flight system first (get_flight_times)\n",
    "2. If the primary system fails (404 error), acknowledge the error and try the backup system (get_flight_times_backup)\n",
    "3. Always explain to the user what happened — be transparent about fallbacks\n",
    "4. If both systems fail, apologize and suggest alternatives\n",
    "\n",
    "After each response, briefly evaluate whether your answer was complete and helpful.\"\"\",\n",
    ")\n",
    "\n",
    "# Test with a destination in primary system\n",
    "print(\"=== Test 1: Destination in primary system ===\")\n",
    "response = await agent.run(\n",
    "    \"What flights are available to Paris?\",\n",
    "    )\n",
    "print(response)\n",
    "\n",
    "# Test with a destination only in backup system\n",
    "print(\"\\n=== Test 2: Destination only in backup system ===\")\n",
    "response = await agent.run(\n",
    "    \"What flights are available to Berlin?\",\n",
    "    )\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1f2a3b4",
   "metadata": {},
   "source": [
    "## Self-Evaluation Pattern\n",
    "\n",
    "Another facet of metacognition is **self-evaluation**: a separate agent (or the same agent in a second pass) reviews a response for completeness, accuracy, and helpfulness.\n",
    "\n",
    "Below we create a `ResponseEvaluator` agent that scores travel-agent responses on three dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2a3b4c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluation_agent = client.as_agent(\n",
    "    tools=[get_flight_times, get_flight_times_backup],\n",
    "    name=\"ResponseEvaluator\",\n",
    "    instructions=\"\"\"You are a quality evaluator for travel agent responses.\n",
    "Given a travel question and the agent's response, evaluate:\n",
    "1. Completeness: Did it answer all parts of the question? (1-5)\n",
    "2. Accuracy: Is the information correct? (1-5)\n",
    "3. Helpfulness: Would a traveler find this useful? (1-5)\n",
    "Provide a brief evaluation with scores and one suggestion for improvement.\"\"\",\n",
    ")\n",
    "\n",
    "# Evaluate the agent's response from Test 1\n",
    "eval_prompt = f\"\"\"Question: What flights are available to Paris?\n",
    "Agent Response: {response}\n",
    "\n",
    "Please evaluate the above response.\"\"\"\n",
    "\n",
    "evaluation = await evaluation_agent.run(eval_prompt)\n",
    "print(\"=== Self-Evaluation ===\")\n",
    "print(evaluation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3b4c5d6",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this lesson you learned how to build **metacognitive agents** using the Microsoft Agent Framework:\n",
    "\n",
    "- **Self-reflection**: Agents that monitor their own reasoning and transparently communicate what happened.\n",
    "- **Error recovery with fallbacks**: A primary + backup tool pattern where the agent detects failures (e.g., 404 errors) and automatically tries an alternative source.\n",
    "- **Self-evaluation**: A separate evaluator agent that scores responses for completeness, accuracy, and helpfulness.\n",
    "\n",
    "These patterns make agents more robust, transparent, and trustworthy — critical qualities for production deployments."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}