{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "4Pjmz-RORV8E" }, "source": [ "# Generate image captions and detect objects\n", "\n", "txtai as the name implies works with text and ai, pretty straightforward. But that doesn't mean it can't work with different types of content. For example, an image can be described with words. We can use that description to compare an image to a query or other documents. This notebook shows how images and text can be embedded into the same space to generate image captions and detect objects." ] }, { "cell_type": "markdown", "metadata": { "id": "Dk31rbYjSTYm" }, "source": [ "# Install dependencies\n", "\n", "Install `txtai` and all dependencies. Since this notebook is using optional pipelines, we need to install the pipeline extras package." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "id": "XMQuuun2R06J" }, "outputs": [], "source": [ "%%capture\n", "!pip install ipyplot git+https://github.com/neuml/txtai#egg=txtai[pipeline]\n", "\n", "# Get test data\n", "!wget -N https://github.com/neuml/txtai/releases/download/v3.5.0/tests.tar.gz\n", "!tar -xvzf tests.tar.gz" ] }, { "cell_type": "markdown", "metadata": { "id": "PNPJ95cdTKSS" }, "source": [ "# Create a captions instance\n", "\n", "The captions pipeline takes an image or list of images and generates captions. This pipelines works using a combination of an image encoder model and a text model. " ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "id": "nTDwXOUeTH2-" }, "outputs": [], "source": [ "%%capture\n", "\n", "from txtai.pipeline import Caption\n", "\n", "# Create caption pipeline\n", "caption = Caption()" ] }, { "cell_type": "markdown", "metadata": { "id": "-vGR_piwZZO6" }, "source": [ "# Generate captions\n", "\n", "The example below shows how to generate captions. A list of images are read from a directory, passed to a caption model and text descriptions are returned." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "-K2YJJzsVtfq", "outputId": "7cfd549a-1db6-47b9-c4ae-623e94ed48d1" }, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "\n", " \n", "
\n", " \n", " \n", " \n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {} }, { "output_type": "display_data", "data": { "text/html": [ "\n", " \n", "
\n", "
\n", "
\n", "

a clock on the side of a wall

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

a painting of flowers on top of a table

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

a computer screen with a picture of a person on it

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

a blurry photo of a sunset with a sky background

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

a blurry photo of a bunch of stuffed animals

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

motorcycles are parked on the side of the road

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

a large tree branch with a person in it

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

a city street at night with traffic lights

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

a book shelf filled with many books

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

a large building with many windows in a city

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
" ], "text/plain": [ "" ] }, "metadata": {} } ], "source": [ "import glob\n", "import ipyplot\n", "\n", "from PIL import Image\n", "\n", "# Get list of images\n", "images = glob.glob('txtai/*jpg')\n", "\n", "# Generate captions\n", "captions = caption(images)\n", "\n", "# Show image/caption pairs\n", "ipyplot.plot_images([Image.open(image) for image in images], captions, img_width=425, force_b64=True)" ] }, { "cell_type": "markdown", "metadata": { "id": "dQmxNGkXw-YN" }, "source": [ "Reviewing the captions, they are all generally in the right ballpark but far from perfect. The default model does a decent job but more robust models are necessary to fully deploy an image captioning model. " ] }, { "cell_type": "markdown", "metadata": { "id": "GxjHgXnz1MCD" }, "source": [ "# Create an objects instance\n", "\n", "The objects pipeline takes an image or list of images and generates a list of detected objects. This pipeline works using an object detection model." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "id": "unoteHQM1l6V" }, "outputs": [], "source": [ "%%capture\n", "\n", "from txtai.pipeline import Objects\n", "\n", "# Create objects pipeline\n", "objects = Objects()" ] }, { "cell_type": "markdown", "metadata": { "id": "HvZE-bww1v1k" }, "source": [ "# Detect objects\n", "\n", "The example below shows how to detect objects. A list of images are read from a directory, passed to an object detection model and detected objects are returned." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "id": "F_qrDbdv2IVu", "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "outputId": "18642f80-faeb-4fc7-f907-6acc6bdf32fc" }, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "\n", " \n", "
\n", " \n", " \n", " \n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {} }, { "output_type": "display_data", "data": { "text/html": [ "\n", " \n", "
\n", "
\n", "
\n", "

[('clock', 0.9837772846221924)]

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

[('vase', 0.9913519620895386)]

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

[('cell phone', 0.9672072529792786)]

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

[]

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

[]

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

[('motorcycle', 0.9990019202232361), ('person', 0.9853999018669128)]

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

[('bird', 0.9167556762695312)]

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

[]

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

[('book', 0.9250583648681641)]

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", " \n", "
\n", "
\n", "

[('umbrella', 0.9032363295555115)]

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
" ], "text/plain": [ "" ] }, "metadata": {} } ], "source": [ "import glob\n", "import ipyplot\n", "\n", "from PIL import Image\n", "\n", "# Get list of images\n", "images = glob.glob('txtai/*jpg')\n", "\n", "# Detect objects\n", "detected = objects(images)\n", "\n", "# Show image/objects pairs\n", "ipyplot.plot_images([Image.open(image) for image in images], detected, img_width=425, force_b64=True)" ] }, { "cell_type": "markdown", "metadata": { "id": "dEX-dbXE3U7W" }, "source": [ "Reviewing the detected objects, once again they are all generally in the right ballpark but far from perfect.\n", "\n", "This model or larger models may do well for a specific use cases in which the model has a high accuracy. For example, the results could be filtered on only accept certain types of objects, which have shown to have high accuracy." ] }, { "cell_type": "markdown", "metadata": { "id": "HeN8e1uy-icp" }, "source": [ "# Wrapping up\n", "\n", "This notebook introduced image captions and object detection. While the default models for both tasks aren't where we'd like them to be, they provide a good baseline to build on. For certain, targeted use cases where the models excel, they can be used now. This is a fast-evolving area and it is fully expected these models will improve!" ] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [], "name": "25 - Generate image captions and detect objects", "provenance": [] }, "kernelspec": { "display_name": "Python 3", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 0 }