# YouTube — Data Extraction

Field-tested against youtube.com on 2026-04-21.
No authentication required for any approach documented here.

---

## Approach 1 (Fastest): oEmbed API — No Auth, No Browser

`https://www.youtube.com/oembed?url=https://www.youtube.com/watch?v={VIDEO_ID}&format=json`

Returns JSON in ~0.3s. Works for any public video. Does **not** require login.

```python
from helpers import http_get
import json

def youtube_oembed(video_id):
    """Fetch oEmbed metadata for a YouTube video.

    Returns title, author, thumbnail URL, and embed iframe HTML.
    """
    url = f"https://www.youtube.com/oembed?url=https://www.youtube.com/watch?v={video_id}&format=json"
    return json.loads(http_get(url))

data = youtube_oembed("dQw4w9WgXcQ")
# {
#   "title":            "Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)",
#   "author_name":      "Rick Astley",
#   "author_url":       "https://www.youtube.com/@RickAstleyYT",
#   "type":             "video",
#   "thumbnail_url":    "https://i.ytimg.com/vi/dQw4w9WgXcQ/hqdefault.jpg",
#   "thumbnail_width":  480,
#   "thumbnail_height": 360,
#   "width":            200,
#   "height":           113,
#   "version":          "1.0",
#   "provider_name":    "YouTube",
#   "html":             '<iframe width="200" height="113" src="https://www.youtube.com/embed/dQw4w9WgXcQ?feature=oembed" ...>'
# }
```

### Bulk oEmbed (ThreadPoolExecutor)

```python
from concurrent.futures import ThreadPoolExecutor
import json
from helpers import http_get

video_ids = ["dQw4w9WgXcQ", "jNQXAC9IVRw", "9bZkp7q19f0"]

def fetch_oembed(vid):
    url = f"https://www.youtube.com/oembed?url=https://www.youtube.com/watch?v={vid}&format=json"
    try:
        return json.loads(http_get(url))
    except Exception as e:
        return {"error": str(e), "id": vid}

with ThreadPoolExecutor(max_workers=5) as ex:
    results = list(ex.map(fetch_oembed, video_ids))
# 3 videos: ~4.1s total (YouTube oEmbed is slower than Spotify's; don't use >5 workers)
```

---

## Approach 2: Watch Page — Full Metadata via ytInitialPlayerResponse

Every `youtube.com/watch?v={ID}` page embeds two JSON blobs in the HTML:

- `ytInitialPlayerResponse` — video details, microformat, caption track list
- `ytInitialData` — comments section structure, related videos

### Extract all video metadata

```python
from helpers import http_get
import json, re

def scrape_video(video_id):
    html = http_get(f"https://www.youtube.com/watch?v={video_id}")

    # ---- ytInitialPlayerResponse ----
    m = re.search(r'var ytInitialPlayerResponse = (\{.*?\});(?:var|</script>)', html, re.DOTALL)
    if not m:
        raise ValueError(f"ytInitialPlayerResponse not found for video {video_id} — video may be private, deleted, or region-blocked")
    pr = json.loads(m.group(1))

    # Check playability before parsing
    status = pr.get("playabilityStatus", {}).get("status")
    if status == "LOGIN_REQUIRED":
        raise ValueError(f"Video {video_id} is age-restricted or login-gated (playabilityStatus: LOGIN_REQUIRED)")
    if status == "ERROR":
        reason = pr.get("playabilityStatus", {}).get("reason", "unknown")
        raise ValueError(f"Video {video_id} is unavailable: {reason}")

    vd  = pr["videoDetails"]
    mf  = pr["microformat"]["playerMicroformatRenderer"]
    caps = pr.get("captions", {}) \
             .get("playerCaptionsTracklistRenderer", {}) \
             .get("captionTracks", [])

    return {
        # Core
        "video_id":      vd["videoId"],
        "title":         vd["title"],
        "author":        vd["author"],
        "channel_id":    vd["channelId"],
        "description":   vd["shortDescription"],
        "duration_s":    int(vd["lengthSeconds"]),
        "view_count":    int(vd["viewCount"]),
        "keywords":      vd.get("keywords", []),
        "is_live":       vd.get("isLiveContent", False),
        "is_private":    vd.get("isPrivate", False),
        # Microformat (richer publishing data)
        "publish_date":  mf.get("publishDate"),   # ISO 8601, e.g. "2009-10-25T00:57:33-07:00"
        "upload_date":   mf.get("uploadDate"),
        "category":      mf.get("category"),       # e.g. "Music", "Gaming", "Education"
        "like_count":    int(mf.get("likeCount", 0)),
        "is_family_safe": mf.get("isFamilySafe"),
        "is_unlisted":   mf.get("isUnlisted"),
        "available_countries": mf.get("availableCountries", []),  # list of ISO 3166-1 alpha-2 codes
        "channel_name":  mf.get("ownerChannelName"),
        "channel_url":   mf.get("ownerProfileUrl"),
        "embed_url":     mf.get("embed", {}).get("iframeUrl"),
        # Thumbnails (all publicly accessible, no auth)
        "thumbnail_hq":  f"https://i.ytimg.com/vi/{video_id}/hqdefault.jpg",    # 480×360, always exists
        "thumbnail_max": f"https://i.ytimg.com/vi/{video_id}/maxresdefault.jpg", # 1280×720, may 404
        # Caption tracks — baseUrl is included for reference but returns empty in practice;
        # use the Show Transcript UI flow in the browser instead (see playback.md)
        "caption_tracks": [
            {
                "lang":     t.get("languageCode"),
                "name":     t.get("name", {}).get("simpleText"),
                "kind":     t.get("kind", "manual"),  # "manual" or "asr" (auto-generated)
                "base_url": t.get("baseUrl"),
            }
            for t in caps
        ],
    }

video = scrape_video("dQw4w9WgXcQ")
# {
#   "video_id":     "dQw4w9WgXcQ",
#   "title":        "Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)",
#   "author":       "Rick Astley",
#   "channel_id":   "UCuAXFkgsw1L7xaCfnd5JJOw",
#   "duration_s":   213,
#   "view_count":   1764468859,
#   "publish_date": "2009-10-24T23:57:33-07:00",
#   "category":     "Music",
#   "like_count":   16942558,
#   "caption_tracks": [
#     {"lang": "en",    "name": "English",                "kind": "manual"},
#     {"lang": "en",    "name": "English (auto-generated)", "kind": "asr"},
#     {"lang": "de-DE", "name": "German (Germany)",        "kind": "manual"},
#     ...
#   ],
# }
```

---

## Approach 3: Search Results — No Auth

`youtube.com/results?search_query={QUERY}` is server-side rendered. The `ytInitialData` blob contains up to ~20 video results.

```python
from helpers import http_get
import json, re
from urllib.parse import quote_plus

def youtube_search(query, max_results=20):
    """Search YouTube videos without a browser or API key."""
    url = f"https://www.youtube.com/results?search_query={quote_plus(query)}"
    html = http_get(url)

    m = re.search(r'var ytInitialData = (\{.*?\});</script>', html, re.DOTALL)
    data = json.loads(m.group(1))

    # Walk the nested structure to find videoRenderer items
    section_contents = (
        data.get("contents", {})
            .get("twoColumnSearchResultsRenderer", {})
            .get("primaryContents", {})
            .get("sectionListRenderer", {})
            .get("contents", [])
    )

    results = []
    for section in section_contents:
        for item in section.get("itemSectionRenderer", {}).get("contents", []):
            vr = item.get("videoRenderer", {})
            if not vr:
                continue
            snippet = vr.get("detailedMetadataSnippets", [])
            desc = (
                "".join(r.get("text", "") for r in snippet[0]["snippetText"]["runs"])
                if snippet else None
            )
            results.append({
                "video_id":   vr["videoId"],
                "url":        f"https://www.youtube.com/watch?v={vr['videoId']}",
                "title":      vr.get("title", {}).get("runs", [{}])[0].get("text"),
                "channel":    vr.get("ownerText", {}).get("runs", [{}])[0].get("text"),
                "channel_url": (
                    "https://www.youtube.com"
                    + vr.get("ownerText", {}).get("runs", [{}])[0]
                                              .get("navigationEndpoint", {})
                                              .get("browseEndpoint", {})
                                              .get("canonicalBaseUrl", "")
                ),
                "duration":   vr.get("lengthText", {}).get("simpleText"),  # e.g. "3:32"
                "views":      vr.get("viewCountText", {}).get("simpleText"),  # e.g. "1,764,468,859 views"
                "published":  vr.get("publishedTimeText", {}).get("simpleText"),  # e.g. "7 years ago"
                "description_snippet": desc,
                "thumbnail":  f"https://i.ytimg.com/vi/{vr['videoId']}/hqdefault.jpg",
            })
            if len(results) >= max_results:
                return results  # exit both loops immediately
    return results

results = youtube_search("python tutorial", max_results=5)
# Returns up to ~14-20 results (YouTube serves fewer than 20 on first page)
# [
#   {
#     "video_id":  "K5KVEU3aaeQ",
#     "title":     "Python Full Course for Beginners",
#     "channel":   "Programming with Mosh",
#     "duration":  "2:02:21",
#     "views":     "6,056,121 views",
#     "published": "1 year ago",
#   }, ...
# ]
```

---

## Approach 4: Channel Metadata — No Auth

Channel pages (`youtube.com/@handle` or `youtube.com/channel/{CHANNEL_ID}`) embed metadata in `ytInitialData`.

```python
from helpers import http_get
import json, re

def scrape_channel(handle_or_id):
    """
    handle_or_id: "@RickAstleyYT"           (handle, with @)
                  "UCuAXFkgsw1L7xaCfnd5JJOw" (channel ID)
    """
    if handle_or_id.startswith("UC"):
        url = f"https://www.youtube.com/channel/{handle_or_id}"
    else:
        url = f"https://www.youtube.com/{handle_or_id}"

    html = http_get(url)
    m = re.search(r'var ytInitialData = (\{.*?\});</script>', html, re.DOTALL)
    data = json.loads(m.group(1))

    # Canonical metadata (always present)
    cmd = data.get("metadata", {}).get("channelMetadataRenderer", {})

    # Subscriber count + handle from pageHeaderViewModel
    ph = (
        data.get("header", {})
            .get("pageHeaderRenderer", {})
            .get("content", {})
            .get("pageHeaderViewModel", {})
    )
    meta_parts = [
        part.get("text", {}).get("content", "")
        for row in ph.get("metadata", {})
                     .get("contentMetadataViewModel", {})
                     .get("metadataRows", [])
        for part in row.get("metadataParts", [])
    ]
    # meta_parts is typically: ["@handle", "4.48m subscribers", "N videos"]

    # Avatar (take the largest source)
    avatar_sources = (
        ph.get("image", {})
          .get("decoratedAvatarViewModel", {})
          .get("avatar", {})
          .get("avatarViewModel", {})
          .get("image", {})
          .get("sources", [])
    )
    avatar_url = avatar_sources[-1]["url"] if avatar_sources else None

    # Channel banner
    banner_sources = (
        ph.get("banner", {})
          .get("imageBannerViewModel", {})
          .get("image", {})
          .get("sources", [])
    )
    banner_url = banner_sources[-1]["url"] if banner_sources else None

    return {
        "channel_id":  cmd.get("externalId"),
        "title":       cmd.get("title"),
        "description": cmd.get("description"),
        "channel_url": cmd.get("channelUrl"),
        "keywords":    cmd.get("keywords", ""),
        "handle":      meta_parts[0] if len(meta_parts) > 0 else None,
        "subscribers": meta_parts[1] if len(meta_parts) > 1 else None,  # e.g. "4.48m subscribers"
        "avatar_url":  avatar_url,
        "banner_url":  banner_url,
    }

channel = scrape_channel("@RickAstleyYT")
# {
#   "channel_id":  "UCuAXFkgsw1L7xaCfnd5JJOw",
#   "title":       "Rick Astley",
#   "description": "2026 UK & Ireland Reflection Tour...",
#   "channel_url": "https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw",
#   "handle":      "@RickAstleyYT",
#   "subscribers": "4.48m subscribers",
#   "avatar_url":  "https://yt3.googleusercontent.com/...",
#   "banner_url":  "https://yt3.googleusercontent.com/...",
# }
```

---

## Thumbnail URLs — All Sizes

All thumbnail sizes are publicly accessible without auth. Construct directly from `video_id`:

```python
def thumbnail_urls(video_id):
    base = f"https://i.ytimg.com/vi/{video_id}"
    return {
        "default":  f"{base}/default.jpg",      # 120×90,  always exists
        "medium":   f"{base}/mqdefault.jpg",     # 320×180, always exists
        "high":     f"{base}/hqdefault.jpg",     # 480×360, always exists
        "standard": f"{base}/sddefault.jpg",     # 640×480, always exists
        "maxres":   f"{base}/maxresdefault.jpg", # 1280×720, may 404 on older/low-res videos
    }
```

---

## Extract Video ID from Any URL

```python
import re

def extract_video_id(url):
    """Extract YouTube video ID (11-char) from any YouTube URL format."""
    m = re.search(r'(?:v=|/v/|youtu\.be/|/embed/|/shorts/)([A-Za-z0-9_-]{11})', url)
    return m.group(1) if m else None

extract_video_id("https://www.youtube.com/watch?v=dQw4w9WgXcQ")  # "dQw4w9WgXcQ"
extract_video_id("https://youtu.be/dQw4w9WgXcQ")                  # "dQw4w9WgXcQ"
extract_video_id("https://www.youtube.com/shorts/dQw4w9WgXcQ")    # "dQw4w9WgXcQ"
extract_video_id("https://www.youtube.com/embed/dQw4w9WgXcQ")     # "dQw4w9WgXcQ"
```

---

## What Requires a Browser

The following are **not accessible** via `http_get` and require the CDP browser (see `playback.md`):

- **Trending / Explore** (`/feed/trending`) — `ytInitialData` loads but video items are empty without cookies
- **Playlist contents** — `ytInitialData` returns only microformat; full video list requires session cookies
- **Comments** — loaded lazily via XHR, not present in initial HTML
- **Shorts feed** — requires JS hydration
- **Channel Videos tab** — video list requires cookies for consistent results
- **Caption text content** — `captionTracks[].baseUrl` URLs return empty bytes regardless of session state; use the browser's Show Transcript UI flow instead (see `playback.md`)
- **Age-restricted videos** — oEmbed returns HTTP 401; `scrape_video()` raises `ValueError("LOGIN_REQUIRED")`

### Watch-page DOM hydration — the wait you need

When you do fall through to the browser for watch-page DOM (e.g. because you need a
rendered UI state, not just metadata), `wait_for_load()` is **not** enough. The `load`
event fires before YouTube's Polymer components hydrate — `h1.ytd-watch-metadata yt-formatted-string`,
`ytd-video-owner-renderer #channel-name a`, and `ytd-watch-info-text` all return `null` for
~2s after load. Add a `wait(3)` after `wait_for_load()` before querying any watch-page
selector.

Field-tested 2026-04-24 on Brave; same behavior observed on ungoogled-chromium. Prefer
the `http_get` + `ytInitialPlayerResponse` path above — the browser path exists for flows
that need live UI state, not for reading metadata.

---

## URL Patterns

| Resource       | URL pattern                                                        |
|----------------|--------------------------------------------------------------------|
| Video          | `https://www.youtube.com/watch?v={VIDEO_ID}`                       |
| Short URL      | `https://youtu.be/{VIDEO_ID}`                                      |
| Shorts         | `https://www.youtube.com/shorts/{VIDEO_ID}`                        |
| Channel handle | `https://www.youtube.com/@{HANDLE}`                                |
| Channel ID     | `https://www.youtube.com/channel/{CHANNEL_ID}`                     |
| Playlist       | `https://www.youtube.com/playlist?list={PLAYLIST_ID}`              |
| Search         | `https://www.youtube.com/results?search_query={QUERY}`             |
| oEmbed         | `https://www.youtube.com/oembed?url={VIDEO_URL}&format=json`       |
| Thumbnail (HQ) | `https://i.ytimg.com/vi/{VIDEO_ID}/hqdefault.jpg`                  |

---

## Gotchas

- **`ytInitialPlayerResponse` regex must use non-greedy match with lookahead** — `(\{.*?\});(?:var|</script>)` with `re.DOTALL` is reliable. Do not use `\{.*\}` (greedy) — it consumes the entire rest of the page.
- **`viewCount` and `lengthSeconds` are strings, not ints** — `vd["viewCount"]` returns `"1764468859"`. Always cast with `int()`.
- **`likeCount` lives in `microformat`, not `videoDetails`** — `videoDetails` does not expose like count. `microformat.playerMicroformatRenderer.likeCount` is a string integer.
- **`availableCountries` is a list of ISO 3166-1 alpha-2 codes** — 249 entries for globally available videos. An empty list means region data is unavailable, not that the video is globally blocked.
- **oEmbed thumbnail is always `hqdefault` (480×360)** — if you need 1280×720, construct the `maxresdefault.jpg` URL directly, but check for 404 on older videos.
- **Search returns ~14–20 results** — YouTube does not guarantee 20 results. Always iterate `itemSectionRenderer.contents` rather than assuming a fixed count.
- **Channel subscriber count is a rounded string** — `"4.48m subscribers"`, not an integer. Parse with regex if sorting: `re.search(r'([\d.]+)\s*([km]?)', text, re.I)`.
- **`meta_parts` order is `[handle, subscribers, video_count]`** — the third element is not always present. Index defensively.
- **Caption `baseUrl` is not fetchable** — `captionTracks[].baseUrl` contains `expire=` and `signature=` params but returns empty bytes in all tested conditions (plain `http_get`, XHR from within the page, and `fetch()` with cookies). Use the Show Transcript UI in the browser for caption text (see `playback.md`).
- **Age-restricted videos** — `scrape_video()` raises `ValueError` with a `LOGIN_REQUIRED` message. `oEmbed` returns HTTP 401 (raises `urllib.error.HTTPError`). Neither approach can access age-restricted content without login.
- **Private / deleted videos** — oEmbed returns HTTP 404 (raises `urllib.error.HTTPError`). Wrap in `try/except`.
- **`ytInitialData` blob terminator is `;</script>`** — using `re.DOTALL` with `(\{.*?\});</script>` is safe; the blob does not contain `;</script>` internally.