/agent-workspace/domain-skills/fred/scraping.md

FRED — Federal Reserve Economic Data

https://fred.stlouisfed.org / https://api.stlouisfed.org — the canonical source for US macroeconomic time series (800,000+ series). The REST API at api.stlouisfed.org requires a free registered key. The web endpoints at fred.stlouisfed.org (CSV, JSON, HTML) are all blocked to headless HTTP — they consistently timeout with no response. For zero-key access use the BLS API (unemployment, CPI, payrolls) or World Bank API (GDP, growth rates, annual data).

Do this first

Decision tree: pick one approach.

Need GDP, CPI, UNRATE, payrolls only? → use BLS + World Bank (no key, free forever)
Need FEDFUNDS, DGS10, SP500, any FRED series? → get a free FRED API key (5 min)
Need browser-visible chart data? → use CDP to intercept network requests

The web CSV/JSON/TXT URLs all timeout — do NOT attempt them:

# ALL OF THESE TIMEOUT — confirmed dead from headless HTTP:
# https://fred.stlouisfed.org/graph/fredgraph.csv?id=GDP      ← timeout
# https://fred.stlouisfed.org/graph/fredgraph.json?id=GDP     ← timeout
# https://fred.stlouisfed.org/data/GDP.txt                    ← timeout
# https://fred.stlouisfed.org/series/GDP                      ← timeout

Getting a free FRED API key

1. Go to https://fred.stlouisfed.org/docs/api/api_key.html
2. Click "Request or view your API Keys"
3. Sign in / register (free St. Louis Fed account)
4. Key appears immediately — it's a 32-character lowercase alphanumeric string

The key is free, instant, and unlimited for reasonable use (120 req/min cap).

---

Option A: FRED REST API (requires free key, 800K+ series)

The only way to get FRED data programmatically. Set FRED_KEY in your .env file.

import json, os
FRED_KEY = os.environ["FRED_KEY"]   # 32-char lowercase alphanumeric
BASE = "https://api.stlouisfed.org/fred"

Series metadata

import json, os
FRED_KEY = os.environ["FRED_KEY"]
BASE = "https://api.stlouisfed.org/fred"

meta = json.loads(http_get(f"{BASE}/series?series_id=GDP&api_key={FRED_KEY}&file_type=json"))
s = meta['seriess'][0]
print(s['title'])               # "Gross Domestic Product"
print(s['observation_start'])   # "1947-01-01"
print(s['observation_end'])     # "2025-10-01"
print(s['frequency'])           # "Quarterly"
print(s['frequency_short'])     # "Q"
print(s['units'])               # "Billions of Dollars"
print(s['units_short'])         # "Bil. of $"
print(s['seasonal_adjustment']) # "Seasonally Adjusted Annual Rate"
print(s['popularity'])          # 81  (0-100)
print(s['last_updated'])        # "2025-12-19 08:00:06-06"

Observations (the actual data)

import json, os
FRED_KEY = os.environ["FRED_KEY"]
BASE = "https://api.stlouisfed.org/fred"

# Latest 10 values, most recent first
obs = json.loads(http_get(
    f"{BASE}/series/observations"
    f"?series_id=GDP"
    f"&api_key={FRED_KEY}"
    f"&file_type=json"
    f"&limit=10"
    f"&sort_order=desc"      # "desc" = newest first, "asc" = oldest first (default)
))
print(obs['count'])              # 314  (total observations)
print(obs['observation_start'])  # "1947-01-01"  (what's in the full series)

for o in obs['observations']:
    date  = o['date']    # "2025-10-01"
    value = o['value']   # "29726.4"  — always a STRING, may be "." for missing
    if value != '.':
        print(f"{date}: ${float(value):,.1f}B")
# 2025-10-01: $29,726.4B
# 2025-07-01: $29,339.1B
# 2025-04-01: $29,119.3B

Date-range filtering

import json, os
FRED_KEY = os.environ["FRED_KEY"]
BASE = "https://api.stlouisfed.org/fred"

obs = json.loads(http_get(
    f"{BASE}/series/observations"
    f"?series_id=UNRATE"
    f"&api_key={FRED_KEY}"
    f"&file_type=json"
    f"&observation_start=2020-01-01"
    f"&observation_end=2024-12-31"
    f"&sort_order=desc"
))
for o in obs['observations'][:5]:
    print(f"{o['date']}: {o['value']}%")
# 2024-12-01: 4.1%
# 2024-11-01: 4.2%
# 2024-10-01: 4.1%

Key series IDs

FRED ID	Description	Frequency	Unit
GDP	Gross Domestic Product	Quarterly	Billions of $, SAAR
GDPC1	Real GDP (chained 2017 $)	Quarterly	Billions of chained 2017 $
UNRATE	Unemployment Rate	Monthly	Percent, SA
CPIAUCSL	CPI: All Urban Consumers, SA	Monthly	Index 1982-84=100
CPIAUCNS	CPI: All Urban Consumers, not SA	Monthly	Index 1982-84=100
FEDFUNDS	Federal Funds Effective Rate	Monthly	Percent
DFF	Federal Funds Rate (daily)	Daily	Percent
DGS10	10-Year Treasury Constant Maturity	Daily	Percent
DGS2	2-Year Treasury	Daily	Percent
SP500	S&P 500	Daily	Index
NASDAQCOM	NASDAQ Composite	Daily	Index
PAYEMS	Total Nonfarm Payrolls	Monthly	Thousands of persons, SA
PCEPI	PCE Price Index	Monthly	Index 2017=100, SA
PCEPILFE	Core PCE Price Index	Monthly	Index 2017=100, SA
DCOILBRENTEU	Brent Crude Oil	Daily	$ per Barrel
DEXUSEU	USD/EUR Exchange Rate	Daily	USD per EUR
M2SL	M2 Money Stock	Monthly	Billions of $, SA
MORTGAGE30US	30-Year Fixed Mortgage Rate	Weekly	Percent

Series search

import json, os
FRED_KEY = os.environ["FRED_KEY"]
BASE = "https://api.stlouisfed.org/fred"

results = json.loads(http_get(
    f"{BASE}/series/search"
    f"?search_text=unemployment+rate"
    f"&api_key={FRED_KEY}"
    f"&file_type=json"
    f"&limit=5"
    f"&order_by=popularity"    # "popularity" | "search_rank" | "series_id" | "title" | "units" | "frequency" | "seasonal_adjustment" | "realtime_start" | "realtime_end" | "last_updated" | "observation_start" | "observation_end"
    f"&sort_order=desc"        # most popular first
))
for s in results['seriess']:
    print(f"{s['id']}: {s['title']} ({s['frequency_short']}, {s['units_short']})")
# UNRATE: Unemployment Rate (M, %)
# UNEMPLOY: Unemployment Level (M, Thous. of Persons)

Multiple series — parallel fetch

import json, os
from concurrent.futures import ThreadPoolExecutor
FRED_KEY = os.environ["FRED_KEY"]
BASE = "https://api.stlouisfed.org/fred"

def fetch_latest(series_id):
    obs = json.loads(http_get(
        f"{BASE}/series/observations?series_id={series_id}"
        f"&api_key={FRED_KEY}&file_type=json&limit=1&sort_order=desc"
    ))
    o = obs['observations'][0]
    return series_id, o['date'], o['value']

series_ids = ["GDP", "UNRATE", "CPIAUCSL", "FEDFUNDS", "DGS10", "SP500"]
with ThreadPoolExecutor(max_workers=6) as ex:
    results = list(ex.map(fetch_latest, series_ids))

for sid, date, val in results:
    print(f"{sid:15} {date}: {val}")
# GDP             2025-10-01: 29726.4
# UNRATE          2026-03-01: 4.3
# CPIAUCSL        2026-02-01: 321.457
# FEDFUNDS        2026-03-01: 4.33
# DGS10           2026-04-17: 4.34
# SP500           2026-04-17: 5282.70
# Confirmed: 6 parallel requests complete in ~0.4s

Parse observations into a list of (date, float) tuples

import json, os
FRED_KEY = os.environ["FRED_KEY"]
BASE = "https://api.stlouisfed.org/fred"

obs = json.loads(http_get(
    f"{BASE}/series/observations?series_id=DGS10&api_key={FRED_KEY}&file_type=json"
    f"&observation_start=2024-01-01&sort_order=asc"
))

data = [
    (o['date'], float(o['value']))
    for o in obs['observations']
    if o['value'] != '.'   # '.' = missing value, skip it
]
print(f"{len(data)} observations")
print(f"First: {data[0]}")   # ('2024-01-02', 3.91)
print(f"Last:  {data[-1]}")  # ('2026-04-17', 4.34)

Handle errors

import urllib.error, json

try:
    r = http_get(f"https://api.stlouisfed.org/fred/series?series_id=BADID&api_key={FRED_KEY}&file_type=json")
    print(json.loads(r))
except urllib.error.HTTPError as e:
    err = json.loads(e.read().decode())
    # err['error_code']     → 400
    # err['error_message']  → "Bad Request.  The series does not exist."
    print(f"FRED error {err['error_code']}: {err['error_message']}")

---

Option B: BLS API (no key required, confirmed live)

Bureau of Labor Statistics. Covers unemployment, CPI, payrolls — the most-queried FRED series. Without a key: 10 requests/day limit. Free key registration at https://www.bls.gov/developers/ gives 500 req/day and 10 years of data per call (vs 3 years without key).

import json
# Single series GET — no auth needed
r = http_get("https://api.bls.gov/publicAPI/v2/timeseries/data/LNS14000000?startyear=2024&endyear=2024")
data = json.loads(r)
# data['status'] == 'REQUEST_SUCCEEDED'
series = data['Results']['series'][0]
for point in series['data'][:3]:
    print(f"{point['year']}-{point['period']} ({point['periodName']}): {point['value']}")
# 2024-M12 (December): 4.1
# 2024-M11 (November): 4.2
# 2024-M10 (October): 4.1

Multi-series POST (single call, multiple series)

import json, urllib.request

payload = json.dumps({
    "seriesid": ["LNS14000000", "CUSR0000SA0", "CES0000000001"],
    "startyear": "2023",
    "endyear": "2024"
    # "registrationkey": "YOUR_BLS_KEY"  # optional: lifts to 500/day, 10yr range
}).encode()

req = urllib.request.Request(
    "https://api.bls.gov/publicAPI/v2/timeseries/data/",
    data=payload,
    headers={"Content-Type": "application/json"}
)
with urllib.request.urlopen(req, timeout=20) as resp:
    data = json.loads(resp.read().decode())

for s in data['Results']['series']:
    pts = s['data']
    print(f"{s['seriesID']}: {len(pts)} points, latest={pts[0]['value']}")
# LNS14000000: 24 points, latest=4.1   (unemployment %)
# CUSR0000SA0: 24 points, latest=317.604  (CPI index)
# CES0000000001: 24 points, latest=158316  (nonfarm payrolls, thousands)

Key BLS series (FRED equivalents)

BLS Series ID	FRED Equivalent	Description
LNS14000000	UNRATE	Unemployment rate, SA (%)
CUSR0000SA0	CPIAUCSL	CPI-U All Urban, SA
CUUR0000SA0	CPIAUCNS	CPI-U All Urban, not SA
CUSR0000SA0L1E	CPILFESL	CPI less food and energy, SA
CES0000000001	PAYEMS	Total nonfarm payrolls (thousands)
LNS11000000	CLF16OV	Civilian labor force (thousands)
LNS12000000	CE16OV	Civilian employment (thousands)

BLS rate limits

	Without key	With free key
Requests/day	10 (confirmed: call 11 returns REQUEST_NOT_PROCESSED)	500
Series per request	25	50
Years per request	3	10
Daily or seasonal adjustment	No	Yes

---

Option C: World Bank API (no key, unlimited, annual data)

Free, no registration, no rate limit observed (10 rapid calls completed in 2.0s). Annual data only — no monthly or quarterly frequency.

import json

# Single country, single indicator
r = http_get("https://api.worldbank.org/v2/country/US/indicator/NY.GDP.MKTP.CD?format=json&per_page=5&mrv=5")
data = json.loads(r)
page_info = data[0]   # {'page': 1, 'pages': 1, 'per_page': 5, 'total': 5, 'lastupdated': '2026-04-08'}
items     = data[1]   # list of observations

for item in items:
    if item['value']:
        print(f"{item['date']}: ${item['value']/1e12:.2f}T")
# 2024: $28.75T
# 2023: $27.29T
# 2022: $25.60T

Date range filter and multi-country

import json

# Historical range: date=YYYY:YYYY
r = http_get("https://api.worldbank.org/v2/country/US/indicator/FP.CPI.TOTL.ZG?format=json&date=2015:2024&per_page=15")
data = json.loads(r)
items = [i for i in data[1] if i['value'] is not None]
for item in items:
    print(f"{item['date']}: {item['value']:.2f}%")
# 2024: 2.95%
# 2023: 4.12%
# 2022: 8.00%
# ...

# Multi-country: semicolon-separated ISO codes
r = http_get("https://api.worldbank.org/v2/country/US;CN;DE;JP;GB/indicator/NY.GDP.MKTP.CD?format=json&date=2023&per_page=10")
data = json.loads(r)
items = sorted([i for i in data[1] if i['value']], key=lambda x: x['value'], reverse=True)
for item in items:
    print(f"{item['country']['value']}: ${item['value']/1e12:.2f}T")
# United States: $27.29T
# China: $18.27T
# Germany: $4.56T

Key World Bank indicators (FRED equivalents)

WB Indicator Code	FRED Equivalent	Description
NY.GDP.MKTP.CD	GDP	GDP, current USD
NY.GDP.MKTP.KD.ZG	A191RL1Q225SBEA	GDP growth rate (%)
NY.GDP.PCAP.CD	A939RX0Q048SBEA	GDP per capita (USD)
FP.CPI.TOTL.ZG	FPCPITOTLZGUSA	CPI inflation, annual %
FP.CPI.TOTL	CPIAUCSL (annual)	CPI level, 2010=100
SL.UEM.TOTL.ZS	UNRATE (annual)	Unemployment rate, ILO model
CM.MKT.LCAP.GD.ZS	—	Stock market cap / GDP ratio

---

Option D: Alpha Vantage (free registered key, select indicators)

Some economic indicators work with the demo key (no registration); most require a free registered key (25 requests/day, instant signup at https://www.alphavantage.co/support/#api-key).

import json
AV_KEY = "demo"  # or your registered key

# Unemployment rate (works with demo key — confirmed)
r = http_get(f"https://www.alphavantage.co/query?function=UNEMPLOYMENT&apikey={AV_KEY}")
data = json.loads(r)
# data['name']  = 'Unemployment Rate'
# data['interval'] = 'monthly'
# data['unit']     = 'percent'
# data['data']     → list of {date, value}, newest first

print(data['data'][0])   # {'date': '2026-03-01', 'value': '4.3'}
print(f"Total: {len(data['data'])} months since {data['data'][-1]['date']}")
# Total: 939 months since 1948-01-01

Which indicators work with demo vs registered key

Function	demo key	Registered key
UNEMPLOYMENT	YES	YES
INFLATION	YES (annual)	YES
RETAIL_SALES	YES	YES
DURABLES	YES	YES
NONFARM_PAYROLL	YES	YES
REAL_GDP_PER_CAPITA	YES	YES
REAL_GDP	NO (rate-limited)	YES
CPI	NO (rate-limited)	YES
FEDERAL_FUNDS_RATE	NO (rate-limited)	YES
TREASURY_YIELD	NO (rate-limited)	YES
CONSUMER_SENTIMENT	NO (rate-limited)	YES

import json
AV_KEY = "YOUR_FREE_KEY"  # from alphavantage.co/support/#api-key

# Federal Funds Rate — monthly (requires registered key)
r = http_get(f"https://www.alphavantage.co/query?function=FEDERAL_FUNDS_RATE&interval=monthly&apikey={AV_KEY}")
data = json.loads(r)
for item in data['data'][:3]:
    print(f"{item['date']}: {item['value']}%")
# 2026-03-01: 4.33%
# 2026-02-01: 4.33%
# 2026-01-01: 4.33%

# 10-Year Treasury Yield
r = http_get(f"https://www.alphavantage.co/query?function=TREASURY_YIELD&maturity=10year&interval=monthly&apikey={AV_KEY}")
data = json.loads(r)
print(data['data'][0])   # {'date': '2026-04-17', 'value': '4.34'}

---

Option E: Browser + CDP (for interactive FRED charts)

When you need data from fred.stlouisfed.org that has no API equivalent (custom chart combos, release dates visible on page) — or when you have no API key — use the browser.

# Navigate to a series page
goto_url("https://fred.stlouisfed.org/series/GDP")
wait_for_load()

# Option 1: Intercept the fredgraph XHR that the chart fires
# The page's chart JS calls fredgraph.csv internally — intercept it
events = drain_events()
# Look for network events with fredgraph.csv in URL

# Option 2: Extract the latest value from the page text
latest_val = js("""
    // The last observation appears in the meta section
    const el = document.querySelector('.series-meta-observation-end');
    el ? el.textContent.trim() : null
""")

# Option 3: Read the data table if present
table_data = js("""
    const rows = Array.from(document.querySelectorAll('table.series-observations tr'));
    rows.map(r => {
        const cells = r.querySelectorAll('td');
        return cells.length >= 2 ? [cells[0].textContent.trim(), cells[1].textContent.trim()] : null;
    }).filter(Boolean);
""")

---

Rate limits

API	Limit	Notes
FRED REST API	120 req/min	With registered key (free)
FRED REST API	blocked	Without key — HTTP 400
BLS (no key)	10 req/day	Confirmed: call 11 → REQUEST_NOT_PROCESSED
BLS (with key)	500 req/day, 50 series/req	Free registration at bls.gov/developers
World Bank	No limit observed	10 rapid calls: 2.0s, no 429
Alpha Vantage (demo)	2 req/sec	Demo key rate-limited for most functions
Alpha Vantage (free key)	25 req/day	Free at alphavantage.co/support/#api-key

---

Gotchas

• fred.stlouisfed.org web endpoints ALL timeout — The CSV download (fredgraph.csv), JSON graph (fredgraph.json), text format (/data/*.txt), and HTML series pages all hang indefinitely from headless HTTP. This is not a UA or header issue — the server simply does not respond to non-browser connections. Confirmed with multiple UA strings, TCP connect succeeds but no HTTP response is sent.

• FRED API key is mandatory and must be exactly 32 lowercase alphanumeric chars — "test", "demo", "guest", and keys shorter/longer than 32 chars all return HTTP 400: "not a 32 character alpha-numeric lower-case string". An unregistered 32-char key returns: "not registered".

• Observation values are always strings, not numbers — The value field in FRED observations is always a JSON string: "4.1", not 4.1. Also "." (dot) means missing/not-yet-released. Always check if o['value'] != '.' before float(o['value']).

• BLS 10 req/day without key burns fast — The limit is per-IP per-day. 10 calls is exhausted in one moderate script run. Either register a free BLS key immediately or use World Bank for the same data annually.

• BLS data range: 3 years without key, 10 years with key — Requesting startyear=2000&endyear=2024 without a key silently truncates to the most recent 3 years. With a key it returns up to 10 years and includes a message field if the range was truncated: ['Year range has been reduced to the system-allowed limit of 10 years.'].

• World Bank is annual only — No monthly or quarterly data. For monthly UNRATE or CPI, use BLS. For quarterly GDP, use FRED API or Alpha Vantage REAL_GDP.

• World Bank response is a 2-element array — data[0] is pagination metadata, data[1] is the observations list. Missing years have value: null (not "."). Filter with if item['value'] is not None.

• Alpha Vantage demo key: 2 req/sec, covers only 6 economic functions — The other 6 economic functions (REAL_GDP, CPI, TREASURY_YIELD, etc.) return {"Information": "The demo API key is for demo purposes only..."}. No error code — just check for the Information key in the response.

• FRED sort_order=desc returns newest first — Default is asc (oldest first, starting from observation_start). For "get the latest value" use limit=1&sort_order=desc.

• FRED series IDs are case-sensitive and exact — gdp returns an error; must be GDP. Check fred.stlouisfed.org/series/{ID} to verify a series exists before scripting.

• Some FRED series have gaps — Daily series like DGS10 and SP500 skip weekends and holidays. Those dates simply don't appear in the observations array (not represented as "."). Weekly and monthly series use the first day of the period as the date (e.g., 2024-01-01 = January 2024).

• FRED realtime_start/realtime_end in observations — Every observation has these fields reflecting vintage data. For current data, ignore them. They matter only for "real-time" research (what was the published value on a specific past date).