# Open Library — Book Data Extraction

`https://openlibrary.org` — Internet Archive's free book catalog. All endpoints are public JSON APIs — no auth, no browser, no scraping required.

## Do this first

**Every task is a direct HTTP call — never open the browser.**

```python
import json
from helpers import http_get

# Search by title
results = json.loads(http_get("https://openlibrary.org/search.json?q=dune&limit=5"))
# results['numFound']  == 49090
# results['docs']      == list of work objects
# results['start']     == 0  (offset for pagination)
```

The search API is your entry point for everything. It returns work-level records (all editions grouped). To get edition details, follow the `key` to the Works or Books API.

---

## Common workflows

### Search by query, author, title, or ISBN

```python
import json
from helpers import http_get

# Free-text search
r = json.loads(http_get("https://openlibrary.org/search.json?q=dune+frank+herbert&limit=5"))

# Author search
r = json.loads(http_get(
    "https://openlibrary.org/search.json?author=tolkien&limit=5"
    "&fields=title,author_name,first_publish_year,isbn"
))
# fields=* returns all available fields; default returns ~15

# Title + author combined
r = json.loads(http_get(
    "https://openlibrary.org/search.json?title=dune&author=frank+herbert&limit=3"
    "&fields=title,author_name,edition_count,first_publish_year"
))
# r['docs'][0]['title']              == 'Dune'
# r['docs'][0]['author_name']        == ['Frank Herbert']
# r['docs'][0]['first_publish_year'] == 1965
# r['docs'][0]['edition_count']      == 120

# ISBN lookup (returns 0–2 results for the same work)
r = json.loads(http_get("https://openlibrary.org/search.json?isbn=9780743273565"))
# r['numFound']            == 2
# r['docs'][0]['title']    == 'The Great Gatsby'
# r['docs'][0]['key']      == '/works/OL468431W'
```

**Sort options** (`&sort=`): `new` (recently added), `old`, `random`, `editions` (most editions), `scans` (most scans). Default is relevance.

**Language filter**: `&language=fre` (ISO 639-2/B codes: `eng`, `fre`, `ger`, `spa`, `ita`, etc.)

**Pagination**: `&limit=N&offset=N`. Max limit not enforced but keep under 100 for reliability.

#### Search doc fields (default — ~15 keys always present)

| Field | Type | Notes |
|---|---|---|
| `key` | str | `/works/OL893415W` — use for Works API |
| `title` | str | Work title |
| `author_name` | list[str] | e.g. `['Frank Herbert']` |
| `author_key` | list[str] | e.g. `['OL79034A']` |
| `first_publish_year` | int | |
| `edition_count` | int | Number of editions across all languages |
| `cover_i` | int | Cover image ID — use with covers API |
| `cover_edition_key` | str | e.g. `OL7353617M` |
| `language` | list[str] | ISO codes of all editions |
| `ia` | list[str] | Internet Archive identifiers (when has_fulltext=true) |
| `ebook_access` | str | `'public'`, `'borrowable'`, `'no_ebook'` |
| `has_fulltext` | bool | |

#### Extra fields with `&fields=*`

```python
# With fields=* you also get:
# 'isbn'                   list[str]   All ISBNs across editions
# 'publisher'              list[str]   All publishers ever
# 'publish_date'           list[str]   All publish dates (strings, inconsistent formats)
# 'publish_year'           list[int]   Parsed years
# 'subject'                list[str]   Subject headings
# 'person'                 list[str]   Subject persons (e.g. 'Big Brother')
# 'place'                  list[str]   Subject places
# 'time'                   list[str]   Subject times
# 'number_of_pages_median' int         Median page count across editions
# 'ratings_average'        float       e.g. 4.29
# 'ratings_count'          int
# 'want_to_read_count'     int
# 'already_read_count'     int
# 'currently_reading_count'int
# 'readinglog_count'       int         Total of all reading log entries
# 'first_sentence'         list[str]
# 'id_goodreads'           list[str]
# 'id_librarything'        list[str]
# 'id_wikidata'            list[str]
# 'ddc'                    list[str]   Dewey Decimal Classification
# 'lcc'                    list[str]   Library of Congress Classification
# 'lccn'                   list[str]
```

---

### Bulk ISBN lookups (parallel)

```python
import json
from helpers import http_get
from concurrent.futures import ThreadPoolExecutor

isbns = ['9780743273565', '9780451524935', '9780618346257']

def lookup_isbn(isbn):
    url = f"https://openlibrary.org/search.json?isbn={isbn}&fields=title,author_name,first_publish_year,key"
    r = json.loads(http_get(url))
    if r['docs']:
        d = r['docs'][0]
        return {'isbn': isbn, 'title': d.get('title'), 'author': d.get('author_name', [None])[0],
                'year': d.get('first_publish_year'), 'key': d.get('key')}
    return {'isbn': isbn, 'found': False}

with ThreadPoolExecutor(max_workers=5) as ex:
    books = list(ex.map(lookup_isbn, isbns))

# [{'isbn': '9780743273565', 'title': 'The Great Gatsby', 'author': 'F. Scott Fitzgerald', 'year': 1920, ...},
#  {'isbn': '9780451524935', 'title': 'Nineteen Eighty-Four', 'author': 'George Orwell',      'year': 1949, ...},
#  {'isbn': '9780618346257', 'title': 'The Fellowship of the Ring', 'author': 'J.R.R. Tolkien', 'year': 1954, ...}]
```

---

### Works API (editions grouped by title)

Returns all metadata for a work (all editions combined). Get the work ID from `key` in search results.

```python
import json
from helpers import http_get

work_id = 'OL893415W'  # from search doc['key'] = '/works/OL893415W'
work = json.loads(http_get(f"https://openlibrary.org/works/{work_id}.json"))

# work['title']           == 'Dune'
# work['key']             == '/works/OL893415W'
# work['covers']          == [11481354, 12375564, 11157826]  ← cover IDs for covers API
# work['subjects']        == ['Dune (Imaginary place)', 'Fiction', ...]
# work['subject_places']  == [...]   ← geographic subjects (may be absent)
# work['subject_people']  == [...]   ← person subjects (may be absent)
# work['subject_times']   == [...]   ← time subjects (may be absent)
# work['authors']         == [{'author': {'key': '/authors/OL79034A'}, 'type': {...}}]
# work['description']     → either str OR {'type': '/type/text', 'value': str}  ← see gotchas
# work['created']         == {'type': '/type/datetime', 'value': '2009-10-15T11:34:21.437031'}
# work['last_modified']   same shape as created
```

Helper for the description field (which has two possible shapes):

```python
def get_description(work: dict) -> str:
    desc = work.get('description', '')
    if isinstance(desc, dict):
        return desc.get('value', '')
    return desc or ''
```

#### Works editions (paginated list of all editions)

```python
editions_resp = json.loads(http_get(
    f"https://openlibrary.org/works/{work_id}/editions.json?limit=10&offset=0"
))
# editions_resp['size']    == 120      (total edition count)
# editions_resp['entries'] == [...]    (up to limit items)
# editions_resp['links']   == {'self': '...', 'work': '...', 'next': '...', 'prev': '...'}
# ← use links['next'] for pagination when offset+limit < size

e = editions_resp['entries'][0]
# e['title']           == 'Duna'
# e['publishers']      == ['Editora Aleph']
# e['publish_date']    == '19/08/2017'   ← inconsistent format, string
# e['isbn_13']         == ['9788576573135']
# e['isbn_10']         == ['857657313X']
# e['covers']          == [10368109]
# e['number_of_pages'] == 680
# e['languages']       == [{'key': '/languages/por'}]
# e['key']             == '/books/OL28969075M'
# e['physical_format'] == 'Paperback'   (often missing)
# e['notes']           → str or {'value': str}  (often missing)
```

---

### Books API (specific edition)

Two sub-APIs: direct JSON for raw data, or `api/books` for enriched data.

#### Direct edition JSON

```python
import json
from helpers import http_get

edition_id = 'OL7353617M'  # from editions list e['key'] or cover_edition_key in search
edition = json.loads(http_get(f"https://openlibrary.org/books/{edition_id}.json"))

# edition['title']           == 'Fantastic Mr. Fox'
# edition['publishers']      == ['Puffin']
# edition['publish_date']    == 'October 1, 1988'
# edition['isbn_13']         == ['9780140328721']
# edition['isbn_10']         == ['0140328726']
# edition['number_of_pages'] == 96
# edition['covers']          == [...]   ← cover IDs
# edition['languages']       == [{'key': '/languages/eng'}]
# edition['works']           == [{'key': '/works/OL45804W'}]
# edition['authors']         == [{'key': '/authors/OL34184A'}]
# edition['identifiers']     == {'goodreads': [...], 'librarything': [...]}
# edition['first_sentence']  == {'value': '...'} or str  (often missing)
# edition['ocaid']           == 'fantast00dahl'  ← Internet Archive ID (if available)
```

#### Bibkeys API (enriched, multiple books at once)

```python
# jscmd=data: cleaned up dict with cover URLs pre-built
r = json.loads(http_get(
    "https://openlibrary.org/api/books"
    "?bibkeys=ISBN:9780743273565,ISBN:9780451524935"
    "&format=json&jscmd=data"
))
# r == {'ISBN:9780743273565': {...}, 'ISBN:9780451524935': {...}}

book = r['ISBN:9780743273565']
# book['title']           == 'The Great Gatsby'
# book['authors']         == [{'url': '...', 'name': 'F. Scott Fitzgerald'}]
# book['publish_date']    == '2021'
# book['publishers']      == [{'name': 'Independently Published'}]
# book['number_of_pages'] == 208
# book['url']             == 'http://openlibrary.org/books/OL46773254M/The_Great_Gatsby'
# book['key']             == '/books/OL46773254M'
# book['cover']           == {'small': '...S.jpg', 'medium': '...M.jpg', 'large': '...L.jpg'}
# book['identifiers']     == {'isbn_13': [...], 'openlibrary': [...]}
# book['subjects']        == [{'name': 'Modern fiction', 'url': '...'}, ...]
# book['subject_places']  == None  ← often null even with jscmd=data

# jscmd=details: raw edition JSON + extra fields
r2 = json.loads(http_get(
    "https://openlibrary.org/api/books"
    "?bibkeys=ISBN:9780743273565&format=json&jscmd=details"
))
item = r2['ISBN:9780743273565']
# item['bib_key']      == 'ISBN:9780743273565'
# item['info_url']     == 'http://openlibrary.org/books/OL...'
# item['preview']      == 'noview' | 'restricted' | 'full'
# item['preview_url']  == URL to read on OL or IA
# item['thumbnail_url']== 'https://covers.openlibrary.org/b/id/14314120-S.jpg'
# item['details']      → raw edition JSON (same as /books/OL...M.json)

# Supported bibkey prefixes: ISBN:, OCLC:, LCCN:, OLID: (e.g. OLID:OL46773254M)
```

---

### Authors API

```python
import json
from helpers import http_get

# Lookup by known author key
author = json.loads(http_get("https://openlibrary.org/authors/OL26320A.json"))
# OL26320A is J.R.R. Tolkien (note: not Frank Herbert as originally stated — verify with search)

# author['name']           == 'J.R.R. Tolkien'
# author['fuller_name']    == 'John Ronald Reuel Tolkien'
# author['personal_name']  == 'J. R. R. Tolkien'
# author['birth_date']     == '3 January 1892'   ← string, not parsed
# author['death_date']     == '2 September 1973'
# author['bio']            → str or {'type': '/type/text', 'value': str}
# author['photos']         == [6155606, 6433524, ...]   ← photo IDs for covers API
# author['links']          == [{'title': '...', 'url': '...', 'type': {...}}, ...]
# author['remote_ids']     == {'wikidata': 'Q892', 'viaf': '95218067', ...}
# author['alternate_names']== ['J. R. R. Tolkien', 'TOLKIEN', ...]
# author['key']            == '/authors/OL26320A'
# author['wikipedia']      → URL string or None

# Author works (paginated)
works = json.loads(http_get("https://openlibrary.org/authors/OL26320A/works.json?limit=5"))
# works['size']    == 415
# works['entries'] == [{title, key, covers, authors, created, ...}, ...]
# works['links']   == {'self': '...', 'next': '...'}
```

#### Author search

```python
r = json.loads(http_get("https://openlibrary.org/search/authors.json?q=tolkien"))
# r['numFound'] == 40
# r['docs'][0]:
#   'name'                   == 'Christopher Tolkien'
#   'key'                    == 'OL2623360A'     ← NOTE: no /authors/ prefix here
#   'birth_date'             == '21 November 1924'
#   'death_date'             == '16 January 2020'
#   'top_work'               == 'The War of the Ring'
#   'work_count'             == 43
#   'top_subjects'           == [...]
#   'alternate_names'        == [...]
#   'ratings_average'        float
#   'want_to_read_count'     int
#   'already_read_count'     int
#   'currently_reading_count'int
```

---

### Cover images

Covers are served directly as JPEG — redirect to Internet Archive CDN. No auth needed.

```
# Book covers — three key types:
https://covers.openlibrary.org/b/id/{cover_id}-{size}.jpg      # by cover ID (most reliable)
https://covers.openlibrary.org/b/isbn/{isbn}-{size}.jpg        # by ISBN
https://covers.openlibrary.org/b/olid/{edition_id}-{size}.jpg  # by edition OLID (unreliable — see gotchas)

# Author photos:
https://covers.openlibrary.org/a/id/{photo_id}-{size}.jpg

# Sizes: S (small), M (medium), L (large)
```

```python
import urllib.request

def get_cover_bytes(cover_id: int, size: str = 'M') -> bytes | None:
    """Fetch cover image bytes. Returns None if no cover (43-byte GIF placeholder)."""
    url = f"https://covers.openlibrary.org/b/id/{cover_id}-{size}.jpg"
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
    with urllib.request.urlopen(req, timeout=15) as resp:
        data = resp.read()
    return None if len(data) == 43 else data   # 43-byte GIF = no cover placeholder

# Or just get the URL for embedding:
def cover_url(cover_id: int, size: str = 'M') -> str:
    return f"https://covers.openlibrary.org/b/id/{cover_id}-{size}.jpg"

# Usage:
from helpers import http_get
import json
work = json.loads(http_get("https://openlibrary.org/works/OL893415W.json"))
if work.get('covers'):
    img = get_cover_bytes(work['covers'][0], 'L')   # first cover, large
    # img is ~20–80KB JPEG bytes, redirected from ia*.archive.org
```

To get cover by ISBN directly (e.g. for UI without a full book lookup):
```python
# Medium-size cover by ISBN:
url = f"https://covers.openlibrary.org/b/isbn/{isbn}-M.jpg"
# Redirects to Internet Archive CDN, content-type: image/jpeg
# Use ?default=false to get 404 instead of 1×1 GIF placeholder for missing covers
url_safe = f"https://covers.openlibrary.org/b/isbn/{isbn}-M.jpg?default=false"
```

---

### Subjects API

```python
import json
from helpers import http_get

# Subject slugs: lowercase, underscores for spaces
r = json.loads(http_get("https://openlibrary.org/subjects/science_fiction.json?limit=5"))
# r['name']          == 'science fiction'
# r['subject_type']  == 'subject'   ← also: 'person', 'place', 'time'
# r['work_count']    == 20973
# r['works']         == [{title, key, cover_id, authors, edition_count, ...}, ...]

w = r['works'][0]
# w['title']           == 'Alice\'s Adventures in Wonderland'
# w['key']             == '/works/OL138052W'
# w['cover_id']        == 10527843
# w['cover_edition_key']== 'OL...'
# w['authors']         == [{'key': '/authors/OL22098A', 'name': 'Lewis Carroll'}]
# w['edition_count']   == 3546
# w['first_publish_year']== ...
# w['has_fulltext']    == True | False
# w['ia']              == 'identifier'   (Internet Archive ID when available)

# Pagination: &offset=N
# Place subject:
r2 = json.loads(http_get("https://openlibrary.org/subjects/place:london.json?limit=5"))
# r2['subject_type'] == 'place', r2['work_count'] == 23927

# Person subject:
# https://openlibrary.org/subjects/person:napoleon.json?limit=5
# Time subject:
# https://openlibrary.org/subjects/time:middle_ages.json?limit=5

# Combine with ebooks=true to filter to only freely readable books:
r3 = json.loads(http_get("https://openlibrary.org/subjects/science_fiction.json?limit=5&ebooks=true"))
# r3['works'][i]['has_fulltext'] == True for all results
```

---

### Trending books

```python
import json
from helpers import http_get

for period in ['daily', 'weekly', 'monthly']:
    r = json.loads(http_get(f"https://openlibrary.org/trending/{period}.json?limit=10"))
    # r['works']  == list of search-doc-style objects
    # r['days']   == int (time window)
    # r['hours']  == int
    # Same fields as search docs (title, author_name, cover_i, key, ...)
    print(period, r['works'][0]['title'])  # e.g. 'Atomic Habits'
```

---

## Rate limits

No authentication required. No API key. No explicit rate limit published.

Observed in testing: 5 requests completed in ~1 second with no throttling, no 429s. The API is served from CDN/Solr — in practice you can make 10–20 parallel requests without issue. For bulk operations (hundreds of ISBNs), use `ThreadPoolExecutor(max_workers=5)` to be a good citizen.

**No `User-Agent` override needed** — the default `Mozilla/5.0` from `http_get` is accepted by all Open Library endpoints (unlike Nominatim which blocks it).

---

## Gotchas

**`description` field has two shapes.** Both are real — check at runtime:
```python
desc = work.get('description', '')
text = desc.get('value', '') if isinstance(desc, dict) else (desc or '')
```

**`/works/OL45804W` is Fantastic Mr. Fox, not Dune.** The OL IDs in the original prompt were placeholders. Always resolve real IDs via the search API rather than hardcoding them.

**Author search `key` has no prefix.** `/search/authors.json` returns `key: 'OL26320A'`, but the Authors API and all other APIs use `/authors/OL26320A`. Add the prefix manually when constructing follow-up URLs.

**Missing cover → 43-byte GIF placeholder, not 404.** Without `?default=false`, the covers API returns a 1×1 transparent GIF instead of HTTP 404 for unknown IDs. Check `len(data) == 43` to detect missing covers.

**`covers.openlibrary.org/b/olid/{work_id}` is unreliable.** OLID-based cover URLs for work IDs (OL...W) return the placeholder even when covers exist. Always use `b/id/{cover_id}` (from `work['covers'][0]`) or `b/isbn/{isbn}` instead.

**Bibkeys API picks one edition per ISBN.** When the same ISBN appears on multiple editions (reprint, reissue), `api/books?bibkeys=ISBN:...` returns one — and it may not be the most common edition.

**`publish_date` is a raw string.** Values like `'October 1, 1988'`, `'19/08/2017'`, `'2021'`, and `'1965-01-01'` all appear. Don't parse without normalization.

**`/works/.../editions.json` pagination uses `links.next`.** Unlike search (which uses `offset=`), check `links['next']` in the response to know if more pages exist:
```python
resp = json.loads(http_get("https://openlibrary.org/works/OL893415W/editions.json?limit=50"))
while 'next' in resp.get('links', {}):
    resp = json.loads(http_get("https://openlibrary.org" + resp['links']['next']))
    # process resp['entries']
```

**404 for non-existent IDs.** `/works/OL99999999W.json`, `/books/OL99999999M.json`, and `/authors/OL99999999A.json` all raise `HTTPError: HTTP Error 404: Not Found`. Wrap in try/except.

**Search `docs` default fields are minimal.** The default response includes ~15 fields. Add `&fields=*` to get all 100+ Solr fields (ratings, ISBNs, publishers, subjects, Goodreads IDs, etc.). Alternatively specify exactly what you need: `&fields=title,isbn,ratings_average`.