', html, re.DOTALL ) results = [] for title, url, price, location in listings: pid_match = re.search(r'/(\d+)\.html$', url) results.append({ "post_id": pid_match.group(1) if pid_match else None, "title": title, "url": url, "price": price.strip() or None, # None if listing has no price "location": location.strip(), }) return results # Usage results = search_craigslist("sfbay", "sss", "macbook pro", max_price=1000) for r in results[:5]: print(r["post_id"], r["price"], r["location"], r["title"][:50]) ``` ### Handling missing price Listings without a price have no `

` element. The regex above returns an empty string for `price`; the example converts that to `None`. A more robust extraction: ```python def parse_listings(html): results = [] for block in re.findall(r'

', html, re.DOTALL): title = re.search(r'title="([^"]+)"', block) url = re.search(r'href="([^"]+)"', block) price = re.search(r'

([^<]+)

', block) loc = re.search(r'

\s*([^<]*?)\s*

', block) if not url: continue url_str = url.group(1) pid = re.search(r'/(\d+)\.html$', url_str) results.append({ "post_id": pid.group(1) if pid else None, "title": title.group(1) if title else None, "url": url_str, "price": price.group(1).strip() if price else None, "location": loc.group(1).strip() if loc else None, }) return results ``` ## Individual listing page extraction Listing pages are also fully server-rendered. All fields are present in the raw HTML. ```python def get_listing(url): headers = {"User-Agent": "Mozilla/5.0"} html = http_get(url, headers=headers) title = re.search(r'([^<]+)', html) price = re.search(r'(\$[\d,]+)', html) # Location is in parentheses right after the price span location = re.search( r'[^<]+\s*$([^)]+)$\s*', html ) posted = re.search(r'class="date timeago"[^>]+datetime="([^"]+)"', html) post_id = re.search(r'post id:\s*(\d+)', html) # Description body body_block = re.search(r'section id="postingbody"[^>]*>(.*?)', html, re.DOTALL) body_text = "" if body_block: raw = re.sub(r'<[^>]+>', '', body_block.group(1)).strip() # Remove the "QR Code Link to This Post" print-only block body_text = re.sub(r'QR Code Link to This Post\s*', '', raw).strip() body_text = re.sub(r'\s+', ' ', body_text) # Images images = re.findall(r'https://images\.craigslist\.org/[^\s"\']+_600x450\.jpg', html) # Attributes (condition, make, model, etc.) attrs = {} for labl, valu in re.findall( r'([^<]+).*?\s*(?:<[^>]+>\s*)*([^<\n]+?)(?:\s*` blocks for bedrooms/bathrooms and square footage, separate from the `

` attribute grid: ```python # BR/BA br_ba = re.search(r'(\d+)BR\s*/\s*(\d+(?:\.\d+)?)Ba', html) # Square footage sqft = re.search(r'(\d+)ft²', html) if br_ba: bedrooms, bathrooms = br_ba.groups() if sqft: sqft_val = sqft.group(1) ``` ## JSON-LD structured data (alternative extraction path) Each search page includes an `ItemList` JSON-LD block with up to 330 items. Useful when you want structured data (price as float, geo coordinates) without regex parsing of HTML: ```python import json, re from helpers import http_get html = http_get("https://sfbay.craigslist.org/search/sss?query=laptop", headers={"User-Agent": "Mozilla/5.0"}) ld_blocks = re.findall(r'', html, re.DOTALL) for raw in ld_blocks: data = json.loads(raw) if data.get('@type') == 'ItemList': for item in data['itemListElement']: listing = item['item'] print( listing.get('name'), listing.get('offers', {}).get('price'), listing.get('offers', {}).get('priceCurrency'), listing.get('offers', {}).get('availableAtOrFrom', {}).get('address', {}).get('addressLocality'), ) ``` JSON-LD item fields available: `name`, `description`, `image` (list of URLs), `offers.price` (float string e.g. `"900.00"`), `offers.priceCurrency`, `offers.availableAtOrFrom.address`, `offers.availableAtOrFrom.geo.latitude`, `offers.availableAtOrFrom.geo.longitude`. Note: JSON-LD items do not include the listing URL or post ID — use the HTML parser for those. Combine both: use JSON-LD for price/geo, HTML for URL/post ID. ## Pagination behavior The `s=` offset parameter in the URL is only respected by the JS-driven XHR layer in a real browser. When accessed via `http_get`, the static HTML fallback renders all results regardless of `s=`: ``` s=0 → same 342 listings s=120 → same 342 listings (confirmed identical URL sets) s=300 → same 342 listings ``` **Recommendation**: Do not attempt pagination via `http_get`. Use search filters to narrow results: ```python # Instead of paginating, narrow by price range under_500 = search_craigslist("sfbay", "sss", "macbook", max_price=500) over_500 = search_craigslist("sfbay", "sss", "macbook", min_price=501) ``` If true pagination is required (e.g. you need more than 350 results), you must use a browser session with `goto_url()` + `wait_for_load()`. ## Bot detection None observed. Craigslist does not block `http_get` requests. During testing: - All 6+ test cities returned full HTML (HTML size 174K–530K bytes per page) - No CAPTCHA page, no redirect to `robot-check`, no `403` - No cookie or session required - Works with minimal `User-Agent` header: `"Mozilla/5.0"` is sufficient Defensive check (in case behavior changes): ```python def is_blocked(html): return ( len(html) < 5000 or "blocked" in html[:2000].lower() or "captcha" in html[:2000].lower() or "cl-static-search-result" not in html ) ``` ## Gotchas - **`data-pid` does not exist in static HTML**: Old Craigslist used `data-pid` attributes. The current static renderer uses `

` with title attribute and embedded ``. Do not search for `data-pid`, `result-row`, or `cl-search-result` — they are absent. - **Post ID comes from the URL, not an attribute**: Extract it as the numeric segment before `.html` in the listing URL: `re.search(r'/(\d+)\.html$', url).group(1)`. - **Price may be absent**: Free listings and "contact for price" listings have no `

`. The regex returns an empty string; convert to `None`. - **`s=` pagination is a no-op in static HTML**: The fallback renderer always returns the full result set. Don't loop over pages — filter instead. - **HTML entities in titles**: Titles may contain `&`, `"`, etc. Use `html.unescape(title)` from the standard library if you need clean text. - **URL structure varies by area**: The area code in the URL (`/sby/`, `/sfc/`, `/eby/`) is the sub-area of the city (e.g. South Bay, San Francisco, East Bay). It is part of the listing URL but not needed for constructing search URLs (which use the city subdomain only). - **`