--- id: impit-http-client title: Impit HTTP Client description: Browser impersonation for HTTP requests using the Impit library --- import CodeBlock from '@theme/CodeBlock'; import BasicUsageSource from '!!raw-loader!./basic-usage.ts'; import CheerioCrawlerSource from '!!raw-loader!./cheerio-crawler.ts'; import HttpCrawlerSource from '!!raw-loader!./http-crawler.ts'; import AdvancedConfigSource from '!!raw-loader!./advanced-config.ts'; ## Introduction The `ImpitHttpClient` is an HTTP client implementation based on the [Impit](https://github.com/apify/impit) library. It enables browser impersonation for HTTP requests, helping you bypass bot detection systems without running an actual browser. :::info Successor to got-scraping Impit is the successor to `got-scraping`, which is no longer actively maintained. We recommend using `ImpitHttpClient` for all new projects. Impit provides better anti-bot evasion through TLS fingerprinting and HTTP/3 support, while maintaining a smaller package size. **Impit will become the default HTTP client in the next major version of Crawlee.** ::: ### Why use Impit? Websites increasingly use sophisticated bot detection that analyzes: - **HTTP fingerprints**: User-Agent strings, header ordering, HTTP/2 pseudo-header sequences - **TLS fingerprints**: Cipher suites, TLS extensions, and cryptographic details in the ClientHello message Standard HTTP clients like `fetch` or `axios` are easily detected because their fingerprints don't match real browsers. Unlike `got-scraping` which only handles HTTP-level fingerprinting, Impit also mimics TLS fingerprints, making requests appear to come from real browsers. ## Installation Install the `@crawlee/impit-client` package: ```bash npm2yarn npm install @crawlee/impit-client ``` :::note The `impit` package includes native binaries and supports Windows, macOS (including ARM), and Linux out of the box. ::: ## Basic usage Pass the `ImpitHttpClient` instance to the `httpClient` option of any Crawlee crawler: {BasicUsageSource} ## Usage with different crawlers ### CheerioCrawler {CheerioCrawlerSource} ### HttpCrawler {HttpCrawlerSource} ## Configuration options The `ImpitHttpClient` constructor accepts the following options: | Option | Type | Default | Description | |--------|------|---------|-------------| | `browser` | `'chrome'` \| `'firefox'` \| `...` | `undefined` | Browser to impersonate. Affects TLS fingerprint and default headers. | | `http3` | `boolean` | `false` | Enable HTTP/3 (QUIC) protocol support. | | `ignoreTlsErrors` | `boolean` | `false` | Ignore TLS certificate errors. Useful for testing or self-signed certificates. | ### Available fingerprints Impit bundles several realistic browser fingerprints. Using version-specific fingerprints can improve success rates against sophisticated bot detection systems that track browser versions and flag outdated signatures. When using generic fingerprints (`chrome` or `firefox`), Impit automatically selects an appropriate version. For most use cases, the generic options are sufficient. However, if you're targeting a site with particularly strict bot detection, or need to match a specific browser environment, you can specify an exact version: ```ts import { ImpitHttpClient } from '@crawlee/impit-client'; // Use a generic Chrome fingerprint const chromeClient = new ImpitHttpClient({ browser: 'chrome' }); // Use a specific Chrome version const chrome131Client = new ImpitHttpClient({ browser: 'chrome131' }); // Or a specific Firefox version const firefox144Client = new ImpitHttpClient({ browser: 'firefox144' }); ``` ### Advanced configuration {AdvancedConfigSource} ## Proxy support Proxies are configured per-request through Crawlee's proxy management system, not on the `ImpitHttpClient` itself. Use `ProxyConfiguration` as you normally would: ```ts import { CheerioCrawler, ProxyConfiguration } from 'crawlee'; import { ImpitHttpClient, Browser } from '@crawlee/impit-client'; const proxyConfiguration = new ProxyConfiguration({ proxyUrls: ['http://proxy1.example.com:8080', 'http://proxy2.example.com:8080'], }); const crawler = new CheerioCrawler({ httpClient: new ImpitHttpClient({ browser: Browser.Chrome }), proxyConfiguration, async requestHandler({ $, request }) { console.log(`Scraped ${request.url}`); }, }); ``` ## How it works Impit achieves browser impersonation at two levels: 1. **HTTP level**: Mimics browser-specific header ordering, HTTP/2 settings, and pseudo-header sequences that antibot services analyze. 2. **TLS level**: Uses a patched version of `rustls` to replicate the exact TLS ClientHello message that browsers send, including cipher suites and extensions. This dual-layer approach makes requests appear to come from a real browser, significantly reducing blocks from bot detection systems. ## Comparison with other solutions | Feature | got-scraping | curl-impersonate | Impit | |---------|--------------|------------------|-------| | TLS fingerprinting | No | Yes | Yes | | HTTP/3 support | No | Yes | Yes | | Native Node.js package | Yes | No (child process) | Yes | | Windows/macOS ARM | Yes | No | Yes | | Package size | ~10 MB | ~20 MB | ~8 MB | **Related links** - [Impit GitHub repository](https://github.com/apify/impit) - [Custom HTTP Client guide](./custom-http-client) - [Proxy Management guide](./proxy-management) - [Avoiding blocking guide](./avoid-blocking)