Web Scraping & API Glossary

The message "execution context was destroyed, most likely because of a navigation" means Playwright started evaluating J...

"Frame was detached" means the iframe you were interacting with was removed from the DOM or replaced while your script h...

page.goto: net::ERRABORTED means the browser started a navigation but the request was cancelled before a response was re...

net::ERRCONNECTIONREFUSED means the browser attempted a TCP connection to the target host and port, and the operating sy...

net::ERRINTERNETDISCONNECTED means the browser has no active network path to reach the target — either the machine itsel...

net::ERRINVALIDAUTH_CREDENTIALS is thrown when a server challenges the browser with HTTP Basic Authentication and the cr...

When Playwright's request fixture (or APIRequestContext) is used alongside page in the same test, their cookies and sess...

This happens when the authentication setup step saves session state to a file, but the test project that runs your tests...

Exact text assertions break on whitespace normalization, added punctuation, translated strings, or copy edits that don't...

Clicking a button that is disabled or in a loading state produces no action — the click goes through Playwright's action...

Cannot read properties of undefined in a Playwright test is a JavaScript runtime error, not a browser or network error. ...

Playwright browser launches fail in CI when the container image is missing the Linux shared libraries that Chromium requ...

This timeout fires when the element exists in the DOM but Playwright cannot confirm it is visible, enabled, and not obsc...

CSS class selectors in Playwright tests couple test code to implementation details of the component's styling. When a de...

This error means a test() call was executed outside of a spec file — most commonly because a helper module or config fil...

Download tests hang when the event listener is attached after the download has already started. The browser fires the do...

"Element is not attached to the DOM" surfaces when a Playwright action targets an element that was in the DOM when queri...

net::ERRNAMENOT_RESOLVED means the browser's DNS resolver could not find an IP address for the hostname in the URL. In P...

The "Executable doesn't exist" error means the @playwright/test package is installed but the corresponding browser binar...

"Execution context was destroyed" appears when Playwright has a JavaScript handle or evaluation in flight and the page n...

File upload failures in Playwright almost always come down to three root causes: targeting the wrong element (a styled <...

Geolocation and browser permissions fail to apply in Playwright tests when they are set at the wrong stage of the browse...

Headless-only failures are among the most confusing Playwright issues because the test is syntactically correct and pass...

Playwright's role-based locators (getByRole, getByLabel) match elements using the computed accessible name, which may co...

locator.isVisible() is a point-in-time query that returns the current visibility state immediately without any retrying....

Leaked state between tests happens when one test leaves side effects that affect subsequent tests in the same worker: co...

locator.count() returns a number immediately — it does not assert, does not retry, and does not fail if the count is wro...

Mobile emulation in Playwright fails to trigger the mobile layout when viewport and device settings are applied after co...

networkidle as a waitUntil condition tells Playwright to wait until there are no network connections for at least 500ms....

"No frame for selector" and similar iframe-related errors happen when test code tries to locate elements inside an ifram...

Index-based locators with .nth() are fragile because they rely on the DOM rendering elements in a specific, stable order...

getByText() matches any element in the document that contains the specified text, including headings, list items, table ...

page.waitForTimeout() (a hard sleep) is the most common anti-pattern in Playwright test suites. It introduces fixed dela...

Parallel Playwright workers interact with a shared database simultaneously, causing data collisions when tests assume ex...

Popup and new-tab tests become flaky when page.waitForEvent('popup') is called after the action that opens the popup. Ju...

A "Protocol error: invalid argument" originates from the Chrome DevTools Protocol (CDP) layer and means Playwright sent ...

Random browser crashes in CI — where tests pass locally but fail unpredictably in CI runners — are almost always resourc...

Playwright retries are designed to improve stability for genuinely intermittent conditions — network latency, slow CI ma...

Route mocking fails silently when the URL pattern doesn't match what the browser actually requests, or when page.route()...

Shadow DOM components encapsulate their internal DOM in a shadow root that is intentionally hidden from the main documen...

Playwright's --shard=N/M splits test files across M runners, but the distribution is based on file count — not test coun...

A strict mode violation in Playwright fires when a locator resolves to more than one element at action time. The error m...

When this error appears alongside a crash — rather than an orderly context.close() call — it means the browser process e...

TimeoutError is thrown when Playwright cannot complete an action or find an actionable element within the configured tim...

waitForResponse misses requests for exactly the same reason waitForEvent('download') and waitForEvent('popup') do — the ...

Weak URL assertions allow tests to pass on the wrong page, producing false positives that hide navigation bugs. expect(p...

Tests that depend on WebSocket connections or Server-Sent Events (SSE) are inherently more sensitive to CI network insta...

Memory-related worker crashes in Playwright parallel runs manifest as browser processes being killed mid-test, producing...

A Playwright strict mode violation means the locator you used resolved to more than one element, but the operation requi...

The error "Target page, context or browser has been closed" surfaces when Playwright tries to interact with a page, fram...

DevTools windows can be treated as regular pages by enabling the handleDevToolsAsPage option when launching or connectin...

BackendNodeId is exposed in the a11y snapshot. Each node in the snapshot includes a backendNodeId that lets you map acce...

Use the capability to retrieve detailed initiator data from CDP when available, and filter out goog: data from events by...

This feature allows opening a page in a tab or a window. newPage() can now be called with window options to choose where...

Overview Landmarks such as header, nav, main, aside, and footer provide semantic regions that assist screen readers and ...

Answer To fix Puppeteer not finding the Chrome binary on Windows, make sure the detector checks the common install locat...

The CDP message ID generator can be configured by passing a custom idGenerator to the Connection constructor. This enabl...

To stop the xdg-open popup in Puppeteer, configure a Chrome policy URLAllowlist and use a Chrome binary that reads that ...

Summary This change adds a public getter to CdpBrowserContext to expose the internal Connection object. It returns the p...

How to expose the url property for links If you need the full URL of a link in Puppeteer, use the url property that was ...

Summary Duplicate header values should normally be merged into a single header value separated by a comma and a space. T...

Fetch.enable wasn't found is raised when trying to enable the Fetch domain for a worker. The fix is to ignore this error...

Puppeteer now dispatches each CDP message in its own JavaScript task by scheduling dispatch with setTimeout. This ensure...

To open DevTools for a page in Puppeteer, use the new Page.openDevTools() method. It calls the DevTools interface for th...

Use the ignoreCache option with Page.reload to reload while ignoring the browser cache. ``js await page.reload({ ignoreC...

Solution Use the Emulation.setUserAgentOverride command via a CDP session to override the user agent instead of relying ...

The deprecation note indicates that the HTTPRequest.postData API is deprecated in Puppeteer. This means you should avoid...

Fixes Puppeteer not waiting for all targets when connecting by only awaiting child targets for tab targets. When connect...

To align with the protocol behavior, create the Response when the responseStarted event fires, rather than after the res...

Summary The pageerror event may emit not only Error objects but also values of unknown type. Treat the payload as unknow...

browser.close() and browser.disconnect() both end your current control flow, but they affect the browser lifecycle diffe...

The test server was removed from the release-please workflow to simplify the release process and remove an unnecessary e...

If you run into TS18028 private identifiers errors when compiling Puppeteer types with TypeScript, set the TypeScript ta...

To fix the TS18028 error, set the TypeScript target to ES2015 or higher. The error occurs because private identifiers (#...

Summary The startup hang in headful mode on Ubuntu was caused by the Firefox Backup Service during startup. A practical ...

Firefox currently mutates the headers object returned by request.headers() in a way that does not reflect in the respons...

This was fixed to prevent accidental mutations of the underlying headers. HttpRequest.headers() no longer allows mutatin...

Summary Firefox addon pages navigated via moz-extension:// are treated as webextension contexts. Puppeteer currently doe...

Answer Avoid blocks by scraping politely and limiting request rates. Respect robots.txt, identify your user agent, and s...

Answer Clean scraped data by trimming whitespace, normalizing formats, and removing duplicates. Validate fields with sch...

Answer Handle pagination by identifying the next page link, page parameter, or API cursor. Start from the first page and...

Answer Use a headless browser to render the page before extracting data. Wait for key selectors to appear or for network...

Answer Web crawling is about discovering and fetching pages, while web scraping is about extracting data from those page...

Answer Web scraping legality depends on the site terms, the data collected, and local laws. Public data may be allowed, ...

Answer Common tools include Beautiful Soup, Scrapy, Playwright, Puppeteer, and Selenium. Lightweight parsers are great f...

Answer Ethical scraping means minimizing harm and respecting site owners and users. Follow robots.txt, terms of service,...

Answer The best format depends on how you plan to use the data. CSV is simple and works well for tabular data and quick ...

Answer Web scraping is the process of extracting specific data from web pages and converting it into structured formats....

Answer Web crawling focuses on discovering and retrieving pages, while web scraping extracts specific data from those pa...

Answer Match crawl frequency to how often content changes and how quickly you need updates. High‑change sites may need m...

Answer To avoid getting blocked, crawl politely and predictably. Respect robots.txt, use reasonable rate limits, and ide...

Answer To crawl JavaScript‑heavy sites, use a headless browser to render pages before extracting content. Wait for criti...

Answer Web crawling legality depends on the website, the data you collect, and the laws in your jurisdiction. Many sites...

Answer Common web crawling tools include Scrapy, Apache Nutch, Playwright, Puppeteer, and managed crawler platforms. Scr...

Answer Common crawler data includes URLs, status codes, headers, page content, metadata, links, and timestamps. Many sys...

Answer Crawl budget is the number of pages a crawler can fetch within time and resource constraints. It is limited by yo...

Answer robots.txt is a file at a site root that tells crawlers which paths they may or may not access. It uses a simple ...

Answer Web crawling is the automated process of discovering and fetching web pages by following links so you can build a...