Screen scraping: tools, Python examples, and automation

A product page shows a title, price, and availability, but you need those details in a dataset you can update and compare. 

You’ll learn how to choose an extraction method, check that the output matches the page, and save useful records. From there, we’ll explore Python and automation options for repeating the task, along with proxy settings when location or connection consistency matters. 

What is screen scraping

Screen scraping is the automated extraction of information presented through a user interface. A screen scraper reads that information and converts it into data another system can use, such as spreadsheet rows or fields in a JSON file.

The source determines how the scraper works:

• Webpages: A browser loads a product page, then the scraper collects its title, price, and availability from the rendered content.

• Legacy applications: Automation reads inventory or customer information from an older business system that lacks a useful export option.

• Images: Optical character recognition (OCR) converts text in a screenshot or scanned image into machine-readable characters.

How does screen scraping work?

On a product page, say Amazon, the browser loads the interface, extraction rules identify values, and validation checks whether the results are usable. The process follows five stages:

1. Open the page. Navigate to the product URL with the location and session settings needed for your task.

2. Wait for the required content. Confirm that the price and availability have loaded before collecting them.

3. Locate and extract the fields. Use selection rules to capture the product title, price, currency, and stock status.

4. Validate the values. Check for missing fields, unexpected currencies, and error messages accidentally captured as product data.

5. Save the output. Store the records with their source URLs and collection times so you can compare results later.

Your Amazon scraper can return extracted fields alongside a screenshot. Keeping both helps investigate discrepancies: you can compare a missing or incorrect value with what the browser displayed and identify where extraction went wrong.

Screen scraping vs. web scraping and APIs

Choose the method that provides the required fields with the least unnecessary processing. Start with the simplest method that provides the data you need. Check for an official API or a built-in catalog export, such as a CSV file, Excel price list, or XML feed.

If neither is suitable, inspect the HTML returned by the server to see whether it contains the required fields or whether they load through JavaScript. This helps determine whether browser rendering or interaction is needed.

Screen scraping and web scraping overlap. Rendering a website and reading its page elements remains web scraping; using a browser does not automatically mean interpreting pixels.

Screen scraping tools: choose by the data source

The right web screen scraping tools depend on where the data appears and how you access it. Open source screen scraping tools cover browser automation and OCR, while broader platforms also support desktop processes.

Yozh Scraper: a ready-made browser extraction service

Yozh Scraper is a free, self-hosted REST API built on Playwright. It renders webpages in a real browser and returns fields selected with CSS or XPath, clean Markdown for large language models (LLMs), raw HTML, and full-page screenshots. Built-in and custom presets let you reuse extraction rules, while an asynchronous job queue handles single-page and batch requests.

After the initial repository and environment setup, docker compose up --build starts both the scraper and its companion crawler. The optional Scraper Tester requires Node.js and starts separately, providing a web interface for testing selectors and inspecting results.

The project uses the MIT license, so you can run it locally without software license fees or paid integrations. You supply the computing resources; hosting, paid proxies, and optional LLM API usage carry separate costs when used, including during testing. The next section walks through configuring an extraction and verifying its results.

Playwright: direct browser control from Python

Choose Playwright when you want to script clicks, filters, and navigation in Python. It provides browser automation, while your code defines extraction, validation, and storage. You maintain the selectors and loading conditions.

Tesseract: text recognition from images

Tesseract is a free, open-source OCR engine that converts text in images into machine-readable output. Use it for scanned documents or information available only as pixels, such as prices, phone numbers, or SKUs embedded in images or drawn on a canvas. Capture the relevant image first, then check the recognized text for errors before saving it.

UiPath: automation across application interfaces

UiPath is a robotic process automation (RPA) platform for desktop and web applications. It suits workflows involving legacy Windows warehouse or accounting systems without useful export features. Automation can move between application windows, read accessible interface controls, and use OCR when text is available only on screen. Extracted table values still need checks for recognition errors and correct row and column alignment.

How to do screen scraping with Yozh Scraper

Start with Books to Scrape, a sandbox bookstore built for scraping practice. Its A Light in the Attic product page has predictable fields and static content, making it useful for learning the controls before tackling dynamic websites.

1. Define the output

Create a record containing the product title, price, availability, source URL, and collection time. Extract the first three from the page; add the URL and a UTC timestamp when saving the record.

2. Start the scraper

With Git and Docker installed, follow the Yozh Scraper installation guide:

git clone https://github.com/CyberYozh-data/yozh-scraper

git clone https://github.com/CyberYozh-data/yozh-scraper
cd yozh-scraper
cp .env.example .env
docker compose up --build

The final command starts both the scraper on port 8000 and its companion crawler on port 8001. Keep Docker running. Open the scraper’s API documentation at http://localhost:8000/docs; the crawler’s documentation is available at http://localhost:8001/docs.

3. Open the visual tester

The tester starts separately and requires Node.js 18+. In another terminal, navigate to the yozh-scraper repository folder and run:

cd scraper-tester
npm install
node server.js

Open http://localhost:7000. In the header, confirm that Scraper URL is http://localhost:8000, then click Check beside it. A green health badge confirms that the scraper is reachable.

4. Configure extraction

In Scrape Page, enter the practice URL, select a direct connection (none), and configure CSS extraction with these rules:

Read each element’s text and mark the fields as required. Set wait_for_selector to .product_main .price_color. Enable screenshot output and keep image blocking disabled.

5. Run a small sample

Open Presets and use the Preset Builder. Paste the same product URL, click Fetch sample, then choose Manual and enter the verified CSS rules. Click Verify and inspect the extracted values before saving.

Choose a descriptive name; the tester automatically adds the user_ prefix. Select the saved preset in Scrape Page and test another book page before expanding collection. AI-assisted field selection is also available when configured.

For later tasks requiring regional access, set CYBERYOZH_API_KEY in .env, apply the updated Docker configuration, and select your purchased proxy connection. Use res_rotating with proxy_geo.country_code to target a country. Yozh Scraper automatically aligns the browser’s locale, timezone, and Accept-Language header with that country. Verify the exit IP’s location and the displayed content before collecting a larger dataset.

6. Save a reusable preset

Open Presets, fetch the same sample, choose Manual, and enter the verified rules. Use Verify, then save the preset with a descriptive name. Select it in Scrape Page and test another book page before expanding collection. AI-assisted field selection is also available when configured.

When a later target requires regional access, configure your CyberYozh API key and purchased proxy connection. Rotating residential proxies support location and geotargeting; check the returned region before collecting a larger dataset.

Python screen scraping: extract and save page data

Python screen scraping gives you control over page navigation, extraction, and storage. This example uses the same Books to Scrape product page and selectors from the visual walkthrough, then saves a validated record as JSON.

Install Playwright with python -m pip install playwright, followed by python -m playwright install chromium. Save the script as scrape_product.py and run python scrape_product.py.

For an optional CyberYozh connection, set these environment variables in the shell running the script:

• CYBERYOZH_PROXY_SERVER: your HTTP proxy endpoint, formatted as http://HOST:PORT.

• CYBERYOZH_PROXY_USERNAME: your proxy username.

• CYBERYOZH_PROXY_PASSWORD: your proxy password.

Without the server variable, the script connects directly. Playwright supports proxy configuration for the browser or an individual browser context.

import json
import os
from datetime import datetime, timezone
from playwright.sync_api import sync_playwright, TimeoutError

URL = "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"

FIELDS = {
    "title": ".product_main h1",
    "price": ".product_main .price_color",
    "availability": ".product_main .availability",
}

options = {}
if server := os.getenv("CYBERYOZH_PROXY_SERVER"):
    options["proxy"] = {
        "server": server,
        "username": os.getenv("CYBERYOZH_PROXY_USERNAME", ""),
        "password": os.getenv("CYBERYOZH_PROXY_PASSWORD", ""),
    }

with sync_playwright() as p:
    browser = p.chromium.launch(**options)
    try:
        page = browser.new_page()
        response = page.goto(URL, wait_until="domcontentloaded")

        if response is None or not response.ok:
            raise SystemExit("Page request failed; check the URL and connection.")

        record = {}
        for field, selector in FIELDS.items():
            element = page.locator(selector)
            try:
                element.wait_for(state="visible", timeout=10000)
                value = " ".join(element.inner_text().split())
            except TimeoutError:
                raise SystemExit(f"Required field missing or not ready: {field}")

            if not value:
                raise SystemExit(f"Required field is empty: {field}")

            record[field] = value

        record.update(
            source_url=page.url,
            collected_at=datetime.now(timezone.utc).isoformat(),
        )

        with open("product.json", "w", encoding="utf-8") as output:
            json.dump(record, output, ensure_ascii=False, indent=2)

        print("Saved product.json")
    finally:
        browser.close()

The script waits for each required element, removes excess whitespace, and preserves the displayed currency. Missing or empty fields stop the run with a clear message before a new record is written. Successful output includes the source URL and UTC collection time; rerunning replaces product.json.

If you already configured Yozh Scraper, Python can reuse your saved preset through its REST API:

1. Submit it to POST /api/v1/scrape/preset/page and retain the returned job_id.

2. Check GET /api/v1/scrape/{job_id} until completion, with an overall timeout and handling for failed or canceled jobs.

3. Retrieve GET /api/v1/scrape/{job_id}/results, validate the fields, and save them.

Automate screen scraping with crawlers, schedules, and AI tools

Once the first product-page extraction works, the next challenge is collecting fresh data across the catalog. Build automation around that tested setup, with limits on discovery, scheduled runs, and checks before saving results.

Discover additional pages with Yozh Crawler

Yozh Crawler, the companion service on port 8001, discovers links and fetches pages through Yozh Scraper. Provide a category page as seed_url, then set scope.mode to same-domain or use regex with include and exclude patterns to restrict discovery. Set scope.max_pages and scope.max_depth to limit page count and link depth. Enable enable_scraping and supply extraction rules to retain extracted fields alongside the discovered URLs.

On e-commerce sites, the same product URL can appear with a trailing slash or the same query parameters in a different order. URL canonicalization normalizes these variations into a shared fingerprint to reduce duplicate visits. In same-domain mode, the crawler also treats the hostname with and without www as equivalent. Changing a parameter’s value, such as a sorting option or product variant, is different from reordering parameters and does not automatically make the URLs equivalent.

Control request frequency with scope.per_domain_rps and simultaneous requests with scope.per_domain_concurrency. Start with a limit of 10 to 20 pages, inspect the discovered URLs, and exclude irrelevant sections such as blog posts or contact pages before expanding collection.

Schedule collection with n8n

Connect n8n to a reachable Yozh REST API endpoint and build this sequence:

Scheduled trigger → submit scraping job → check completion → validate results → save data.

Use n8n’s Schedule Trigger to define the interval and an HTTP Request node to submit the job. Retain its job ID, then use a wait-and-check loop to monitor completion. Add an overall timeout and a failure branch so unsuccessful jobs receive attention.

Before saving, check required fields and attach the collection time. This integration uses Yozh’s API and n8n’s standard nodes, with scheduling managed in n8n.

Connect an AI assistant through MCP

Model Context Protocol (MCP) lets AI assistants such as Claude call external tools. For a local setup, connect Claude Code to [http://localhost:8000/mcp](http://localhost:8000/mcp) using Streamable HTTP. You can then ask: “Scrape this product page and summarize the prices in a table.” The assistant can submit the scraping job, check its progress, retrieve the results, and organize the extracted values. :chatgpt-content-reference{index="0"}

Yozh Scraper also supports AI-generated presets. Use from_prompt to describe the fields you need in plain language, or from_schema to supply field names and data types. Together with a sample page, these inputs let the model generate selectors that you can verify before saving. :chatgpt-content-reference{index="1"}

For automatic selector repair, configure self_heal and supply an LLM configuration for the preset run. If a required field returns empty, the scraper attempts to regenerate its selector from the current page markup. Repaired selectors can be saved back to user presets; built-in presets remain read-only. Compare the repaired values with the page screenshot, since a populated field can still contain the wrong price or product detail. :chatgpt-content-reference{index="2"}

To enable Yozh’s AI generation and repair features, configure an API key for OpenAI, Anthropic, Google Gemini, or OpenRouter in .env and select a compatible model. API usage is billed separately under your chosen provider’s pricing. :chatgpt-content-reference{index="3"}

Choose the connection and session settings for your task

When screen scraping websites, choose connection settings around the location you need, session length, and how often you revisit a page. CyberYozh’s proxy options fit different requirements:

• Rotating residential ($4/GB pay-as-you-go): Use regional IPs for collecting independent product pages, with sticky sessions up to 24 hours when connected steps need continuity. Unused traffic rolls over, and larger packages offer lower per-GB rates.

• Static ISP (from $5.29/month): Choose a fixed IP with unlimited traffic for repeated visits, authenticated collection, and tasks that require a consistent network address.

• Mobile LTE/4G/5G (from $1.70/day for shared plans): Consider carrier connections for Google search scraping and other targets with strict traffic filters. Carrier-grade NAT (CGNAT) lets many ordinary mobile subscribers share a public IP, so blocking the entire address can also affect legitimate visitors. This makes carrier connections worth testing, although shared IPs can still receive CAPTCHAs or rate limits. Unlimited traffic keeps bandwidth costs predictable when browser rendering repeatedly downloads scripts, styles, and images. For parallel collection, rent multiple mobile channels and distribute browser workers across them, adjusting concurrency to connection capacity and the target’s limits.

• Datacenter: Consider these for suitable public sources where a small test confirms that access and extraction work.

For pages requiring a login or a consistent delivery ZIP code, session continuity depends on both saved browser state and proxy IP persistence. Cookies and localStorage can retain authentication information and regional preferences, depending on how the website stores them.

Yozh Scraper lets you create a session through POST /api/v1/sessions, authenticate through POST /api/v1/sessions/{id}/login, and pass the returned session_id with subsequent scraping requests. This reuses saved browser state and keeps the session’s proxy configuration consistent. Pair it with a static IP or an appropriately configured sticky connection when the outward-facing address must remain stable. A fixed IP cannot renew an expired login token, while saved cookies cannot guarantee that a website will accept a sudden IP change.

Common screen scraping problems and how to fix them

When a run produces incomplete or inconsistent records, compare the saved screenshot with the sample output from the same job. This helps distinguish a page-loading problem from an extraction or formatting error.

After changing a Yozh preset, rerun a small sample before restarting the full collection. For detailed retry handling, see 429 too many requests.

Is screen scraping legal?

Screen scraping can be legal, but legality depends on the source, access method, type of data, jurisdiction, and intended use. Being able to view a page does not automatically authorize every form of collection or reuse.

Before running a job, review the site’s contractual terms, access restrictions, and any required permissions or licenses. Respect its robots.txt directives, but remember that an Allow rule does not itself grant legal permission to collect or reuse content. Check applicable copyright and database rights before republishing or selling extracted material.

If you collect personal information, comply with applicable data protection regulations, including the GDPR where it applies. Publicly visible information can still be protected, as international privacy regulators have emphasized. Establish a valid lawful basis for processing and collect only the personal data necessary for your stated purpose.

Privacy obligations also remain relevant. International privacy regulators have emphasized that publicly accessible personal information can still be protected by data protection laws.

For this example, keep Yozh Scraper limited to the selected practice pages and product fields. For production work, document the permitted scope, collect only necessary data, restrict access to stored records, and set retention and deletion rules. Include saved HTML and screenshots in those controls, since they may capture additional information.