How to Scrape Products From a Website

What is product data scraping?

Product data scraping is the automated collection of structured information from ecommerce websites, retailer pages, marketplaces, and online catalogs.

A scraper reads page information and converts it into consistent fields such as product name, brand, SKU, category, current and original price, currency, availability, description, specifications, variants, seller information, rating and review count, image URL, product URL, and collection timestamp.

The output may be a spreadsheet, CSV, JSON, database table, API response, or recurring data feed—either for a one-time catalog or for monitoring prices, stock, promotions, sellers, and assortment changes.

Start with the dataset, not the website

A common mistake is opening developer tools before deciding what the finished dataset should contain. Start by answering:

1. Which pages must be covered?
2. Which fields are required?
3. What does one row represent?
4. How will the data be used?

Product, variant, offer, and observation

These entities should not be treated as interchangeable. A product is the general item; a variant is a configuration such as size and color; an offer is a seller-specific listing; an observation is the information collected at a particular time and location.

What product fields should you extract?

The correct fields depend on the business objective. Common groups include identity (name, brand, SKU, GTIN, product ID, URL), classification, pricing, availability and fulfillment, attributes, marketplace information, customer-feedback fields, and collection metadata such as timestamp, country, store, scrape status, and error status.

Do not collect every visible field simply because it is available. Extra fields increase extraction complexity, storage, validation, and maintenance.

Check how the website loads its product data

• Static HTML — Product information is in the HTML returned by the server; a normal HTTP request may be enough.
• JSON-LD structured data — Many pages include Schema.org Product data inside a script type="application/ld+json" element.
• Embedded JSON — Product data may sit in JavaScript variables or framework state.
• Network requests — Prices, inventory, variants, or reviews may load from an endpoint after the initial page.
• JavaScript-rendered content — The initial HTML may be a shell; Playwright or similar browser automation may be required.

How to scrape one product page with Python

The following example uses fictional selectors. Replace them only after inspecting a permitted target page.

Install the libraries

pip install requests beautifulsoup4

Basic scraper

from __future__ import annotations

import json
from dataclasses import asdict, dataclass
from decimal import Decimal, InvalidOperation
from typing import Optional

import requests
from bs4 import BeautifulSoup


@dataclass
class Product:
    name: str
    url: str
    sku: Optional[str]
    price: Optional[str]
    availability: Optional[str]


def clean_text(value: Optional[str]) -> Optional[str]:
    if value is None:
        return None
    cleaned = " ".join(value.split())
    return cleaned or None


def parse_price(value: Optional[str]) -> Optional[str]:
    if not value:
        return None
    numeric = (
        value.replace(",", "")
        .replace("$", "")
        .replace("€", "")
        .replace("£", "")
        .strip()
    )
    try:
        return str(Decimal(numeric))
    except InvalidOperation:
        return None


def scrape_product(url: str) -> Product:
    headers = {
        "User-Agent": (
            "Mozilla/5.0 (compatible; ProductResearchBot/1.0; "
            "+https://your-company.example/bot-information)"
        )
    }
    response = requests.get(url, headers=headers, timeout=30)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")

    name_element = soup.select_one("h1.product-name")
    sku_element = soup.select_one("[data-product-sku]")
    price_element = soup.select_one(".product-price")
    availability_element = soup.select_one(".availability")

    if name_element is None:
        raise ValueError("Required product name was not found")

    return Product(
        name=clean_text(name_element.get_text()) or "",
        url=url,
        sku=(
            clean_text(sku_element.get("data-product-sku"))
            if sku_element
            else None
        ),
        price=parse_price(
            clean_text(price_element.get_text())
            if price_element
            else None
        ),
        availability=(
            clean_text(availability_element.get_text())
            if availability_element
            else None
        ),
    )


def main() -> None:
    product_url = "https://store.example/products/demo-running-shoe"
    try:
        product = scrape_product(product_url)
    except requests.RequestException as exc:
        raise SystemExit(f"Request failed: {exc}") from exc
    except ValueError as exc:
        raise SystemExit(f"Extraction failed: {exc}") from exc
    print(json.dumps(asdict(product), indent=2))


if __name__ == "__main__":
    main()

A possible result would be:

{
  "name": "Demo Running Shoe",
  "url": "https://store.example/products/demo-running-shoe",
  "sku": "DRS-100-BLK",
  "price": "79.99",
  "availability": "In stock"
}

This output is illustrative. It is not evidence from a live Nenodata extraction project.

The script can extract one permitted static page, but it does not yet find other product URLs, process pagination, run JavaScript, extract variants, retry failures, normalize currencies, detect duplicates, validate catalog completeness, monitor layout changes, or save historical observations.

Extract JSON-LD before relying on CSS selectors

CSS selectors can change when a website redesigns. Structured product data may provide a cleaner source when it is present and complete.

from __future__ import annotations

import json
from typing import Any, Optional

from bs4 import BeautifulSoup


def find_product_json_ld(html: str) -> Optional[dict[str, Any]]:
    soup = BeautifulSoup(html, "html.parser")
    scripts = soup.select('script[type="application/ld+json"]')

    for script in scripts:
        raw_json = script.string
        if not raw_json:
            continue
        try:
            data = json.loads(raw_json)
        except json.JSONDecodeError:
            continue

        candidates = data if isinstance(data, list) else [data]
        for candidate in candidates:
            if not isinstance(candidate, dict):
                continue
            if candidate.get("@type") == "Product":
                return candidate
            graph = candidate.get("@graph", [])
            if isinstance(graph, list):
                for item in graph:
                    if isinstance(item, dict) and item.get("@type") == "Product":
                        return item
    return None

Do not assume JSON-LD contains every required field. Compare it with the visible page and confirm whether prices, variants, stock, and sellers are represented correctly.

Export product data to CSV

import csv
from dataclasses import asdict
from pathlib import Path
from typing import Iterable


def save_products_to_csv(
    products: Iterable[Product],
    output_path: str,
) -> None:
    rows = [asdict(product) for product in products]
    if not rows:
        raise ValueError("No products were provided")

    path = Path(output_path)
    with path.open("w", newline="", encoding="utf-8-sig") as file:
        writer = csv.DictWriter(file, fieldnames=list(rows[0].keys()))
        writer.writeheader()
        writer.writerows(rows)

For recurring monitoring, add fields such as collected_at, currency, location, seller, variant ID, and scrape status. Without timestamps and context, a price record cannot reliably support historical analysis.

How to find every product on a website

Product URLs may be found through category and subcategory pages, pagination, infinite scrolling, XML sitemaps, internal search, brand pages, seller storefronts, approved product feeds, or public catalog endpoints.

Follow category pagination

from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup


def discover_product_urls(start_url: str) -> set[str]:
    discovered: set[str] = set()
    visited_pages: set[str] = set()
    next_url: str | None = start_url

    while next_url and next_url not in visited_pages:
        visited_pages.add(next_url)
        response = requests.get(next_url, timeout=30)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "html.parser")

        for link in soup.select("a.product-card-link[href]"):
            absolute_url = urljoin(next_url, link["href"])
            discovered.add(absolute_url)

        next_element = soup.select_one("a.next-page[href]")
        next_url = (
            urljoin(next_url, next_element["href"])
            if next_element
            else None
        )

    return discovered

Normalize canonical product URLs carefully. Remove only parameters confirmed not to change product identity or location context—a parameter may control size, seller, store, language, or delivery location.

Reconcile expected and discovered products: displayed count, unique URLs discovered, attempted, successful, failed, empty, duplicate, product/variant/offer records, and removed products. Agree the data model before interpreting row counts.

How to scrape JavaScript-rendered products

Install Playwright and browsers:

pip install playwright
playwright install

from __future__ import annotations

import json

from playwright.sync_api import (
    TimeoutError as PlaywrightTimeoutError,
    sync_playwright,
)


def scrape_dynamic_product(url: str) -> dict[str, str | None]:
    with sync_playwright() as playwright:
        browser = playwright.chromium.launch(headless=True)
        try:
            page = browser.new_page(viewport={"width": 1440, "height": 900})
            page.goto(url, wait_until="domcontentloaded", timeout=45_000)
            page.locator("h1.product-name").wait_for(state="visible", timeout=15_000)

            name = page.locator("h1.product-name").inner_text()
            price_locator = page.locator(".product-price")
            stock_locator = page.locator(".availability")

            return {
                "name": name.strip(),
                "price": (
                    price_locator.inner_text().strip()
                    if price_locator.count()
                    else None
                ),
                "availability": (
                    stock_locator.inner_text().strip()
                    if stock_locator.count()
                    else None
                ),
                "url": page.url,
            }
        finally:
            browser.close()


def main() -> None:
    try:
        product = scrape_dynamic_product(
            "https://store.example/products/demo-running-shoe"
        )
    except PlaywrightTimeoutError as exc:
        raise SystemExit(
            "The required product content did not become visible"
        ) from exc
    print(json.dumps(product, indent=2))


if __name__ == "__main__":
    main()

Wait for meaningful content—title visible, price element present, loading indicator gone, expected network response, or minimum product-card count—rather than a fixed sleep.

Prefer structured data in HTML, then static selectors, then approved page data responses, then browser rendering. Use the simplest method that reliably returns the required permitted data.

Handling product variants

Colors, sizes, packs, conditions, and sellers do not necessarily mean distinct products. Choose one row per product, per variant, or per seller offer based on the use case.

{
  "product_id": "shoe-model-100",
  "variant_id": "shoe-model-100-black-10",
  "sku": "DRS-100-BLK-10",
  "color": "Black",
  "size": "10",
  "price": 79.99,
  "currency": "USD",
  "availability": "in_stock"
}

Do not combine a price from one variant with the stock status of another. Variant selectors often update several parts of the page at once.

Location-dependent product data

Price, availability, delivery estimates, promotions, and seller offers may depend on country, ZIP code, store, currency, language, customer status, or collection time. Store location alongside the result—otherwise a later “price change” could actually be a location change.

Clean and normalize the data

Keep separate fields for price text, numeric amount, and currency. Map availability into an agreed classification while retaining original text. Normalize units and pack quantities before comparing prices. Preserve raw source values so normalization can be corrected later.

How to validate a product scrape

Evaluate URL coverage, required-field completeness, duplicates (canonical URL, SKU, GTIN, product/variant/seller keys—not names alone), range and format checks, run-over-time comparisons, and manual sample review after initial development or redesigns.

These figures are illustrative and are not Nenodata performance results. Replace with an approved run report before presenting as original evidence.

Common product-scraping problems

• No products returned — Save returned HTML and compare with the browser.
• Only the first page — Record every category page visited and reconcile counts.
• Missing prices — Define which price is required (variant, seller, location, promotion).
• Duplicates — Normalize canonical URLs and use stable identifiers.
• Variants overwrite one another — Assign separate variant IDs and attributes.
• Sudden breakage — Retain run logs, failed HTML, screenshots, and completeness alerts.
• Inflated row counts — Document the row model and report products, variants, offers, and observations separately.

Product-scraping methods compared

No single method is best for every project. A browser extension may suit 50 URLs collected once; it may be inefficient for a multi-country catalog refreshed daily into a warehouse.

When to build internally or use a managed service

An internal script may fit a stable, limited, temporary project with a simple schema and straightforward delivery—if your team can own maintenance. Account for development, testing, infrastructure, monitoring, validation, documentation, and staff continuity.

A managed service may be more practical when many sites must be covered, catalogs change often, JavaScript rendering or complex matching is required, prices vary by location, refreshes are recurring, failures need remediation, or output must feed APIs, warehouses, or BI tools.

Nenodata lists ecommerce product-catalog extraction, seller analytics, inventory tracking, review aggregation, price history, and category insights among its capabilities, with cleaning, normalization, validation, deduplication, and scheduled delivery. Exact coverage must be assessed per source. See ecommerce data extraction services and how Nenodata extracts and delivers data.

Responsible product-data collection

Before collecting data, consider public accessibility, authentication, applicable terms, personal data, copyright and database rights, request impact, intended use, jurisdictions, and whether permission or legal review is required. Public accessibility alone does not answer every question about permitted collection or reuse.

Turning product data into a recurring pipeline

1. Connect — approved sources, locations, page types, access conditions
2. Extract — product, variant, offer, and observation fields
3. Transform — clean, normalize, validate, deduplicate, map to schema
4. Deliver — CSV, Excel, JSON, API, webhook, database, warehouse, or dashboard

For price monitoring, preserve product identity, offer context, availability, location, and collection time. See ecommerce price scraping and competitor price intelligence.

Product-scraping requirements checklist

Before developing or commissioning a scraper, document:

• Sources — domains, example URLs, countries, stores, auth, exclusions
• Data — required/optional fields, identifiers, variant/seller/location rules
• Coverage — expected counts, pagination, variants, sellers, removed products
• Schedule — one-time or recurring, refresh frequency, maximum data age
• Quality — thresholds, duplicates, validation, failure handling, reconciliation
• Delivery — format, schema, time zone, schedule, retention
• Governance — intended use, source approval, restrictions, review ownership

How Nenodata can support product-data extraction

A basic script can suit a small, stable source. More complex requirements usually need discovery, schema planning, validation, scheduling, delivery, and maintenance. The practical starting point is to define target websites, example URLs, required fields, product and variant rules, locations, refresh frequency, expected output, and validation requirements.

Nenodata can then assess the source and propose a scoped workflow. Do not promise coverage, speed, accuracy, or real-time delivery until targets have been tested and approved.

Frequently asked questions