How to Scrape Products From a Website

Start with the dataset, not the website

A common mistake is opening developer tools before deciding what the finished dataset should contain. Start by answering:

1. 01

   Which pages must be covered?

2. 02

   Which fields are required?

3. 03

   What does one row represent?

4. 04

   How will the data be used?

Product, variant, offer, and observation

These entities should not be treated as interchangeable. A product is the general item; a variant is a configuration such as size and color; an offer is a seller-specific listing; an observation is the information collected at a particular time and location.

What product fields should you extract?

The correct fields depend on the business objective. Common groups include identity (name, brand, SKU, GTIN, product ID, URL), classification, pricing, availability and fulfillment, attributes, marketplace information, customer-feedback fields, and collection metadata such as timestamp, country, store, scrape status, and error status.

• Identity

  • Name
  • Brand
  • SKU
  • GTIN
  • Product ID
  • URL

• Classification

  • Category

• Pricing

  • Current price
  • Original price
  • Currency

• Availability

  • Availability
  • Fulfillment

• Attributes

  • Description
  • Specifications
  • Variants
  • Image URL

• Marketplace

  • Seller information

• Customer Feedback

  • Rating
  • Review count

• Collection Metadata

  • Timestamp
  • Country
  • Store
  • Scrape status
  • Error status

Do not collect every visible field simply because it is available. Extra fields increase extraction complexity, storage, validation, and maintenance.

How to scrape one product page with Python

The following example uses fictional selectors. Replace them only after inspecting a permitted target page.

Install the libraries

pip install requests beautifulsoup4

Basic scraper

from __future__ import annotations

import json
from dataclasses import asdict, dataclass
from decimal import Decimal, InvalidOperation
from typing import Optional

import requests
from bs4 import BeautifulSoup


@dataclass
class Product:
    name: str
    url: str
    sku: Optional[str]
    price: Optional[str]
    availability: Optional[str]


def clean_text(value: Optional[str]) -> Optional[str]:
    if value is None:
        return None
    cleaned = " ".join(value.split())
    return cleaned or None


def parse_price(value: Optional[str]) -> Optional[str]:
    if not value:
        return None
    numeric = (
        value.replace(",", "")
        .replace("$", "")
        .replace("€", "")
        .replace("£", "")
        .strip()
    )
    try:
        return str(Decimal(numeric))
    except InvalidOperation:
        return None


def scrape_product(url: str) -> Product:
    headers = {
        "User-Agent": (
            "Mozilla/5.0 (compatible; ProductResearchBot/1.0; "
            "+https://your-company.example/bot-information)"
        )
    }
    response = requests.get(url, headers=headers, timeout=30)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")

    name_element = soup.select_one("h1.product-name")
    sku_element = soup.select_one("[data-product-sku]")
    price_element = soup.select_one(".product-price")
    availability_element = soup.select_one(".availability")

    if name_element is None:
        raise ValueError("Required product name was not found")

    return Product(
        name=clean_text(name_element.get_text()) or "",
        url=url,
        sku=(
            clean_text(sku_element.get("data-product-sku"))
            if sku_element
            else None
        ),
        price=parse_price(
            clean_text(price_element.get_text())
            if price_element
            else None
        ),
        availability=(
            clean_text(availability_element.get_text())
            if availability_element
            else None
        ),
    )


def main() -> None:
    product_url = "https://store.example/products/demo-running-shoe"
    try:
        product = scrape_product(product_url)
    except requests.RequestException as exc:
        raise SystemExit(f"Request failed: {exc}") from exc
    except ValueError as exc:
        raise SystemExit(f"Extraction failed: {exc}") from exc
    print(json.dumps(asdict(product), indent=2))


if __name__ == "__main__":
    main()

A possible result would be:

{
  "name": "Demo Running Shoe",
  "url": "https://store.example/products/demo-running-shoe",
  "sku": "DRS-100-BLK",
  "price": "79.99",
  "availability": "In stock"
}

This output is illustrative. It is not evidence from a live Nenodata extraction project.

The script can extract one permitted static page, but it does not yet find other product URLs, process pagination, run JavaScript, extract variants, retry failures, normalize currencies, detect duplicates, validate catalog completeness, monitor layout changes, or save historical observations.

Extract JSON-LD before relying on CSS selectors

CSS selectors can change when a website redesigns. Structured product data may provide a cleaner source when it is present and complete.

from __future__ import annotations

import json
from typing import Any, Optional

from bs4 import BeautifulSoup


def find_product_json_ld(html: str) -> Optional[dict[str, Any]]:
    soup = BeautifulSoup(html, "html.parser")
    scripts = soup.select('script[type="application/ld+json"]')

    for script in scripts:
        raw_json = script.string
        if not raw_json:
            continue
        try:
            data = json.loads(raw_json)
        except json.JSONDecodeError:
            continue

        candidates = data if isinstance(data, list) else [data]
        for candidate in candidates:
            if not isinstance(candidate, dict):
                continue
            if candidate.get("@type") == "Product":
                return candidate
            graph = candidate.get("@graph", [])
            if isinstance(graph, list):
                for item in graph:
                    if isinstance(item, dict) and item.get("@type") == "Product":
                        return item
    return None

Do not assume JSON-LD contains every required field. Compare it with the visible page and confirm whether prices, variants, stock, and sellers are represented correctly.

Export product data to CSV

import csv
from dataclasses import asdict
from pathlib import Path
from typing import Iterable


def save_products_to_csv(
    products: Iterable[Product],
    output_path: str,
) -> None:
    rows = [asdict(product) for product in products]
    if not rows:
        raise ValueError("No products were provided")

    path = Path(output_path)
    with path.open("w", newline="", encoding="utf-8-sig") as file:
        writer = csv.DictWriter(file, fieldnames=list(rows[0].keys()))
        writer.writeheader()
        writer.writerows(rows)

For recurring monitoring, add fields such as collected_at, currency, location, seller, variant ID, and scrape status. Without timestamps and context, a price record cannot reliably support historical analysis.

How to scrape JavaScript-rendered products

Install Playwright and browsers:

pip install playwright
playwright install

from __future__ import annotations

import json

from playwright.sync_api import (
    TimeoutError as PlaywrightTimeoutError,
    sync_playwright,
)


def scrape_dynamic_product(url: str) -> dict[str, str | None]:
    with sync_playwright() as playwright:
        browser = playwright.chromium.launch(headless=True)
        try:
            page = browser.new_page(viewport={"width": 1440, "height": 900})
            page.goto(url, wait_until="domcontentloaded", timeout=45_000)
            page.locator("h1.product-name").wait_for(state="visible", timeout=15_000)

            name = page.locator("h1.product-name").inner_text()
            price_locator = page.locator(".product-price")
            stock_locator = page.locator(".availability")

            return {
                "name": name.strip(),
                "price": (
                    price_locator.inner_text().strip()
                    if price_locator.count()
                    else None
                ),
                "availability": (
                    stock_locator.inner_text().strip()
                    if stock_locator.count()
                    else None
                ),
                "url": page.url,
            }
        finally:
            browser.close()


def main() -> None:
    try:
        product = scrape_dynamic_product(
            "https://store.example/products/demo-running-shoe"
        )
    except PlaywrightTimeoutError as exc:
        raise SystemExit(
            "The required product content did not become visible"
        ) from exc
    print(json.dumps(product, indent=2))


if __name__ == "__main__":
    main()

Wait for meaningful content—title visible, price element present, loading indicator gone, expected network response, or minimum product-card count—rather than a fixed sleep.

Prefer structured data in HTML, then static selectors, then approved page data responses, then browser rendering. Use the simplest method that reliably returns the required permitted data.

Handling product variants

Colors, sizes, packs, conditions, and sellers do not necessarily mean distinct products. Choose one row per product, per variant, or per seller offer based on the use case.

{
  "product_id": "shoe-model-100",
  "variant_id": "shoe-model-100-black-10",
  "sku": "DRS-100-BLK-10",
  "color": "Black",
  "size": "10",
  "price": 79.99,
  "currency": "USD",
  "availability": "in_stock"
}

Do not combine a price from one variant with the stock status of another. Variant selectors often update several parts of the page at once.

Location-dependent product data

Price, availability, delivery estimates, promotions, and seller offers may depend on country, ZIP code, store, currency, language, customer status, or collection time. Store location alongside the result—otherwise a later “price change” could actually be a location change.

Clean and normalize the data

Keep separate fields for price text, numeric amount, and currency. Map availability into an agreed classification while retaining original text. Normalize units and pack quantities before comparing prices. Preserve raw source values so normalization can be corrected later.

Common product-scraping problems

• No products returned

  Save returned HTML and compare with the browser.

• Only the first page

  Record every category page visited and reconcile counts.

• Missing prices

  Define which price is required (variant, seller, location, promotion).

• Duplicates

  Normalize canonical URLs and use stable identifiers.

• Variants overwrite one another

  Assign separate variant IDs and attributes.

• Sudden breakage

  Retain run logs, failed HTML, screenshots, and completeness alerts.

• Inflated row counts

  Document the row model and report products, variants, offers, and observations separately.

Product-scraping methods compared

• Manual copy and paste

  Setup:

  Low initially

  Skill:

  Low

  Coverage:

  Poor

  Maintenance:

  High manual effort

  Best for:

  A few records

• Browser extension

  Setup:

  Low

  Skill:

  Low

  Coverage:

  Limited to moderate

  Maintenance:

  User-managed

  Best for:

  Small, occasional projects

• No-code scraper

  Setup:

  Low to moderate

  Skill:

  Low

  Coverage:

  Moderate

  Maintenance:

  Tool and user-managed

  Best for:

  Standard page structures

• Custom Python script

  Setup:

  Moderate

  Skill:

  Medium to high

  Coverage:

  Moderate to high

  Maintenance:

  Internal team

  Best for:

  Controlled technical projects

• Scraping API

  Setup:

  Moderate

  Skill:

  Medium

  Coverage:

  High where supported

  Maintenance:

  Shared with provider

  Best for:

  Teams wanting programmatic access

• Managed extraction service

  Setup:

  Higher scoping effort

  Skill:

  Low customer coding requirement

  Coverage:

  Designed around project scope

  Maintenance:

  Provider-managed

  Best for:

  Recurring or complex requirements

• Hybrid workflow

  Setup:

  Variable

  Skill:

  Medium

  Coverage:

  High

  Maintenance:

  Shared

  Best for:

  Teams needing standard and custom coverage

No single method is best for every project. A browser extension may suit 50 URLs collected once; it may be inefficient for a multi-country catalog refreshed daily into a warehouse.

When to build internally or use a managed service

An internal script may fit a stable, limited, temporary project with a simple schema and straightforward delivery—if your team can own maintenance. Account for development, testing, infrastructure, monitoring, validation, documentation, and staff continuity.

A managed service may be more practical when many sites must be covered, catalogs change often, JavaScript rendering or complex matching is required, prices vary by location, refreshes are recurring, failures need remediation, or output must feed APIs, warehouses, or BI tools.

Nenodata lists ecommerce product-catalog extraction, seller analytics, inventory tracking, review aggregation, price history, and category insights among its capabilities, with cleaning, normalization, validation, deduplication, and scheduled delivery. Exact coverage must be assessed per source. See ecommerce data extraction services and how Nenodata extracts and delivers data.

Responsible product-data collection

Before collecting data, consider public accessibility, authentication, applicable terms, personal data, copyright and database rights, request impact, intended use, jurisdictions, and whether permission or legal review is required. Public accessibility alone does not answer every question about permitted collection or reuse.

Product-scraping requirements checklist

Before developing or commissioning a scraper, document:

Frequently asked questions

Sources used

• Beautiful Soup documentation — HTML/XML parsing and element selection
• Playwright for Python documentation — installation, browsers, and automation APIs
• Schema.org Product — structured vocabulary for products
• Nenodata services, How It Works, Ecommerce Price Scraping, and Price Intelligence