How to Scrape Data From a Website: A Practical Guide

What is website data scraping?

Web scraping retrieves content from webpages and converts selected information into structured records.

For example, an ecommerce scraper might collect:

• Product name, price, availability, seller, rating, and product URL

The output can be stored in CSV, Excel, JSON, a database, or delivered through an API. For recurring price collection, see our guide to ecommerce price scraping.

Scraping differs from crawling. Crawling discovers pages and links, while scraping extracts selected fields from those pages. Learn the difference between crawling and scraping.

Define the required output first

Do not begin by selecting a programming language or tool. Start by documenting what the final dataset must contain.

At minimum, define:

• Source website and example URLs
• Required fields, page types, and estimated volume
• Refresh frequency, output format, duplicate rules, missing-value rules, and completeness checks

This definition gives you a measurable result. A scraper that returns 7,200 records is not necessarily successful when the source contains 8,000 relevant products.

Check for an API or export

Before scraping, check whether the source already offers:

• A public or licensed API, CSV or Excel export, RSS or XML feed
• Reporting download, partner integration, data feed, or webhook

An official data source may offer documented fields and more stable access than parsing a webpage. However, an API may not expose every field visible on the site. Compare its coverage, restrictions, update frequency, and cost with your actual requirement.

Use web scraping only when it is an appropriate and permitted method for obtaining the required information.

Determine whether the page is static or dynamic

This decision determines whether a basic HTTP request is enough.

Static pages

On a static page, the required data is included in the HTML returned by the server. A Python script can usually request the page, parse the HTML, select elements, and save the fields.

Requests and Beautiful Soup are commonly used for this type of task. Requests supports HTTP response handling and explicit timeouts, while Beautiful Soup can select parsed elements using tags, attributes, and CSS selectors.

JavaScript-rendered pages

Some pages return a basic HTML shell and load visible data through JavaScript after the browser opens. A normal HTTP request may return a successful status code without containing the products, prices, or records visible on screen.

To diagnose this:

1. Open the page in a browser and view the original page source.
2. Compare it with the rendered DOM in developer tools.
3. Review relevant requests in the Network panel.
4. Check whether an official API provides the required data.

When the data appears only after rendering, browser automation such as Playwright may be needed. Do not assume that a network endpoint found in developer tools is an unrestricted public API.

Seven ways to scrape website data

Nenodata presents web scraping, custom data extraction, JavaScript rendering, data transformation, and file or API delivery as service areas. See professional website data extraction.

Python example for a static page

The following pattern demonstrates how a basic static-page scraper is structured. Before publishing or using it, replace the sample URL and selectors with a tested, approved demonstration source.

Install the libraries

pip install requests beautifulsoup4

Request and parse the page

from __future__ import annotations

import requests
from bs4 import BeautifulSoup

URL = "https://approved-demo-site.example/catalog"

try:
    response = requests.get(URL, timeout=20)
    response.raise_for_status()
except requests.RequestException as exc:
    raise SystemExit(f"Unable to retrieve the page: {exc}") from exc

soup = BeautifulSoup(response.text, "html.parser")

A successful response does not guarantee that the required data is present. Inspect response.text before writing extraction logic.

Extract fields

Assume each item has this simplified structure:

<article class="product-card">
  <h2 class="product-name">Example Product</h2>
  <span class="price">$49.00</span>
  <a class="details" href="/products/example-product">View product</a>
</article>

The extraction logic could be:

from urllib.parse import urljoin

records: list[dict[str, str]] = []

for card in soup.select("article.product-card"):
    name_node = card.select_one(".product-name")
    price_node = card.select_one(".price")
    link_node = card.select_one("a.details")

    if name_node is None or link_node is None:
        continue

    relative_url = link_node.get("href", "")

    records.append(
        {
            "name": name_node.get_text(" ", strip=True),
            "price": (
                price_node.get_text(" ", strip=True)
                if price_node
                else ""
            ),
            "url": urljoin(URL, relative_url),
        }
    )

The checks prevent one missing element from immediately crashing the entire run. In a real project, missing required fields should also be logged for review.

Export the result to CSV

import csv
from pathlib import Path

output_path = Path("products.csv")

with output_path.open("w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(
        file,
        fieldnames=["name", "price", "url"],
    )
    writer.writeheader()
    writer.writerows(records)

print(f"Saved {len(records)} records to {output_path}")

Validate the extracted records

Creating a file is not the end of the task. Check:

• Expected versus extracted record count
• Blank required fields, duplicate identifiers, invalid URLs, inconsistent prices
• Missing pages and second-run differences

A basic validation function could be:

def validate_records(records: list[dict[str, str]]) -> None:
    seen_urls: set[str] = set()

    for index, record in enumerate(records, start=1):
        if not record["name"]:
            raise ValueError(f"Record {index} has no name")

        if not record["url"]:
            raise ValueError(f"Record {index} has no URL")

        if record["url"] in seen_urls:
            raise ValueError(
                f"Duplicate URL found: {record['url']}"
            )

        seen_urls.add(record["url"])

Validation rules must reflect the business meaning of each field. Do not silently convert every missing value to zero.

Handle pagination

Pagination may use numbered pages, next-page links, offset parameters, cursors, load-more buttons, infinite scrolling, or JavaScript requests. Do not assume that adding ?page=2 will work.

For numbered pages, a simplified loop might look like:

import time

BASE_URL = "https://approved-demo-site.example/catalog?page={page}"

all_records: list[dict[str, str]] = []

for page_number in range(1, 11):
    page_url = BASE_URL.format(page=page_number)

    response = requests.get(page_url, timeout=20)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    cards = soup.select("article.product-card")

    if not cards:
        break

    for card in cards:
        name_node = card.select_one(".product-name")

        if name_node:
            all_records.append(
                {
                    "name": name_node.get_text(" ", strip=True),
                    "source_page": str(page_number),
                }
            )

    time.sleep(1)

Check whether featured records repeat across pages. Where possible, deduplicate using a stable source identifier rather than the product name.

Scrape dynamic content with Playwright

A simplified browser-automation pattern looks like this:

from playwright.sync_api import sync_playwright

URL = "https://approved-demo-site.example/dynamic-catalog"

with sync_playwright() as playwright:
    browser = playwright.chromium.launch(headless=True)
    page = browser.new_page()

    page.goto(
        URL,
        wait_until="domcontentloaded",
        timeout=30_000,
    )

    cards = page.locator("article.product-card")
    cards.first.wait_for(state="visible")

    records: list[dict[str, str]] = []

    for index in range(cards.count()):
        card = cards.nth(index)

        records.append(
            {
                "name": card.locator(
                    ".product-name"
                ).inner_text(),
                "price": card.locator(
                    ".price"
                ).inner_text(),
            }
        )

    browser.close()

For a real page, wait for a specific element, a known record count, a loading indicator to disappear, or a load-more action to complete. Avoid relying on a long fixed sleep as the main synchronization method.

Clean and structure the data

Raw webpage text often requires additional processing.

Normalize text

def normalize_text(value: str) -> str:
    return " ".join(value.split())

Separate prices into useful fields

Instead of storing only $1,299.00, consider:

{
  "price_amount": 1299.00,
  "price_currency": "USD",
  "price_display": "$1,299.00"
}

Do not infer a currency solely from its symbol when the source serves several countries.

Preserve missing-value meaning

Price unavailable, price equal to zero, field not displayed, extraction failed, and product out of stock are not equivalent. Store enough context to distinguish them.

Use reliable duplicate keys

Product ID, SKU, listing ID, canonical URL, or source record ID are stronger keys than product names alone.

Add collection metadata

For recurring extraction, include source URL, collection timestamp, region, run ID, scraper version, and validation status.

Choose the right output format

Nenodata's workflow page lists files, APIs, and webhooks among delivery options. See how Nenodata processes website data. Confirm exact delivery timing and availability for your project during scoping.

Common scraping problems

Requests documentation confirms that requests do not time out unless a timeout is explicitly set.

Collect website data responsibly

A publicly viewable page does not automatically mean that every collection or reuse scenario is permitted.

Before collecting data:

• Review applicable terms and policies; prefer an official API or feed where appropriate
• Respect authentication and access controls; avoid private or restricted information
• Use reasonable request rates; collect only fields needed for the stated purpose
• Apply suitable data-protection controls; obtain legal advice for high-risk use cases

A robots.txt file mainly instructs search-engine crawlers about URL access. Google states that it is not a mechanism for keeping a page out of search results. It should not be treated as a complete legal determination for an unrelated data-collection project.

This section is operational guidance, not legal advice.

When a basic scraper is no longer enough

A small script may be suitable when one static source is involved, the dataset is limited, collection is occasional, the structure is stable, and missing records have low business impact.

A production workflow may also require:

• Page discovery, multiple template handling, retries, and duplicate checks
• Completeness validation, run history, change monitoring, scheduled delivery, and failure alerts

This distinction matters when scraped data feeds pricing, research, sales, inventory, or operational decisions. Read enterprise web scraping considerations.

Should you build or outsource the workflow?

Build internally when the source is simple, the scale is limited, and your team can maintain the code.

Consider managed extraction when:

• Several changing websites are involved and updates must run on a fixed schedule
• Data must be normalized across sources with API delivery into internal systems
• Failed or incomplete records create business risk and engineering maintenance is becoming a distraction

Before choosing a provider, ask:

• How is completeness measured and how are failed pages reported?
• How are layout changes detected and which fields are validated?
• Which output formats are supported and who owns monitoring?

Nenodata's live site describes web scraping, JavaScript rendering, data transformation, and structured delivery as current offerings. Specific project feasibility and performance must be confirmed during scoping.

Website scraping checklist

Before launching a scraper, confirm:

• ☐Required fields are documented.
• ☐Example URLs have been reviewed.
• ☐An API or export has been considered.
• ☐Static versus dynamic behavior has been checked.
• ☐Pagination has been identified.
• ☐Expected record counts are known.
• ☐Missing-value rules are defined.
• ☐Duplicate rules are defined.
• ☐Output format is confirmed.
• ☐Timeouts and error handling are configured.
• ☐A second run has been compared with the first.
• ☐Access requirements have been reviewed.
• ☐Someone owns ongoing maintenance.

Final takeaway

The best website-scraping method depends on the source, required volume, update frequency, and consequences of missing data.

Use a basic HTTP scraper for suitable static pages. Use browser automation when public content genuinely depends on JavaScript rendering. In every case, validate record counts, missing fields, duplicates, and repeat-run consistency.

When the requirement involves multiple sources, scheduled updates, structured delivery, or ongoing monitoring, evaluate the full maintenance workload rather than only the initial extraction code.