What is website data scraping?
Web scraping retrieves content from webpages and converts selected information into structured records.
For example, an ecommerce scraper might collect:
- Product name, price, availability, seller, rating, and product URL
Webpage
- Product name
- Price
- Availability
- Seller
- Rating
- URL
Scraping
Selects required information
Structured record
CSV · Excel · JSON · Database · API
The output can be stored in CSV, Excel, JSON, a database, or delivered through an API. For recurring price collection, see our guide to ecommerce price scraping.
Crawling
Discovers pages and links
Website → category → page → page
Scraping
Extracts selected fields
Product page → name / price / URL
Scraping differs from crawling. Crawling discovers pages and links, while scraping extracts selected fields from those pages. Learn the difference between crawling and scraping.
Define the required output first
At minimum, define:
- Source website and example URLs
- Required fields, page types, and estimated volume
- Refresh frequency, output format, duplicate rules, missing-value rules, and completeness checks
Source
Example URLs
Schema
SKU / Name / Price / Availability / URL
Volume
Expected record count
Schedule
Refresh frequency
Quality
Duplicate + missing-value rules
Delivery
CSV / API
| Requirement | Example |
|---|---|
| Page type | Product-detail pages |
| Required fields | SKU, name, price, availability, URL |
| Estimated volume | 8,000 products |
| Refresh frequency | Daily |
| Output | CSV and API |
| Duplicate rule | One record per source product ID |
| Missing price | Keep the record and mark the price unavailable |
This definition gives you a measurable result. A scraper that returns 7,200 records is not necessarily successful when the source contains 8,000 relevant products.
Use the checklist at the end of this guide to confirm requirements before you choose a tool or write code.
View the ChecklistCheck for an API or export
Before scraping, check whether the source already offers:
- A public or licensed API, CSV or Excel export, RSS or XML feed
- Reporting download, partner integration, data feed, or webhook
Before scraping, check for
- API
- CSV / Excel
- RSS / XML
- Partner integration
- Data feed
- Reporting export
- Webhook
Official data source available? Yes → Evaluate its field coverage, restrictions, freshness and cost
No / insufficient coverage → Evaluate appropriate web extraction
An official data source may offer documented fields and more stable access than parsing a webpage. However, an API may not expose every field visible on the site. Compare its coverage, restrictions, update frequency, and cost with your actual requirement.
Determine whether the page is static or dynamic
This decision determines whether a basic HTTP request is enough.
Static pages
- Server
- HTML already contains data
- Requests
- Beautiful Soup
- Structured record
JavaScript-rendered pages
- Server
- Basic HTML shell
- Browser executes JavaScript
- Visible data appears
- Playwright / browser automation
Static pages
On a static page, the required data is included in the HTML returned by the server. A Python script can usually request the page, parse the HTML, select elements, and save the fields.
Requests and Beautiful Soup are commonly used for this type of task. Requests supports HTTP response handling and explicit timeouts, while Beautiful Soup can select parsed elements using tags, attributes, and CSS selectors.
JavaScript-rendered pages
Some pages return a basic HTML shell and load visible data through JavaScript after the browser opens. A normal HTTP request may return a successful status code without containing the products, prices, or records visible on screen.
To diagnose this:
- Open the page in a browser and view the original page source.
- Compare it with the rendered DOM in developer tools.
- Review relevant requests in the Network panel.
- Check whether an official API provides the required data.
01
View original page source
02
Compare with rendered DOM
03
Inspect relevant Network requests
04
Check for an official API
When the data appears only after rendering, browser automation such as Playwright may be needed. Do not assume that a network endpoint found in developer tools is an unrestricted public API.
Seven ways to scrape website data
Manual collection
Best use: A few one-time records
Main limitation: Slow and inconsistent
Spreadsheet import
Best use: Simple public tables
Main limitation: Limited dynamic-page support
Browser extension
Best use: Small lists with repeated layouts
Main limitation: Fragile on complex pagination
No-code tool
Best use: Visual setup without programming
Main limitation: Platform limits and recurring cost
Python HTTP scraper
Best use: Static HTML and custom processing
Main limitation: Does not render browser JavaScript
Browser automation
Best use: Dynamic and interactive pages
Main limitation: Higher maintenance and resource use
Managed extraction
Best use: Recurring or business-critical collection
Main limitation: Requires scope and budget
Method path — may fit
- How much data? Very small / one-time → Manual
- Is the source a simple public table? Yes → Spreadsheet
- Need simple point-and-click extraction? → Browser extension / no-code
- Is the data present in raw HTML? Yes → Python + Requests / Beautiful Soup
- Does content require JavaScript or interaction? Yes → Browser automation
- Is extraction recurring, multi-source or business-critical? → Evaluate managed extraction
01
1. Manual collection
Manual copy and paste can be acceptable when only a few records are needed, the task will not be repeated, and the source uses a simple table. It becomes unsuitable when the data spans many pages or needs regular updates.
02
2. Spreadsheet imports
Excel and Google Sheets can import some public HTML tables or feeds. This approach often fails with JavaScript-rendered content, infinite scroll, nested product cards, location-based results, complex pagination, and required user interactions.
03
3. Browser extensions
Browser extensions let users select visible elements and export repeated records without coding. Confirm whether the extension collected every page, records loaded after scrolling, optional fields, correct URLs, and unique rows. A correct-looking preview does not prove completeness.
04
4. No-code and AI-assisted tools
No-code tools can identify repeated page structures and schedule extraction. Test total record count, pagination, missing values, duplicate rows, export structure, second-run consistency, and error reporting. AI-assisted field detection may still choose the wrong element or miss alternate layouts.
05
5. Python with Requests and Beautiful Soup
Suitable when data is present in returned HTML, you need custom cleaning rules, the structure is reasonably consistent, and someone can maintain the code. Requests does not apply a timeout unless one is provided, so production code should set an explicit timeout and handle request errors.
06
6. Browser automation with Playwright
Playwright controls a real browser environment, allowing JavaScript to execute before extraction. Use it when content appears after rendering, clicking or scrolling loads records, or client-side pagination is used. Playwright locators support automatic waiting—prefer meaningful page attributes over fragile nested selectors.
07
7. Managed data extraction
Appropriate when thousands of pages must be collected, multiple websites must be combined, data must be refreshed on a schedule, records need cleaning or deduplication, or internal engineering capacity is limited.
Nenodata presents web scraping, custom data extraction, JavaScript rendering, data transformation, and file or API delivery as service areas. See professional website data extraction.
Python example for a static page
The following pattern demonstrates how a basic static-page scraper is structured. Before publishing or using it, replace the sample URL and selectors with a tested, approved demonstration source.
Install the libraries
Bash
pip install requests beautifulsoup4Request and parse the page
Python
from __future__ import annotations
import requests
from bs4 import BeautifulSoup
URL = "https://approved-demo-site.example/catalog"
try:
response = requests.get(URL, timeout=20)
response.raise_for_status()
except requests.RequestException as exc:
raise SystemExit(f"Unable to retrieve the page: {exc}") from exc
soup = BeautifulSoup(response.text, "html.parser")Extract fields
Assume each item has this simplified structure:
HTML
<article class="product-card">
<h2 class="product-name">Example Product</h2>
<span class="price">$49.00</span>
<a class="details" href="/products/example-product/">View product</a>
</article>The extraction logic could be:
Python
from urllib.parse import urljoin
records: list[dict[str, str]] = []
for card in soup.select("article.product-card"):
name_node = card.select_one(".product-name")
price_node = card.select_one(".price")
link_node = card.select_one("a.details")
if name_node is None or link_node is None:
continue
relative_url = link_node.get("href", "")
records.append(
{
"name": name_node.get_text(" ", strip=True),
"price": (
price_node.get_text(" ", strip=True)
if price_node
else ""
),
"url": urljoin(URL, relative_url),
}
)Export the result to CSV
Python
import csv
from pathlib import Path
output_path = Path("products.csv")
with output_path.open("w", newline="", encoding="utf-8") as file:
writer = csv.DictWriter(
file,
fieldnames=["name", "price", "url"],
)
writer.writeheader()
writer.writerows(records)
print(f"Saved {len(records)} records to {output_path}")Validate the extracted records
Creating a file is not the end of the task. Check:
- Expected versus extracted record count
- Blank required fields, duplicate identifiers, invalid URLs, inconsistent prices
- Missing pages and second-run differences
Raw extracted records
- Record count
- Required fields
- Duplicate check
- URL validation
- Value consistency
- Repeat run
Validated dataset
Basic validation example
A basic validation function could be:
Python
def validate_records(records: list[dict[str, str]]) -> None:
seen_urls: set[str] = set()
for index, record in enumerate(records, start=1):
if not record["name"]:
raise ValueError(f"Record {index} has no name")
if not record["url"]:
raise ValueError(f"Record {index} has no URL")
if record["url"] in seen_urls:
raise ValueError(
f"Duplicate URL found: {record['url']}"
)
seen_urls.add(record["url"])Handle pagination
Pagination may use numbered pages, next-page links, offset parameters, cursors, load-more buttons, infinite scrolling, or JavaScript requests. Do not assume that adding ?page=2 will work.
- Numbered pages
- Next page
- Offset
- Cursor
- Load more
- Infinite scroll
- JavaScript requests
Page 1 → Page 2 → Page 3 → … → stop when no records remain
For numbered pages, a simplified loop might look like:
Python
import time
BASE_URL = "https://approved-demo-site.example/catalog?page={page}"
all_records: list[dict[str, str]] = []
for page_number in range(1, 11):
page_url = BASE_URL.format(page=page_number)
response = requests.get(page_url, timeout=20)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
cards = soup.select("article.product-card")
if not cards:
break
for card in cards:
name_node = card.select_one(".product-name")
if name_node:
all_records.append(
{
"name": name_node.get_text(" ", strip=True),
"source_page": str(page_number),
}
)
time.sleep(1)Check whether featured records repeat across pages. Where possible, deduplicate using a stable source identifier rather than the product name.
Scrape dynamic content with Playwright
A simplified browser-automation pattern looks like this:
- Open browser
- Load JavaScript
- Wait for meaningful element
- Locate cards
- Extract fields
- Close browser
Python
from playwright.sync_api import sync_playwright
URL = "https://approved-demo-site.example/dynamic-catalog"
with sync_playwright() as playwright:
browser = playwright.chromium.launch(headless=True)
page = browser.new_page()
page.goto(
URL,
wait_until="domcontentloaded",
timeout=30_000,
)
cards = page.locator("article.product-card")
cards.first.wait_for(state="visible")
records: list[dict[str, str]] = []
for index in range(cards.count()):
card = cards.nth(index)
records.append(
{
"name": card.locator(
".product-name"
).inner_text(),
"price": card.locator(
".price"
).inner_text(),
}
)
browser.close()Clean and structure the data
Raw webpage text often requires additional processing.
Normalize text
Python
def normalize_text(value: str) -> str:
return " ".join(value.split())Separate prices into useful fields
Instead of storing only $1,299.00, consider:
JSON
{
"price_amount": 1299.00,
"price_currency": "USD",
"price_display": "$1,299.00"
}Do not infer a currency solely from its symbol when the source serves several countries.
Preserve missing-value meaning
Price unavailable, price equal to zero, field not displayed, extraction failed, and product out of stock are not equivalent. Store enough context to distinguish them.
These are not equivalent
- Price = 0
- Price unavailable
- Field absent
- Extraction failure
- Out of stock
Use reliable duplicate keys
Product ID, SKU, listing ID, canonical URL, or source record ID are stronger keys than product names alone.
Add collection metadata
For recurring extraction, include source URL, collection timestamp, region, run ID, scraper version, and validation status.
- Source URL
- Timestamp
- Region
- Run ID
- Scraper version
- Validation status
Choose the right output format
CSV
Best use: Flat data and simple transfer
Main limitation: Does not represent nested data well
Excel
Best use: Manual review and business analysis
Main limitation: Less suitable for large automation workflows
JSON
Best use: Applications and nested records
Main limitation: Less convenient for spreadsheet users
Database
Best use: Large recurring datasets
Main limitation: Requires database management
API
Best use: System-to-system delivery
Main limitation: Requires endpoint and access management
Nenodata's workflow page lists files, APIs, and webhooks among delivery options. See how Nenodata processes website data. Confirm exact delivery timing and availability for your project during scoping.
Common scraping problems
Visible data is absent from the response
Likely cause: JavaScript rendering
Next step: Compare raw HTML with the rendered DOM
Only the first page is collected
Likely cause: Pagination was not handled
Next step: Identify the real page or cursor mechanism
Duplicate rows appear
Likely cause: Page overlap or repeated featured records
Next step: Deduplicate using a stable identifier
Some fields are blank
Likely cause: Multiple layouts or optional values
Next step: Inspect varied records and log missing fields
The script breaks after a redesign
Likely cause: Fragile selectors
Next step: Use meaningful selectors and validation
The request hangs
Likely cause: Missing timeout or unresponsive source
Next step: Add explicit timeouts and error handling
Access errors appear
Likely cause: Rate limits or access restrictions
Next step: Stop and review permitted access options
Requests documentation confirms that requests do not time out unless a timeout is explicitly set.
Operational guidance
Collect website data responsibly
A publicly viewable page does not automatically mean that every collection or reuse scenario is permitted.
Before collecting data:
- Review applicable terms and policies; prefer an official API or feed where appropriate
- Respect authentication and access controls; avoid private or restricted information
- Use reasonable request rates; collect only fields needed for the stated purpose
- Apply suitable data-protection controls; obtain legal advice for high-risk use cases
A robots.txt file mainly instructs search-engine crawlers about URL access. Google states that it is not a mechanism for keeping a page out of search results. It should not be treated as a complete legal determination for an unrelated data-collection project.
This section is operational guidance, not legal advice.
When a basic scraper is no longer enough
A small script may be suitable when one static source is involved, the dataset is limited, collection is occasional, the structure is stable, and missing records have low business impact.
Prototype
- One source
- Occasional extraction
- Stable template
- Small dataset
Production workflow
- Multiple templates
- Retries
- Deduplication
- Validation
- Monitoring
- Scheduled delivery
- Failure alerts
- Discovery
- Retrieval
- Extraction
- Normalization
- Deduplication
- Validation
- Delivery
- Monitoring
A production workflow may also require:
- Page discovery, multiple template handling, retries, and duplicate checks
- Completeness validation, run history, change monitoring, scheduled delivery, and failure alerts
This distinction matters when scraped data feeds pricing, research, sales, inventory, or operational decisions. Read enterprise web scraping considerations.
Should you build or outsource the workflow?
Build internally
Build internally when the source is simple, the scale is limited, and your team can maintain the code.
Consider managed extraction
Consider managed extraction when:
- Several changing websites are involved and updates must run on a fixed schedule
- Data must be normalized across sources with API delivery into internal systems
- Failed or incomplete records create business risk and engineering maintenance is becoming a distraction
Before choosing a provider, ask:
- How is completeness measured and how are failed pages reported?
- How are layout changes detected and which fields are validated?
- Which output formats are supported and who owns monitoring?
Nenodata's live site describes web scraping, JavaScript rendering, data transformation, and structured delivery as current offerings. Specific project feasibility and performance must be confirmed during scoping.
Have a recurring extraction requirement?
Share one public target URL, the fields you need, and your preferred output format. Nenodata can review source feasibility and discuss a representative sample before a recurring workflow is configured.
- Public target URL
- Required fields
- Preferred output format
Website scraping checklist
Before launching a scraper, confirm:
Pre-launch website scraping checklist
- Required fields are documented.
- Example URLs have been reviewed.
- An API or export has been considered.
- Static versus dynamic behavior has been checked.
- Pagination has been identified.
- Expected record counts are known.
- Missing-value rules are defined.
- Duplicate rules are defined.
- Output format is confirmed.
- Timeouts and error handling are configured.
- A second run has been compared with the first.
- Access requirements have been reviewed.
- Someone owns ongoing maintenance.
Final takeaway
Choose by page behavior
Static → HTTP scraper
JavaScript-dependent → Browser automation
Always validate
Counts, missing fields, duplicates, repeat-run consistency
Evaluate operations
Multiple sources, scheduled updates, structured delivery, monitoring
The best website-scraping method depends on the source, required volume, update frequency, and consequences of missing data.
Use a basic HTTP scraper for suitable static pages. Use browser automation when public content genuinely depends on JavaScript rendering. In every case, validate record counts, missing fields, duplicates, and repeat-run consistency.
When the requirement involves multiple sources, scheduled updates, structured delivery, or ongoing monitoring, evaluate the full maintenance workload rather than only the initial extraction code.
Need help scoping a recurring extraction workflow?
Talk to a Nenodata specialist about your source, schema, and delivery requirements.
- Source
- Schema
- Delivery