Table of Contents

How to Scrape Emails From Multiple Websites

Scraping email addresses from multiple websites involves building a process that can visit a list of permitted websites, locate relevant pages, identify publicly displayed email addresses, clean the results, remove duplicates, and save everything in a structured database or spreadsheet.

Doing this across multiple domains is considerably more complicated than scraping one website. The main challenges are URL management, crawling, different website structures, rate control, JavaScript-rendered content, data cleaning, deduplication, and maintaining a reliable record of where every address came from. At larger scale, concurrency and bookkeeping become major parts of the problem.

This guide focuses on legitimate research and business use of publicly available information from websites you are permitted to crawl.

What Does Scraping Emails From Multiple Websites Mean?

Instead of processing one website:

Website A
   ↓
Find emails

you process a list:

Website A ──┐
Website B ──┤
Website C ──┤
Website D ──┤──→ Email Extraction
Website E ──┤
Website F ──┘

For example, your input file might contain:

company-a.com
company-b.com
company-c.com
company-d.com
company-e.com

The scraper processes each domain independently and produces something like:

Website	Email	Type	Source Page
company-a.com	info@company-a.com	General	/contact
company-a.com	sales@company-a.com	Sales	/sales
company-b.com	hello@company-b.com	General	/
company-c.com	press@company-c.com	Press	/press

Why Multiple-Website Scraping Is More Difficult

Scraping 10 websites manually is relatively straightforward.

Scraping 10,000 websites creates several new problems.

1. Different website structures

One website may use:

/contact

another:

/contact-us

another:

get-in-touch

and another may have no dedicated contact page.

2. Different technologies

You may encounter:

Static HTML
WordPress
React
Vue
Angular
JavaScript applications
Server-rendered websites
Embedded forms
PDFs
Dynamically loaded content

A basic HTTP scraper may not see content generated after JavaScript executes.

3. Different access rules

Every domain can have different:

robots.txt rules
Terms of service
Rate limits
Crawl restrictions
Server configurations

A multi-domain crawler should therefore treat each website separately rather than assuming one global policy applies to every domain

The Basic Multi-Website Workflow

A reliable process looks like this:

Website List
     ↓
Validate URLs
     ↓
Check Website Rules
     ↓
Create Crawl Queue
     ↓
Visit Homepage
     ↓
Discover Relevant Pages
     ↓
Extract Emails
     ↓
Clean Results
     ↓
Deduplicate
     ↓
Verify
     ↓
Store Source Information
     ↓
Export

For larger projects:

URL Database
     ↓
Crawl Queue
     ↓
Multiple Workers
     ↓
Domain-Level Rate Limiting
     ↓
HTML / Browser Extraction
     ↓
Email Parser
     ↓
Data Cleaning
     ↓
Database

Step 1: Prepare Your Website List

Start with a CSV or spreadsheet.

Example:

ID	Company	Website
1	Company A	company-a.com
2	Company B	company-b.com
3	Company C	company-c.com
4	Company D	company-d.com

You can also store:

Country
Industry
Category
Priority
Date added
Processing status

A larger file might look like:

id
company
domain
industry
country
priority
status

Step 2: Normalize Website URLs

Users may provide websites in different formats:

company.com
www.company.com
https://company.com
https://www.company.com/

Your system should normalize these into a consistent representation.

For example:

https://company.com

This makes it easier to:

Identify duplicates
Group pages by domain
Track crawling
Store results

Step 3: Remove Duplicate Domains

Your input might contain:

company.com
www.company.com
https://company.com
https://company.com/

These are likely the same website.

Normalize first and then deduplicate.

Step 4: Check the Website’s Crawling Rules

Before beginning the crawl, check:

https://example.com/robots.txt

robots.txt communicates instructions to automated crawlers about which areas should or shouldn’t be crawled. It is important to follow applicable instructions and also review the site’s terms

For a large project, store the result:

Domain
robots_checked
crawl_allowed
checked_at

For example:

Domain	Robots Checked	Status
company-a.com	Yes	Allowed
company-b.com	Yes	Restricted
company-c.com	Yes	Allowed

Step 5: Check Terms and Conditions

robots.txt isn’t the only consideration.

A website’s terms may contain restrictions concerning:

Automated access
Data extraction
Commercial use
Reproduction
Database creation

For business projects, review the applicable rules before scaling up. Current scraping guidance consistently recommends checking both robots.txt and terms of service

Step 6: Check for an API First

Before scraping HTML, check whether the information is available through:

Official API
Public dataset
Partner feed
Export function

An API can be more stable and structured than HTML scraping and may reduce the amount of crawling necessary.

Step 7: Start With a Small Test

Don’t immediately submit 100,000 websites.

Start with:

10 websites

Then:

50 websites

Then:

100 websites

Measure:

Emails found
Websites successfully processed
Errors
Duplicates
Processing time
False positives
Pages crawled

Only scale after the workflow works correctly.

Step 8: Visit the Homepage

For each website:

Company A
     ↓
Homepage

Extract:

Visible text
Links
mailto: links
Relevant metadata where appropriate

The homepage often contains an email in the footer or contact section.

Step 9: Discover Relevant Internal Pages

The homepage should be treated as the starting point, not the entire website.

Look for links containing words such as:

contact
contact-us
about
about-us
team
staff
leadership
company
support
sales
press
media
careers

For example:

Homepage
   ├── About
   ├── Services
   ├── Team
   ├── Contact
   ├── Press
   └── Careers

Prioritizing contact-bearing pages is generally more efficient than blindly crawling every page.)

Step 10: Set a Crawl Depth

You don’t want your scraper to follow every link forever.

For example:

Depth 0
Homepage

Depth 1
Contact
About
Team

Depth 2
Individual team profiles

A practical system might use:

Maximum depth: 2
Maximum pages/domain: 20–50

The appropriate limits depend on the project.

Step 11: Extract `mailto:` Addresses

A website might contain:

<a href="mailto:sales@example.com">
Sales
</a>

Your scraper can identify:

mailto:

and extract:

sales@example.com

This is generally more reliable than searching only visible text.

Step 12: Extract Email Patterns From Text

You can also search webpage text for patterns resembling:

name@example.com

A common pattern is:

[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}

It can identify examples such as:

info@example.com
sales@example.co.uk
john.smith@example.org

However, regex should not be considered an email verification system.

Step 13: Search the Page Source Carefully

Some websites may contain an address in:

HTML attributes
mailto: links
Embedded structured content
Metadata
JavaScript variables

However, blindly searching every script can produce irrelevant matches.

A better approach is:

Visible content
+
mailto links
+
relevant HTML attributes

before expanding into more complicated extraction.

Step 14: Handle Obfuscated Emails

Some websites intentionally format addresses like:

john [at] example [dot] com

or:

john(at)example(dot)com

or:

john at example dot com

A normalization stage can recognize common forms.

For example:

john [at] example [dot] com

can become:

john@example.com

But normalization should be conservative to avoid turning ordinary webpage text into false email addresses.

Step 15: Handle JavaScript Websites

Some websites don’t contain all visible content in the initial HTML.

Instead:

Initial HTML
      ↓
JavaScript
      ↓
Content loads
      ↓
Rendered page

A basic requests scraper may therefore miss information.

For these websites, authorized crawling may require a browser automation tool capable of rendering JavaScript.

Examples include:

Playwright
Puppeteer
Selenium

The general architecture becomes:

URL
 ↓
Browser
 ↓
JavaScript executes
 ↓
Rendered DOM
 ↓
Email extraction

Browser rendering should be used for sites you are permitted to crawl, not as a means of circumventing access restrictions.

Step 16: Don’t Assume Every Website Has an Email

This is one of the most important lessons.

Some websites have:

Email

Others have:

Contact form

Others:

Phone

Others:

Social media

And some provide no obvious contact channel.

Your database should therefore support:

email_found
email_not_found
contact_form
phone_only
restricted
error

Step 17: Store the Source URL

Never store only:

sales@example.com

Prefer:

Email:
sales@example.com

Website:
example.com

Source:
https://example.com/contact

Collected:
2026-08-25

This makes the dataset auditable.

Step 18: Store the Website Domain

Every email should remain associated with its domain.

Example:

Email	Domain
info@abc.com	abc.com
sales@abc.com	abc.com
contact@xyz.com	xyz.com

This makes later filtering much easier.

Step 19: Classify Emails

Not all addresses serve the same purpose.

General

info@
hello@
contact@

Sales

sales@
sales-team@

Support

support@
help@
customer-service@

Media

press@
media@
communications@

Careers

careers@
jobs@
recruitment@

Individual

john.smith@
mary.jones@

Classification makes the dataset considerably more useful.

Step 20: Separate Generic From Individual Addresses

Consider:

info@example.com

versus:

john.smith@example.com

The first is a general business contact.

The second may identify an individual and therefore may require additional privacy considerations depending on how you process and use it.

For that reason, don’t automatically treat every extracted address as equivalent.

Step 21: Normalize Emails

You might receive:

INFO@EXAMPLE.COM
info@example.com
info@example.com

Normalize them to:

info@example.com

Typical cleaning includes:

Removing whitespace
Removing surrounding punctuation
Standardizing case
Decoding HTML entities
Normalizing obvious obfuscation
Removing duplicates

Step 22: Deduplicate Within Each Website

Suppose:

Homepage → info@example.com
Contact → info@example.com
About → info@example.com
Footer → info@example.com

The result should be:

info@example.com

rather than four records.

Step 23: Deduplicate Across Websites

You may also discover the same email on different websites.

For example:

info@example.com
info@example.com

A global deduplication stage can remove duplicates.

However, don’t automatically delete different addresses belonging to the same domain:

sales@example.com
support@example.com
info@example.com

These are different contacts and may all be useful.

Step 24: Validate the Data

Email extraction is not the same as verification.

An extracted address might be:

Incorrect
Old
Abandoned
Typographically malformed
A placeholder
A generic mailbox
A catch-all address

A validation process can examine:

Syntax
 ↓
Domain
 ↓
DNS/MX
 ↓
Other deliverability signals

The exact level of verification depends on the purpose of the project.

Step 25: Never Treat Guessed Emails as Scraped Emails

Suppose the website says:

John Smith

but doesn’t publish his email.

You shouldn’t automatically generate:

john.smith@example.com

and put it into the database as though you extracted it.

Maintain separate fields such as:

observed_email
inferred_email

if your project legitimately needs both.

Step 26: Build a Multi-Website CSV

A useful output could be:

company,domain,email,type,source_url,status
ABC Ltd,abc.com,info@abc.com,general,https://abc.com/contact,found
ABC Ltd,abc.com,sales@abc.com,sales,https://abc.com/contact,found
XYZ Ltd,xyz.com,hello@xyz.com,general,https://xyz.com,found
Example Ltd,example.com,,,https://example.com/contact,not_found

This is far better than a simple text file containing thousands of addresses.

Step 27: Create a Processing Status

For thousands of websites, track each domain.

For example:

pending
processing
completed
no_email
restricted
error
retry

Your system might look like:

Domain	Status	Emails
abc.com	Completed	3
xyz.com	Completed	1
example.com	No email	0
company.com	Restricted	0
business.com	Retry	0

This prevents the scraper from repeatedly processing the same websites.

Step 28: Use a Queue

For larger jobs, create a queue:

URL 1 → Pending
URL 2 → Pending
URL 3 → Pending
URL 4 → Pending

Workers retrieve URLs:

Worker 1 → URL 1
Worker 2 → URL 2
Worker 3 → URL 3
Worker 4 → URL 4

When finished:

URL 1 → Completed
URL 2 → Completed
URL 3 → Error
URL 4 → Completed

This is much more reliable than one enormous loop.

Step 29: Use Concurrency Carefully

Concurrency means processing multiple websites at the same time.

Instead of:

Website A
   ↓
Website B
   ↓
Website C
   ↓
Website D

you can process:

Website A ──┐
Website B ──┤
Website C ──┼──→ Workers
Website D ──┤
Website E ──┘

This can dramatically reduce total processing time because web requests are largely I/O-bound.

But concurrency should be controlled, particularly per domain. Sending ten simultaneous requests to ten different domains is very different from sending ten requests simultaneously to one domain.

Step 30: Use Domain-Level Rate Limits

A common mistake is having:

20 workers

all accessing:

example.com

at once.

Instead, establish limits such as:

Maximum simultaneous requests per domain: 1–2

while allowing work across many different domains.

This improves reliability and reduces unnecessary server load. Current crawling guidance recommends deliberate throttling and domain-aware concurrency.

Step 31: Handle HTTP Errors

Your scraper should understand common responses.

200

Success

301/302

Redirect

403

Forbidden

404

Not found

429

Too many requests

500+

Server error

If you receive a 429 response, don’t immediately send more requests. Pause and follow any applicable Retry-After guidance. Repeated 403 responses should generally trigger a stop or review rather than increasingly aggressive retries

Step 32: Use Retries Carefully

A temporary network failure might justify:

Retry 1
 ↓
Wait
 ↓
Retry 2
 ↓
Wait longer
 ↓
Retry 3

This is called backoff.

Don’t use unlimited retries.

Otherwise:

Website unavailable
      ↓
Retry
      ↓
Retry
      ↓
Retry
      ↓
Retry

can create unnecessary traffic.

Step 33: Cache Results

If you already downloaded:

https://example.com/contact

don’t unnecessarily download it again.

Store:

URL
HTML/content
Date
Status

Then reuse the existing content where appropriate.

Caching reduces:

Network traffic
Processing time
Duplicate requests
Server load

Caching and incremental crawling are recommended practices for reliable large-scale scraping.

Step 34: Use Sitemaps

Some websites publish:

/sitemap.xml

A sitemap can help identify pages without crawling every link from the homepage.

For example:

sitemap.xml
   ↓
/about
/contact
/team
/services
/blog

You can then prioritize pages relevant to email discovery.

Using sitemaps is specifically recommended as an efficient way to focus crawling on important pages

Step 35: Keep a Crawl Log

For each request, store information such as:

timestamp
domain
URL
HTTP status
response time
emails found
error

Example:

Domain	URL	Status	Emails	Time
abc.com	/	200	1	1.2s
abc.com	/contact	200	2	1.5s
xyz.com	/	403	0	0.8s

This is invaluable when troubleshooting.

Step 36: Handle Website Changes

A website might change:

Old:
<div class="contact-email">

New:
<div data-contact="email">

A scraper that relies heavily on fragile CSS selectors can suddenly stop working.

Prefer robust extraction approaches based on:

Semantic elements
Link types
Stable attributes
Text patterns
Page context

rather than highly specific generated CSS classes.

Python Architecture for Multiple Websites

A basic architecture can look like:

websites = [
    "https://example1.com",
    "https://example2.com",
    "https://example3.com"
]

for website in websites:
    pages = discover_relevant_pages(website)

    for page in pages:
        html = fetch_page(page)
        emails = extract_emails(html)
        save_results(website, page, emails)

The important concept is not the exact code.

It is the separation of:

Website discovery
↓
Page discovery
↓
Fetching
↓
Extraction
↓
Cleaning
↓
Storage

A Better Production Architecture

For a larger system:

                 ┌───────────────┐
                 │ Website Input │
                 └───────┬───────┘
                         ↓
                 ┌───────────────┐
                 │ URL Validator │
                 └───────┬───────┘
                         ↓
                 ┌───────────────┐
                 │ Crawl Queue   │
                 └───────┬───────┘
                         ↓
              ┌──────────┴──────────┐
              ↓          ↓           ↓
           Worker 1   Worker 2    Worker 3
              ↓          ↓           ↓
           Fetch      Fetch        Fetch
              └──────────┬──────────┘
                         ↓
                 ┌───────────────┐
                 │ Email Parser  │
                 └───────┬───────┘
                         ↓
                 ┌───────────────┐
                 │ Data Cleaning │
                 └───────┬───────┘
                         ↓
                 ┌───────────────┐
                 │ Deduplication │
                 └───────┬───────┘
                         ↓
                 ┌───────────────┐
                 │ Verification  │
                 └───────┬───────┘
                         ↓
                 ┌───────────────┐
                 │ Database/CSV  │
                 └───────────────┘

Scraping 10 Websites

For 10 websites, you can keep the system simple.

Input CSV
   ↓
Python
   ↓
Requests
   ↓
BeautifulSoup
   ↓
Regex
   ↓
CSV

You probably don’t need elaborate infrastructure.

Scraping 100 Websites

At 100 websites, add:

URL validation
Status tracking
Duplicate removal
Retries
Logging
Basic concurrency
Domain rate limits

Scraping 1,000 Websites

At 1,000 websites, consider:

Queue
Worker pool
Database
Caching
Per-domain throttling
Better error handling
Browser rendering when necessary

Scraping 10,000+ Websites

At this scale, consider:

Database
+
Queue
+
Workers
+
Monitoring
+
Caching
+
Domain scheduler
+
Structured logging

The technical problem becomes an infrastructure problem as much as a scraping problem.

Example Data Model

A database table might contain:

website_id
company_name
domain
page_url
email
email_type
crawl_date
verification_status
crawl_status

You could also create separate tables:

websites
pages
emails
crawl_jobs
verification_results

This avoids putting everything into one enormous table.

Example Output

Suppose you process five websites.

The raw extraction might produce:

info@company-a.com
info@company-a.com
sales@company-a.com
hello@company-b.com
hello@company-b.com
support@company-c.com

After normalization and deduplication:

info@company-a.com
sales@company-a.com
hello@company-b.com
support@company-c.com

Your final CSV could be:

Company	Email	Type	Source
Company A	info@company-a.com	General	/contact
Company A	sales@company-a.com	Sales	/contact
Company B	hello@company-b.com	General	/
Company C	support@company-c.com	Support	/support

Measuring Performance

Don’t judge your scraper only by speed.

Track:

Website success rate

Successful websites
÷
Total websites

Email discovery rate

Websites producing emails
÷
Websites successfully crawled

Duplicate rate

Duplicate emails
÷
Raw email matches

Verification rate

Verified addresses
÷
Unique extracted addresses

Relevant-contact rate

Relevant contacts
÷
Unique extracted addresses

These metrics give you a much better understanding of the quality of the system.

Example Performance Report

Imagine processing 1,000 permitted websites:

Websites submitted:              1,000
Successfully crawled:              850
No email found:                    300
Websites with email:               550
Raw email matches:               1,800
Unique emails:                   1,350
Verified/usable:                 1,000
Generic addresses:                720
Individual addresses:             280

This tells you much more than:

“We scraped 1,800 emails.”

Common Problems When Scraping Multiple Websites

Problem 1: Some websites work and others don’t

This is normal.

Different websites use different technologies.

Solution

Build multiple extraction strategies:

Static HTML
+
mailto
+
Rendered browser

Problem 2: Too Many Websites Return 403

Possible causes include:

Website restrictions
Excessive request rate
Automated-access controls

Solution

Do not simply increase request volume or attempt to defeat the restriction.

Instead:

Check access rules
Reduce traffic
Respect the site’s restrictions
Stop where access is prohibited
Use an official API if available

Problem 3: Too Many 429 Errors

A 429 generally indicates excessive request frequency.

Solution

Implement:

Rate limit
+
Backoff
+
Retry-After handling

Problem 4: Scraper Takes Too Long

Possible causes:

Sequential requests
Excessive crawl depth
Slow websites
Browser rendering on every page
No caching

Solution

Use:

Controlled concurrency
+
Page prioritization
+
Caching
+
Timeouts

Problem 5: Too Many False Positives

The scraper might find strings such as:

image@example.com
test@example.com
example@example.com

or email-like strings inside unrelated content.

Solution

Use:

Better parsing
Context filtering
Domain checks
Deduplication
Validation
Manual sampling

Problem 6: Too Many Duplicates

This often happens because the same address appears in:

Header
Footer
Contact page
About page
Multiple team pages

Solution

Normalize and deduplicate globally.

Problem 7: JavaScript Pages Return Nothing

Solution

Determine whether the email is actually present in the rendered page.

If so, use authorized browser rendering.

If the address isn’t publicly exposed, don’t attempt to manufacture it.

Problem 8: Contact Forms Instead of Emails

Your result should say:

Contact form available

rather than:

No information

A contact form can still be valuable business information.

Problem 9: Old Emails

Websites may contain historical documents.

For example:

2019 annual report

could contain:

formeremployee@company.com

Store:

source_date
crawl_date
document_date

where available.

Problem 10: The Same Company Has Multiple Domains

For example:

company.com
company.co.uk
company.eu

You may want to group them under:

Company Name

while preserving the individual domains.

Legal and Ethical Considerations

Email scraping from multiple websites requires more care than a small one-site experiment.

Before starting:

Check website rules

Review:

robots.txt
Terms of Service

Avoid private information

Do not scrape:

Login-protected pages
Private databases
Private accounts
Information requiring unauthorized access

Minimize collection

Collect only what you actually need.

Protect stored data

If your dataset contains personal information, secure it appropriately.

Consider applicable privacy laws

Depending on your location, the website, and intended use, data-protection and electronic-marketing laws may apply.

Current guidance specifically emphasizes public business contacts, lawful processing, secure storage, and responsible outreach.

Scraping and Email Marketing Are Separate Steps

This distinction is extremely important.

Your workflow might be:

Website scraping
      ↓
Email extraction
      ↓
Verification
      ↓
Compliance review
      ↓
Marketing decision

Finding an email does not automatically give you permission to send unsolicited marketing messages.

If your objective is marketing, separately evaluate:

Applicable email-marketing laws
Lawful basis
Consent requirements
Opt-out requirements
Suppression lists
Business-to-business rules
Geographic requirements

How to Make the Process More Efficient

Use targeted crawling

Don’t crawl everything.

Prioritize:

/contact
/about
/team
/leadership
/press
/support

Use batches

Instead of:

10,000 websites

run:

Batch 1: 100
Batch 2: 100
Batch 3: 100

This makes errors easier to identify and recover from. Batch processing is also recommended in crawling guidance to reduce excessive simultaneous load.

Cache completed pages

Avoid downloading the same page repeatedly.

Store failures

Don’t lose URLs that failed.

Create:

retry_queue.csv

or a database queue.

Separate extraction from verification

Do:

Crawler
 ↓
Raw database
 ↓
Cleaning
 ↓
Verification

rather than performing everything in one step.

This makes the system easier to debug.

Recommended Technology Stack

Beginner

Python
Requests
BeautifulSoup
Regular Expressions
CSV

Best for:

Small projects
Simple websites
Learning

Intermediate

Python
Requests
BeautifulSoup
Selenium/Playwright
Pandas
SQLite/PostgreSQL

Best for:

Hundreds or thousands of websites
JavaScript-heavy websites
Structured datasets

Advanced

Python
Async HTTP
Browser automation
Queue
Database
Caching
Monitoring
Domain scheduler

Best for:

Large-scale projects
Recurring crawls
Enterprise research systems

A Practical Multi-Website Strategy

If your goal is to process hundreds of business websites, a strong workflow is:

Phase 1 — Input

Website list

Phase 2 — Validation

Valid domain?
Accessible?
Permitted?

Phase 3 — Discovery

Homepage
↓
Contact
↓
About
↓
Team
↓
Press

Phase 4 — Extraction

mailto
+
visible text
+
authorized rendered content

Phase 5 — Cleaning

Normalize
↓
Remove false positives
↓
Deduplicate

Phase 6 — Classification

General
Sales
Support
Press
Careers
Individual

Phase 7 — Verification

Syntax
↓
Domain
↓
Deliverability signals

Phase 8 — Storage

CSV
Database
CRM

Phase 9 — Compliance

Purpose
+
Privacy
+
Outreach rules
+
Opt-outs

Final Checklist

Before running a multi-website email scraper, check:

Conclusion

Scraping emails from multiple websites is best understood as a multi-stage data collection system, not simply a regular-expression exercise.

For a small project:

CSV
 ↓
Python
 ↓
Website
 ↓
Extract
 ↓
Clean
 ↓
Export

may be enough.

For hundreds or thousands of websites:

Website Database
       ↓
URL Queue
       ↓
Controlled Workers
       ↓
Domain-Level Rate Limits
       ↓
Page Discovery
       ↓
HTML/Browser Extraction
       ↓
Email Detection
       ↓
Normalization
       ↓
Deduplication
       ↓
Verification
       ↓
Database

is a much stronger architecture.

The most important principle is quality over raw volume. A responsible multi-site scraper should know which websites it is allowed to crawl, limit its impact on each server, distinguish genuine email addresses from false positives, preserve the source of every result, and clearly separate extracted information from inferred or guessed information. Current web-crawling guidance also recommends throttling, caching, batching, handling 429 responses appropriately, and respecting site-specific crawling rules.

For legitimate business research, the end goal should not simply be “scrape as many emails as possible.” It should be “build an accurate, current, traceable, and appropriately collected business-contact dataset.”

How to Scrape Emails From Multiple Websites – Case Studies and Comments

Scraping emails from multiple websites can save substantial research time when a business needs to process hundreds or thousands of permitted websites. The most effective workflows generally combine website discovery, targeted page crawling, email extraction, normalization, deduplication, source tracking, and verification rather than simply searching every page for an @ symbol.

Below are practical case studies and comments showing how these approaches work in real-world situations.

Case Study 1: Processing 4,500 Websites for Brand Research

Background

A brand-protection research project began with a much larger collection of potentially relevant websites. Researchers narrowed the dataset to approximately 4,500 domains that appeared most relevant to the brand being investigated.

The team then automatically inspected website HTML and extracted strings matching the general format of an email address. At least one email was found on just over 1,000 of the sites when focusing on homepages.

Workflow

Large website dataset
        ↓
Identify relevant domains
        ↓
Prioritize approximately 4,500 domains
        ↓
Inspect HTML
        ↓
Extract email patterns
        ↓
Compare addresses across domains
        ↓
Cluster related websites

What Made the Case Interesting?

The researchers weren’t interested only in collecting emails.

They examined which email addresses appeared on multiple websites.

For example:

Website A → info@example-provider.com
Website B → info@example-provider.com
Website C → info@example-provider.com

The shared address could provide a clue that the websites were connected.

Comment

This demonstrates that an email address can function as a data relationship, not merely a contact detail.

However, researchers must be careful: a common email address can also belong to a shared service provider and does not automatically prove that two businesses are related

Case Study 2: Automating Agency Prospect Research

Background

A marketing agency had a list of company websites and needed to identify publicly displayed business email addresses.

The manual workflow was:

Open website
 ↓
Find Contact
 ↓
Copy email
 ↓
Return to spreadsheet
 ↓
Open next website

When hundreds of websites were involved, this became extremely repetitive.

Automated Workflow

CSV of websites
        ↓
Website crawler
        ↓
Homepage
        ↓
Contact/About/Team pages
        ↓
Email extraction
        ↓
Normalization
        ↓
CSV/database

Modern website-extraction systems commonly support multiple URLs in one run, bounded same-site crawling, contact-page prioritization, deduplication, and source-page tracking.

Comment

The important improvement isn’t simply speed.

The system also creates a consistent structure:

Company	Website	Email	Source
Company A	companya.com	info@companya.com	/contact
Company B	companyb.com	sales@companyb.com	/sales
Company C	companyc.com	hello@companyc.com	/

This makes the resulting data easier to review and import into other systems.

Case Study 3: Processing 500 Companies Instead of Browsing Manually

Background

A workflow provider described a scenario involving 500 companies.

At an estimated 10 minutes of manual research per company, the work would represent approximately:

83 hours of manual effort.

The automated approach instead accepted a list of domains and processed them in bulk.

Manual Process

Company 1 → 10 minutes
Company 2 → 10 minutes
Company 3 → 10 minutes
...
Company 500 → 10 minutes

Automated Process

500 websites
      ↓
Batch processing
      ↓
Website crawling
      ↓
Email extraction
      ↓
Structured dataset

Comment

This illustrates one of the strongest reasons to automate multiple-website extraction:

repetitive navigation is replaced with batch processing.

However, automation doesn’t eliminate the need for quality control.

A scraper can process 500 websites quickly while still returning:

Duplicate addresses
Old addresses
Generic addresses
False positives
Websites with no email
Contact forms instead of emails

Case Study 4: B2B Email Extractor Using CSV Input

Background

A developer created a B2B email extraction tool to address the problem of manually researching company websites.

The tool accepted a Google Maps export or another CSV containing companies and websites, then scanned homepages and deeper website pages for corporate email addresses.

Workflow

Google Maps/CSV
       ↓
Company list
       ↓
Website URL
       ↓
Homepage
       ↓
Deep subpages
       ↓
Corporate emails

Why Deep Pages?

A homepage may contain:

Company information
Services
Phone

while the actual email appears on:

/contact
/team
/about
/support

Comment

This is one of the most important lessons in multi-site scraping:

A website should not be treated as a single webpage.

A targeted crawler can improve discovery by following relevant internal pages.

Case Study 5: Google Maps Businesses → Websites → Emails

Background

Another workflow begins with a list of businesses rather than websites.

The system first identifies businesses, obtains their websites, and then attempts to locate email addresses on those websites.

Workflow

Business category
       ↓
Business listings
       ↓
Company website
       ↓
Contact pages
       ↓
Email extraction
       ↓
Spreadsheet

A 2026 workflow example describes using n8n to find businesses, visit their websites for email addresses, and then handle cases where websites don’t expose an email.

Comment

This approach is particularly useful for:

Local service businesses
Suppliers
Agencies
Contractors
Retailers
Professional services

The website becomes the second stage of the research process.

Case Study 6: Email Extraction Plus Social Profiles

Background

Some bulk website workflows don’t stop at email addresses.

They extract:

Email
Facebook
LinkedIn
X/Twitter
Instagram

from each company website.

One current workflow accepts multiple domains and returns a primary email, additional email addresses, and social profiles in a structured dataset.

Example

Company A
 ├── info@company.com
 ├── sales@company.com
 ├── LinkedIn
 └── Instagram

Comment

This creates a richer prospect record than an email-only database.

For example:

Company	Email	LinkedIn	Instagram
Company A	info@a.com	Found	Found
Company B	sales@b.com	Found	Not found
Company C	—	Found	Found

Case Study 7: Structured Email Extraction With Source Tracking

A more sophisticated approach stores not just the email but where the email was found.

For example:

Email:
sales@example.com

Source:
https://example.com/contact

Original website:
https://example.com

Discovery method:
mailto-link

Modern extraction systems can record source URLs and discovery context, such as whether the address appeared in a mailto: link, visible text, or supported obfuscated text)

Comment

This is much better than maintaining a simple list:

sales@example.com
info@example.com
hello@example.com

because you can later answer:

“Where did this email come from?”

That makes auditing and data-quality review easier.

Case Study 8: Extracting Emails From 10,000+ Domains

Background

Large-scale website scraping becomes a different engineering problem.

A practitioner attempting to process more than 10,000 domains reported that a custom crawler was faster than manual browsing but initially missed emails that were easy to find manually

The Problem

A basic crawler might do:

Homepage
 ↓
Regex
 ↓
Email

But real websites may have:

Homepage
 ↓
Contact page
 ↓
Team page
 ↓
JavaScript
 ↓
Footer
 ↓
PDF

Comment

This is a critical lesson:

Speed doesn’t compensate for poor coverage.

A scraper that processes 10,000 websites but misses half the useful information may be less valuable than a slower system with better page discovery.

Case Study 9: Website Contact Pages and Decision-Makers

Background

A B2B prospecting workflow focused on company websites to identify:

Emails
Team members
Decision-makers
Contact pages

The approach was to crawl live company websites and rank potential contacts according to their usefulness for prospecting. A published 2026 case study reports processing 100 sites in under a minute in its particular implementation.

Workflow

Company websites
       ↓
Contact pages
       ↓
Team pages
       ↓
Email extraction
       ↓
Employee identification
       ↓
Contact ranking

Comment

This demonstrates an important distinction.

Email scraping answers:

What email addresses are publicly displayed?

Decision-maker research answers:

Who is the appropriate person to contact?

These are related but different problems.

Case Study 10: Website Email Extraction for Research

Background

A research team wants to build a database of companies in a particular industry.

Instead of collecting only emails, it creates a structured record.

Company
Website
Industry
Country
Email
Email type
Source page
Date collected

Example

ABC Logistics
abc-logistics.com
Logistics
United Kingdom
info@abc-logistics.com
General
/contact
2026-08-25

Comment

This is particularly useful for:

Market research
Industry analysis
Academic research
Competitor mapping
Supplier research

The email is only one field in the dataset.

Case Study 11: Scraping Websites for Supplier Research

Background

An e-commerce business wants to identify potential suppliers.

It creates a list of supplier websites and processes them in bulk.

Workflow

Supplier websites
       ↓
Homepage
       ↓
Wholesale page
       ↓
Contact page
       ↓
Sales email
       ↓
Supplier database

Data Collected

Company
Product category
Website
Sales email
Wholesale email
Phone
Country

Comment

This is a good example of how website extraction can be useful without immediately turning the results into a marketing list.

The data can support:

Procurement
Sourcing
Vendor discovery
Partnership research

Case Study 12: Recruitment Research

Background

A recruitment company wants to identify publicly listed employer contacts.

Instead of trying to generate private email addresses, it looks for explicitly published business channels.

Examples include:

careers@company.com
recruitment@company.com
jobs@company.com

Workflow

Company website
       ↓
Careers page
       ↓
Recruitment information
       ↓
Public email
       ↓
Recruitment database

Comment

This is more reliable than assuming an employee’s email address based on a naming pattern.

For example, if a website doesn’t publish:

john.smith@company.com

the scraper shouldn’t pretend that an inferred address was actually found.

Case Study 13: Scraping Only High-Value Pages

Background

A company initially crawled every page on every domain.

This created unnecessary work.

A website might contain:

200 blog posts
50 product pages
10 service pages
1 contact page

If the objective is email discovery, crawling every blog post isn’t necessarily efficient.

Improved Workflow

Homepage
 ↓
Identify relevant links
 ↓
Contact
About
Team
Support
Press
Careers
 ↓
Extract

Some current extraction systems deliberately use bounded crawling and prioritize contact-related pages instead of performing unrestricted whole-site crawling

Comment

Targeted crawling is usually better than blind crawling for contact discovery.

Case Study 14: Separating Emails by Type

Suppose a crawler finds:

info@example.com
sales@example.com
support@example.com
press@example.com
john@example.com

Instead of storing them all identically, classify them.

Email	Type
info@example.com	General
sales@example.com	Sales
support@example.com	Support
press@example.com	Media
john@example.com	Individual

Comment

This improves the usefulness of the dataset.

A company doing partnership research may want:

partnership@
business@
sales@

while a journalist may prefer:

press@
media@
communications@

Case Study 15: Deduplicating Thousands of Results

Background

A crawler processes 1,000 websites.

It finds:

2,500 raw email matches

But many addresses appear repeatedly.

For example:

Homepage → info@example.com
About → info@example.com
Contact → info@example.com
Footer → info@example.com

After deduplication:

1,600 unique email addresses

Comment

This demonstrates why:

raw email count ≠ useful email count.

A good system should distinguish:

Raw matches
Unique addresses
Verified addresses
Relevant addresses

Case Study 16: Contact Form Instead of Email

Background

A crawler processes 1,000 websites.

Some websites return:

Email found: 600

Others:

Contact form: 200

Others:

Phone only: 100

And some:

No obvious contact channel: 100

Comment

A contact-form-only result shouldn’t necessarily be treated as a technical failure.

The website may deliberately choose not to publish an email address.

A mature dataset can therefore contain:

email
contact_form_url
phone
no_contact

rather than forcing every website into an email/no-email binary.

Case Study 17: Extracting Only Publicly Displayed Emails

A modern lightweight extraction workflow can be deliberately restricted to:

Public page
 ↓
Visible email
OR
mailto link
OR
supported public obfuscation

It does not generate guessed employee addresses or access pages behind authentication

Comment

This is a useful design principle:

Extraction should be different from guessing.

If the website displays:

sales@example.com

record it.

If the website only displays:

John Smith

don’t automatically turn that into:

john.smith@example.com

and label it “scraped.”

Case Study 18: Using Email Addresses to Discover Website Relationships

The 2025 brand-protection case study provides another interesting application.

Researchers found the same email address appearing on different domains and used that information as one possible clue for establishing relationships between websites

Example

Website A
info@shared-domain.com

Website B
info@shared-domain.com

Website C
info@shared-domain.com

This creates a potential relationship:

A ──┐
B ──┼── Shared email
C ──┘

Comment

This can be useful for:

Brand protection
OSINT research
Investigative research
Network analysis

But shared infrastructure can produce false associations, so an email match should be treated as a lead for further investigation, not proof of ownership.

Case Study 19: Building a Multi-Website Spreadsheet

For a small company, a sophisticated database may not be necessary.

A spreadsheet can contain:

Website	Email	Type	Source Page	Status
company1.com	info@company1.com	General	/contact	Review
company2.com	sales@company2.com	Sales	/about	Review
company3.com	—	—	/contact	Form only

Workflow

Website CSV
 ↓
Scraper
 ↓
CSV output
 ↓
Excel/Google Sheets
 ↓
Manual review

Comment

The best system is not necessarily the most sophisticated one.

For 50–200 websites, a spreadsheet-based workflow may be completely adequate.

Case Study 20: Large-Scale Automated Workflow

At much larger scale, a production system may look like:

                Website List
                     ↓
              URL Normalization
                     ↓
                Crawl Queue
                     ↓
        ┌────────────┼────────────┐
        ↓            ↓            ↓
     Worker 1     Worker 2     Worker 3
        ↓            ↓            ↓
     Website      Website      Website
        └────────────┼────────────┘
                     ↓
               Page Discovery
                     ↓
              Email Extraction
                     ↓
               Normalization
                     ↓
                Deduplication
                     ↓
              Quality Checking
                     ↓
                 Database

Comment

At this point, the biggest challenge is no longer regex.

It becomes:

Queue management
Concurrency
Rate control
Error handling
Storage
Monitoring
Data quality
Website changes

Comments From Practitioners

Comment 1: “Deep scanning makes a difference”

Practitioners repeatedly point out that emails aren’t necessarily located on the homepage.

They can appear on:

Contact pages
Team pages
About pages
Footer
Staff pages
Support pages

A 2026 community project specifically describes scanning homepages and then deep-scanning subpages to find corporate email addresses.

Lesson

Don’t build a crawler that assumes:

Homepage = entire website

Comment 2: “Manual browsing can find things a basic crawler misses”

A practitioner processing more than 10,000 domains reported that a custom crawler was much faster than manual browsing but less accurate initially

Lesson

Automation needs testing against real websites.

Don’t assume that:

“The script ran successfully”

means:

“The scraper found everything.”

Comment 3: “Generic emails dominate”

A major problem with website email scraping is that many published addresses are role-based:

info@
contact@
sales@
support@
hello@

These can be genuine and useful, but they usually aren’t equivalent to a named decision-maker address.

Lesson

Classify addresses instead of treating every result as a direct contact.

Comment 4: “Source tracking is extremely valuable”

Suppose your database contains:

sales@example.com

Six months later, someone asks:

Where did this come from?

Without provenance, you may have to search for the answer again.

With source tracking:

Email: sales@example.com
Source: example.com/contact
Collected: August 25, 2026

the answer is immediate.

Current extraction systems increasingly preserve source-page information for exactly this reason.

Comment 5: “Don’t guess email addresses”

If a website publishes:

john.smith@example.com

it can be extracted.

If the website publishes only:

John Smith

creating:

john.smith@example.com

is a different operation.

Lesson

Maintain a clear distinction between:

observed

and

inferred.

Comment 6: “Verification is separate from extraction”

A scraper can tell you:

sales@example.com

was displayed on a page.

It cannot automatically prove:

The mailbox is currently active.

Email-extraction systems explicitly distinguish extraction from mailbox verification

Lesson

Use a separate verification stage when the project requires deliverability information.

Comment 7: “A contact form isn’t a failure”

Some websites intentionally don’t publish emails.

Instead they provide:

Contact form

or:

Book a consultation

Lesson

Store:

contact_form_url

rather than marking the website as completely useless.

Comment 8: “Start with a small batch”

A good workflow is:

10 websites
 ↓
Test
 ↓
50 websites
 ↓
Test
 ↓
100 websites
 ↓
Scale

This allows you to discover:

False positives
Missing emails
Website compatibility problems
Rate-limit issues
Duplicate problems

before processing thousands of domains.

Comment 9: “Targeted crawling beats unlimited crawling”

If you’re looking for contact information, prioritize pages such as:

/contact
/about
/team
/staff
/support
/press

instead of crawling every blog article.

Current multi-site extraction tools commonly use bounded page counts and contact-related link filtering to control crawl expansion.

Comment 10: “Don’t measure success by raw email volume”

Imagine two systems:

System A

10,000 raw matches

System B

5,000 unique
4,200 clean
3,600 relevant

System B may be much more useful.

Better metrics

Measure:

Websites successfully processed
Websites with emails
Unique emails
Duplicate rate
Verification rate
Relevant-contact rate
Contact-form rate

Comment 11: “Website scraping is useful for more than lead generation”

The same technology can support:

Market research
Supplier discovery
Recruitment
Brand protection
Academic research
Competitor analysis
Business intelligence
Website auditing

The brand-protection case demonstrates how extracted emails can also be used as signals for identifying relationships among online properties.

Comment 12: “The website may contain more valuable information than the email”

Suppose a website gives you:

Company
Industry
Products
Location
Leadership
Contact email
Social profiles

The complete company profile may be much more valuable than:

info@example.com

This is why many modern website-extraction workflows combine emails with phone numbers, social profiles, company information, and source URLs

Case Study Comparison

Case Study	Main Objective	Main Lesson
4,500-domain research	Brand investigation	Email addresses can reveal relationships
Agency prospecting	Reduce manual work	Batch processing improves efficiency
500-company workflow	Automate research	Automation can replace repetitive browsing
B2B extractor	Corporate contact discovery	Deep pages matter
Business-to-website workflow	Local lead research	Business lists can feed website crawling
Social + email extraction	Enrichment	Email is only one data point
Source-tracked extraction	Data quality	Provenance matters
10,000+ domains	Large-scale scraping	Architecture becomes critical
Decision-maker research	B2B prospecting	Email extraction and contact identification differ
Supplier research	Procurement	Scraping has non-marketing uses

What These Case Studies Teach

1. Multiple websites require structured processing

Don’t manually open one website after another.

Use:

Input
 ↓
Queue
 ↓
Crawler
 ↓
Extractor
 ↓
Database

2. Homepage-only extraction is often insufficient

Prioritize relevant internal pages.

3. Email extraction isn’t email verification

An address being publicly displayed doesn’t guarantee that the mailbox remains active.

4. Generic addresses are common

Expect many:

info@
contact@
sales@
support@

rather than named decision-makers.

5. Source URLs should be preserved

Always know where each email came from.

6. Don’t confuse inference with extraction

A guessed email shouldn’t be presented as a scraped email.

7. Quality matters more than volume

A smaller, clean, traceable dataset can be more useful than a huge raw list.

Recommended Multi-Website Workflow

Based on the case studies, a strong workflow is:

1. Prepare permitted website list
             ↓
2. Normalize domains
             ↓
3. Remove duplicates
             ↓
4. Review access/usage rules
             ↓
5. Check for official APIs where appropriate
             ↓
6. Crawl homepage
             ↓
7. Discover contact-related pages
             ↓
8. Extract publicly displayed emails
             ↓
9. Record source URL
             ↓
10. Normalize addresses
             ↓
11. Remove duplicates
             ↓
12. Classify addresses
             ↓
13. Verify where appropriate
             ↓
14. Export structured data
             ↓
15. Review before use

Example Final Dataset

After processing multiple websites, a useful final dataset might look like:

Company	Domain	Email	Type	Source	Status
ABC Ltd	abc.com	info@abc.com	General	/contact	Review
ABC Ltd	abc.com	sales@abc.com	Sales	/contact	Review
XYZ Inc	xyz.com	hello@xyz.com	General	/	Review
Global Services	global.com	support@global.com	Support	/support	Review
Example Co	example.com	—	Contact form	/contact	Form

This is significantly more useful than a raw list of email addresses because it preserves company identity, domain, email type, source, and status.

Final Comments

The case studies show that scraping emails from multiple websites is most effective when it is treated as a structured research and data-quality workflow, rather than simply an exercise in collecting as many addresses as possible.

The strongest systems generally follow this model:

Multiple Websites
       ↓
Targeted Crawling
       ↓
Relevant Pages
       ↓
Public Email Extraction
       ↓
Normalization
       ↓
Deduplication
       ↓
Classification
       ↓
Verification
       ↓
Source Tracking
       ↓
Structured Database

The biggest practical lesson is quality over quantity. A system that extracts 5,000 accurately sourced and properly classified addresses can be more valuable than one that produces 20,000 raw matches with duplicates, stale addresses, false positives, and no source information.

It is also important to remember that finding a publicly displayed email does not automatically establish permission to send marketing messages to it. Website collection and subsequent email use are separate questions and should be evaluated under the applicable website rules, privacy requirements, and marketing laws.

For legitimate business research, the ideal objective is therefore not:

“How many emails can I scrape?”

but:

“How can I build an accurate, traceable, appropriately collected contact dataset from the websites I am permitted to process?”

How to Scrape Emails From Multiple Websites

What Does Scraping Emails From Multiple Websites Mean?

Why Multiple-Website Scraping Is More Difficult

1. Different website structures

2. Different technologies

3. Different access rules

The Basic Multi-Website Workflow

Step 1: Prepare Your Website List

Step 2: Normalize Website URLs

Step 3: Remove Duplicate Domains

Step 4: Check the Website’s Crawling Rules

Step 5: Check Terms and Conditions

Step 6: Check for an API First

Step 7: Start With a Small Test

Step 8: Visit the Homepage

Step 9: Discover Relevant Internal Pages

Step 10: Set a Crawl Depth

Step 11: Extract mailto: Addresses

Step 12: Extract Email Patterns From Text

Step 13: Search the Page Source Carefully

Step 14: Handle Obfuscated Emails

Step 15: Handle JavaScript Websites

Step 16: Don’t Assume Every Website Has an Email

Step 17: Store the Source URL

Step 18: Store the Website Domain

Step 19: Classify Emails

General

Sales

Support

Media

Careers

Individual

Step 20: Separate Generic From Individual Addresses

Step 21: Normalize Emails

Step 22: Deduplicate Within Each Website

Step 23: Deduplicate Across Websites

Step 24: Validate the Data

Step 25: Never Treat Guessed Emails as Scraped Emails

Step 26: Build a Multi-Website CSV

Step 27: Create a Processing Status

Step 28: Use a Queue

Step 29: Use Concurrency Carefully

Step 30: Use Domain-Level Rate Limits

Step 31: Handle HTTP Errors

200

301/302

403

404

429

500+

Step 32: Use Retries Carefully

Step 33: Cache Results

Step 34: Use Sitemaps

Step 35: Keep a Crawl Log

Step 36: Handle Website Changes

Python Architecture for Multiple Websites

A Better Production Architecture

Scraping 10 Websites

Scraping 100 Websites

Scraping 1,000 Websites

Scraping 10,000+ Websites

Example Data Model

Example Output

Measuring Performance

Website success rate

Email discovery rate

Duplicate rate

Verification rate

Relevant-contact rate

Example Performance Report

Common Problems When Scraping Multiple Websites

Problem 1: Some websites work and others don’t

Solution

Problem 2: Too Many Websites Return 403

Solution

Problem 3: Too Many 429 Errors

Solution

Problem 4: Scraper Takes Too Long

Solution

Problem 5: Too Many False Positives

Step 11: Extract `mailto:` Addresses