Ethics and the law: “white hat” parsing. How to collect data from websites without violating the law and rules (robots.txt, Terms of Service).

Roman

Nov 26, 2025

Proxy

Web scraping (parsing) has come a long way from the “wild west” of the 2000s to a modern industry with strict rules. Today, data collection is the foundation for e-commerce, AI training, and marketing analytics.

But there is a problem: websites don't want to give away their data for free. They protect themselves legally (Terms of Service) and technically (Anti-Bot systems).

How can you collect information legally? Where is the line between analytics and a DDoS attack? And why is compliance with robots.txt not just a matter of courtesy, but a matter of survival for your business?

In this article, we will examine the code of honor for “white” parsers and the technical rules that will keep you out of trouble with the law.

---

#### **Part 1. What is “white” parsing?**

“White” parsing is the collection of **publicly available** data in compliance with the rules of the donor site and personal data legislation.

**Three pillars of legality:**

1. **The data is public:** You do not hack accounts, guess passwords, or bypass paywalls. You take what any unregistered visitor can see.
2. **You do not harm the site:** Your script does not overload the server (no DDoS) and does not interfere with the work of real users.
3. **You do not violate copyright:** You collect data (facts, prices), not content (copyrighted texts, photos) for republication on your own behalf.

**Important note:** Collecting personal data (names, phone numbers, email addresses of individuals) is a minefield. The EU has GDPR, while the US has a patchwork system with an emphasis on CCPA (California) and CFAA precedent. If you parse a user database for spam mailing, this is **“black” parsing**, which is illegal by definition.

---

#### **Part 2. Technical etiquette: Robots.txt and User-Agent**

Before you run a parser, you must “knock on the door.”

##### **1. The `robots.txt` file: Law or recommendation?**

This is a text file in the root directory of any website (`site.com/robots.txt`). It contains instructions for robots.

* **What to look for there:**
  * `User-agent: *` — rules for all bots.
  * `Disallow: /admin/` — do not go here.
  * `Crawl-delay: 10` — pause for 10 seconds between requests.

**Is this a law?** Legally — no (in most countries). **Should you comply?** Technically — **yes**. If `robots.txt` prohibits scanning and you ignore it, the site's security systems will classify your traffic as malicious. The result — an instant IP ban.

##### **2. User-Agent: Don't pretend to be someone you're not**

Many parsers masquerade as `Chrome/120.0...`. In “white” parsing, it is considered good practice to use **your own** User-Agent, which contains the bot owner's contact information.

* *Example:* `MyPriceBot/1.0 (+http://mysite.com/bot-contact)` This shows the site administrator that you are not a malicious user and gives them a way to contact you if your bot is creating a load, instead of immediately banning your subnet.

---

#### **Part 3. Legal trap: Terms of Service (ToS)**

If `robots.txt` is a “Do Not Enter” sign, then **Terms of Service (User Agreement)** is a contract.

The most dangerous area is parsing **after authorization**. When you register on a website and check the “I agree to the terms” box, you are entering into a legal agreement. If the terms state “Any automated data collection is prohibited” (as is the case with Facebook, LinkedIn, and Instagram), then running a parser within your account is a **breach of contract**.

**Consequences:**

1. Account ban without the right to restore it.
2. In rare cases, a lawsuit for breach of contract.

**The “white” parser rule:** Try to collect data **without authorization**. Public data (prices, catalogs) is not protected by copyright law (facts are not copyrighted), and courts (for example, the *HiQ Labs vs LinkedIn* case) often side with parsers if the data was publicly available.

---

#### **Part 4. How not to crash a website: Rate Limiting**

The most common reason for blocking is not *what* you are downloading, but *how* fast you are doing it.

If you send 100 requests per second to a small online store, you are not an analyst to them, you are a DDoS attack.

**Rules of courtesy:**

1. **Limit requests:** Pause (sleep) between requests.
2. **Monitor response codes:** If the site starts returning `429 Too Many Requests` or `503 Service Unavailable`, your script should **stop immediately** and increase the delay. Continuing to hammer a “down” server is technical rudeness.
3. **Parse at night:** Load the target site during hours of least activity by its real users.

---

#### **Part 5. Infrastructure: The role of proxies in ethical parsing**

Even if you follow all the rules, collecting large amounts of data from a single IP address will raise suspicion among security systems (WAF).

To distribute the load and avoid automatic filters, you need to use **high-quality proxies**.

**Which type should you choose for “white” parsing?**

1. **Server proxies (Datacenter):** Ideal for **mass collection of open data** (catalogues, prices) from websites that do not have strict protection. They are cheap, fast, and allow you to comply with ethics (not to overload the network of home users).
2. **Residential proxies:** Needed for websites with strict limits. They allow your requests to look like requests from different people in different regions.
   * *Ethical consideration:* Only use ethical residential proxies (Ethical Proxy Networks) where users have consented to the use of their traffic.

At **CyberYozh App**, we strictly monitor the purity of our pools. We provide tools for ethical parsing:

* **IP rotation:** To avoid overloading a single address.
* **Accurate geo-targeting:** To get relevant data without unnecessary requests.

---

### **Conclusion: Reputation is more valuable than data**

“White” parsing is a long-term game. By breaking the rules, ignoring `robots.txt`, and DDOSing sites, you may get data faster, but you risk losing access to the source forever.

Follow technical etiquette, respect donor resources, and use the right infrastructure. This is the only way to build a sustainable data business.

👉 **Building a legal parser?** Make sure it has a solid foundation. Choose the right [**server or resident proxies from the CyberYozh App catalog**](https://app.cyberyozh.com/catalog/). We'll help you scale your data collection while staying on the light side of the force.

Useful?

Share the article!

Ethics and the law: “white hat” parsing. How to collect data from websites without violating the law and rules (robots.txt, Terms of Service).

Not with us yet?