moonetix / datacenter-ip-ranges
Daily cloud, hosting, CDN, Tor, iCloud Private Relay and search/AI crawler IP ranges from the providers' own feeds, with a dependency-free lookup and crawler verification helper.
Package info
github.com/moonetix/datacenter-ip-ranges
Language:JavaScript
pkg:composer/moonetix/datacenter-ip-ranges
Requires
- php: >=8.1
Requires (Dev)
None
Suggests
None
Provides
None
Conflicts
None
Replaces
None
README
Cloud, hosting and CDN IP ranges from the providers' own feeds, rebuilt every day, plus the ranges that search engines and AI companies publish for their crawlers. Lookup and crawler verification helpers for Node.js, PHP and Python, and ready-made files for nginx, nftables and ipset.
Maintained by Moonetix.
A sign-up from an AWS address, an offer click from a DigitalOcean droplet, a "user" on a GitHub Actions runner: real people rarely browse from a datacenter. This repo collects the address ranges that AWS, Google Cloud, Oracle Cloud, DigitalOcean, Linode, Vultr, Cloudflare, Fastly and GitHub publish about themselves, merges them into plain CIDR lists, and keeps Tor exits and iCloud Private Relay in separate lists so you can treat them differently.
The crawler ranges that Google, Bing, Apple, DuckDuckGo, OpenAI, Perplexity, Mistral AI, Common Crawl and Ahrefs publish are a separate list too, with verifyBot() to tell a real Googlebot or GPTBot from a scraper that only copies its user agent.
Every range comes from a feed the provider documents. Nothing is scraped from third-party lists, nothing is guessed from ASN names.
git clone --depth 1 https://github.com/moonetix/datacenter-ip-ranges node datacenter-ip-ranges/bin/check.js 3.5.140.1 8.8.8.8 192.168.1.10
3.5.140.1 aws cloud 3.5.128.0/19
8.8.8.8 google-services services 8.8.8.0/24
192.168.1.10 not listed
To try an address without installing anything, open the live demo. It runs the same matcher in your browser on the current lists.
Only need the list? curl -fsSL https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/all.txt gives one CIDR per line.
Why check for datacenter addresses
Most people browse from home, mobile or office networks. Traffic from rented servers is usually automated, so a datacenter address is a useful signal wherever automation costs you money:
- Sign-ups and free trials. One server can create hundreds of accounts and claim a bonus or trial with each.
- Paid actions. Ad networks, affiliate programs and offerwalls reject or reverse conversions from hosting networks, so a reward paid for such traffic is often money you never get back.
- Clicks, votes and scraping. Headless browsers on cloud instances inflate counts and load your servers.
- Fake crawlers. Scrapers send a Googlebot or GPTBot user agent to get past rate limits and robots rules. The crawler lists show which of those requests really come from the crawler's operator.
- Cheap to check. A lookup here takes well under a microsecond in Node.js, so it can run on every request.
A match is a signal, not a verdict. See Limitations.
The lists
| File | Contents |
|---|---|
data/all.txt |
Every datacenter range: categories cloud, cdn, services. IPv4 first, then IPv6. One CIDR per line. |
data/ipv4.txt, data/ipv6.txt |
The same, split by family. |
data/providers/<id>.txt |
One provider, for example aws.txt, cloudflare.txt, tor.txt. |
data/categories/<category>.txt |
One category: cloud, cdn, services, privacy-relay, tor, crawler. |
data/firewall/ |
datacenter.nft, datacenter.ipset, tor.nft, tor.ipset: the datacenter and Tor lists as nftables sets and ipset restore files. |
data/webserver/ |
datacenter.nginx.conf, tor.nginx.conf, datacenter.apache.conf, tor.apache.conf: the same two lists as an nginx geo block and an Apache 2.4 Require not ip block. |
data/ranges.json, data/ranges.csv |
Every provider range with provider, category, cidr and source URL. All categories. |
data/sources.json, data/SOURCES.md |
Per feed: status, range counts, last fetch time, feed URL, terms. |
all.txt, ipv4.txt and ipv6.txt do not contain Tor exits, iCloud Private Relay or the crawler lists. Tor and Private Relay carry real users and deserve a different policy, and a crawler is something you may want to let in. Use data/categories/tor.txt, data/categories/privacy-relay.txt and data/categories/crawler.txt for them.
Ranges are normalized: host bits cleared, IPv6 in RFC 5952 form, overlapping and adjacent blocks merged, private and documentation ranges removed. A merged block can be larger than any single block in the provider's feed; the set of addresses is the same.
Categories
| Category | Providers | Typical traffic |
|---|---|---|
cloud |
AWS, Google Cloud, Oracle Cloud, DigitalOcean, Akamai Cloud (Linode), Vultr | VPS and cloud instances: bots, scrapers, self-hosted proxies and VPNs |
cdn |
Cloudflare, Fastly | Edge networks: requests relayed or sent by code running on the CDN. Cloudflare WARP exits from addresses outside the published ranges, so WARP users are not matched. |
services |
Google services (goog.json minus cloud.json), GitHub (incl. Actions runners) | Crawlers, fetchers, CI runners, automation |
privacy-relay |
iCloud Private Relay | Safari users on iCloud+. Real people, shared addresses. |
tor |
Tor exit nodes | Tor Browser users and automated tools on Tor |
crawler |
Googlebot and the other Google crawlers and fetchers, Bingbot, Applebot, DuckDuckBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, MistralAI-User, CCBot, AhrefsBot | Search engine, AI and SEO crawlers, and fetches made on behalf of a chat user. Some of these addresses also sit in the services or cloud lists. See Verify crawlers. |
Raw files
Use the files straight from GitHub. They change only when a provider changes its ranges.
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/all.txt
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/ipv4.txt
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/ipv6.txt
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/providers/aws.txt
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/categories/tor.txt
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/ranges.json
Download once a day, not on every request:
curl -fsSL -o /etc/dcip/all.txt \ https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/all.txt
nginx can tag requests with the geo module. Rebuild the map after each download:
sed 's/$/ 1;/' /etc/dcip/all.txt > /etc/nginx/datacenter.map && nginx -s reload
geo $remote_addr $is_datacenter { default 0; include /etc/nginx/datacenter.map; } # Pass it to your app, for example: proxy_set_header X-Datacenter $is_datacenter;
Firewalls
data/firewall/ has the datacenter list (the same ranges as all.txt) and the Tor exits in two firewall formats. They are rebuilt with the lists, so a daily download keeps them current.
nftables: each file defines an IPv4 and an IPv6 interval set to include inside a table.
curl -fsSL -o /etc/nftables.d/datacenter.nft \ https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/firewall/datacenter.nft
table inet filter {
include "/etc/nftables.d/datacenter.nft"
chain input {
type filter hook input priority 0;
tcp dport 443 ip saddr @datacenter_v4 counter
tcp dport 443 ip6 saddr @datacenter_v6 counter
}
}
ipset: each file creates <name>-v4 and <name>-v6 hash:net sets and fills them.
curl -fsSL https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/firewall/tor.ipset | ipset restore -exist iptables -I INPUT -p tcp --dport 443 -m set --match-set tor-v4 src -j DROP
Count before you drop. A firewall rule cannot tell a scraper from a person on a cloud desktop, and it also blocks webhooks and postbacks your providers send from their servers.
Web servers
data/webserver/ has the same two lists as the firewall files, ready to include in nginx or Apache. datacenter is exactly the set in all.txt and datacenter.nft: categories cloud, cdn and services. Tor exits, iCloud Private Relay and the crawler lists are not in it; Tor has its own file. The files are rebuilt with the lists, so download them once a day and reload the server.
nginx: each file is a geo block with IPv4 and IPv6 ranges together that sets $datacenter_ip or $tor_ip to 1. Include it in the http context and test the variable in a server or location block.
curl -fsSL -o /etc/nginx/datacenter.nginx.conf \
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/webserver/datacenter.nginx.conf
nginx -t && nginx -s reload
http { include /etc/nginx/datacenter.nginx.conf; server { if ($datacenter_ip) { return 403; } } }
The geo block matches $remote_addr. Behind a proxy or CDN, set up the realip module first, or every visitor gets the proxy's address.
Apache 2.4: each file is a <RequireAll> block that grants everyone except the listed ranges. Include it inside a <Directory>, <Location> or <VirtualHost> block, in place of your own Require all granted.
curl -fsSL -o /etc/apache2/tor.apache.conf \
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/webserver/tor.apache.conf
apachectl configtest && apachectl graceful
<Location "/"> Include /etc/apache2/tor.apache.conf </Location>
Apache checks Require not ip ranges one after another on each request. The Tor file is small; for the datacenter list, the nginx geo block or a firewall set costs less per request.
Node.js
Node 22 or newer, no dependencies, TypeScript declarations included.
npm install datacenter-ip-ranges
The package contains the lists as they were at its release. To follow the daily lists, download the raw files and pass dataDir (see below), or install from GitHub with npm install github:moonetix/datacenter-ip-ranges.
const { isDatacenterIp, lookup } = require('datacenter-ip-ranges'); isDatacenterIp('3.5.140.1'); // true (AWS) isDatacenterIp('::ffff:104.16.0.1'); // true (Cloudflare, IPv4-mapped form from a dual-stack socket) isDatacenterIp('192.168.1.10'); // false isDatacenterIp('not an ip'); // false lookup('172.224.226.1'); // [ { provider: 'icloud-private-relay', category: 'privacy-relay' } ] lookup('8.8.8.8'); // [ { provider: 'google-services', category: 'services' } ] lookup('198.51.100.7'); // [] lookup('nope'); // null
isDatacenterIp() checks cloud, cdn and services. lookup() checks every category and returns all lists that contain the address; Private Relay egress often sits inside Cloudflare or Akamai space, so one address can match more than one list.
For a custom set, build your own matcher. It loads in about 100 ms and is meant to be built once per process:
const { load } = require('datacenter-ip-ranges'); const cloudOnly = load({ categories: ['cloud'] }); const torOrRelay = load({ categories: ['tor', 'privacy-relay'] }); const awsOnly = load({ providers: ['aws'] }); const fromMyCopy = load({ dataDir: '/etc/dcip' }); // a folder with sources.json, providers/ cloudOnly.contains(req.socket.remoteAddress); torOrRelay.lookup('185.220.101.1');
The matcher cuts all ranges into disjoint segments, each with a bit mask of the lists that cover it, and answers with one binary search. Plain IPv4 strings skip the general parser. On a small VPS, one million IPv4 lookups take about 160 ms; IPv6 lookups use BigInt and take a few microseconds each.
createMatcher(lists) from datacenter-ip-ranges/matcher builds a matcher from your own CIDR lists, if you want the same engine for an allow list or your own blocklist.
PHP
PHP 8.1 or newer, one file, no extensions beyond the standard ones.
composer require moonetix/datacenter-ip-ranges
Composer autoloads the file. Without Composer, clone the repo (or copy php/ and data/ side by side) and include it:
require __DIR__ . '/datacenter-ip-ranges/php/DatacenterIp.php'; if (is_datacenter_ip($_SERVER['REMOTE_ADDR'])) { // Hold the reward for review, ask for verification, or skip the bonus. } // Other categories or your own copy of the data: $torOrRelay = DatacenterIp::fromDataDir('/etc/dcip', ['tor', 'privacy-relay']); $torOrRelay->contains('185.220.101.1');
is_datacenter_ip() loads the lists once per process. Under PHP-FPM that is once per request, about 25 ms. For busy sites, export once after each download and let OPcache keep the arrays in shared memory:
// After downloading new lists (cron): DatacenterIp::fromDataDir('/etc/dcip')->exportPhp('/etc/dcip/datacenter.php'); // In the request: $dc = DatacenterIp::fromPhp('/etc/dcip/datacenter.php'); $dc->contains($_SERVER['REMOTE_ADDR']);
Python
Python 3.8 or newer, standard library only, one file. Copy python/datacenter_ip.py next to a copy of data/, or pass data_dir:
from datacenter_ip import is_datacenter_ip, lookup, verify_bot, load is_datacenter_ip("3.5.140.1") # True (AWS) lookup("185.220.101.1") # [{'provider': 'tor', 'category': 'tor'}] tor_or_relay = load("/etc/dcip", categories=("tor", "privacy-relay")) tor_or_relay.contains("185.220.101.1")
The functions load the lists once per process. From the command line, python3 python/datacenter_ip.py 3.5.140.1 prints every list that contains the address, with the same exit codes as bin/check.js.
Verify crawlers
Anyone can put Googlebot in a user agent. The operators below publish the addresses their crawlers use, and verifyBot() checks a request against them:
const { verifyBot } = require('datacenter-ip-ranges'); verifyBot(req.headers['user-agent'], clientIp); // null no known crawler claimed // { bot: 'google', name: 'Google', verified: true, provider: 'googlebot' } // { bot: 'gptbot', name: 'OpenAI GPTBot', verified: false, provider: null } the claim is false
$check = verify_bot($_SERVER['HTTP_USER_AGENT'] ?? '', $clientIp); if ($check !== null && !$check['verified']) { // Claims to be a crawler, is not one: treat it like any other scraper. }
verify_bot(user_agent, client_ip) # same result as a dict
verified: true means the address is in a list the claimed crawler's operator publishes. A typical policy lets verified crawlers past rate limits and treats false claims like any other scraper. Whether you want a given crawler at all is a separate decision for robots.txt or your own rules.
| Bot | User-agent tokens | Lists |
|---|---|---|
Googlebot, Storebot-Google, Google-InspectionTool, GoogleOther, Google-CloudVertexBot, AdsBot-Google, Mediapartners-Google, APIs-Google, FeedFetcher-Google, Google-Read-Aloud, Google-Site-Verification, GoogleProducer, Google-Safety |
googlebot, google-special-crawlers, google-user-fetchers, google-user-fetchers-google |
|
| Bing | bingbot, adidxbot, BingPreview, MicrosoftPreview |
bingbot |
| Applebot | Applebot |
applebot |
| DuckDuckBot | DuckDuckBot |
duckduckbot |
| OpenAI GPTBot | GPTBot |
gptbot |
| OpenAI OAI-SearchBot | OAI-SearchBot |
oai-searchbot |
| OpenAI ChatGPT-User | ChatGPT-User |
chatgpt-user |
| PerplexityBot | PerplexityBot |
perplexitybot |
| Perplexity-User | Perplexity-User |
perplexity-user |
| MistralAI-User | MistralAI-User |
mistralai-user |
| Common Crawl CCBot | CCBot |
ccbot |
| Ahrefs | AhrefsBot, AhrefsSiteAudit |
ahrefsbot |
Tokens match case-insensitively anywhere in the user agent. BOTS (DatacenterIp::BOTS, datacenter_ip.BOTS) exports the table, and claimedBot() returns the entry a user agent claims without checking the address. Crawlers whose operators publish no address list, such as YandexBot or Baiduspider, are not covered; their operators document reverse DNS checks instead.
Command line
node bin/check.js 3.5.140.1 8.8.8.8 192.168.1.10 172.224.226.1
3.5.140.1 aws cloud 3.5.128.0/19
8.8.8.8 google-services services 8.8.8.0/24
192.168.1.10 not listed
172.224.226.1 icloud-private-relay privacy-relay 172.224.226.0/26
Options: --json for JSON output, --only cloud,cdn to limit categories, --data <dir> for another copy of the data, - to read addresses from stdin. Exit codes follow grep: 0 if any address is listed, 1 if none is, 2 for invalid input. Installed from GitHub, the same command is available as npx datacenter-ip.
# Which of yesterday's sign-up IPs came from a datacenter? cut -d, -f3 signups.csv | node bin/check.js --only cloud,cdn,services - | grep -v 'not listed'
PHP has the same check: php php/check.php 3.5.140.1.
Behind a proxy or CDN
Check the address of the visitor, not of your own infrastructure. If your site runs behind Cloudflare, a load balancer or a reverse proxy, the TCP peer is that proxy, and every request matches. Read the client address from the header your proxy sets (CF-Connecting-IP, X-Forwarded-For, X-Real-IP), and trust that header only when the request actually came from your proxy. The cdn lists help with the second part:
const cdn = load({ categories: ['cdn'] }); const peer = req.socket.remoteAddress; const client = cdn.contains(peer) ? req.headers['cf-connecting-ip'] : peer;
Do not apply these lists to your postback endpoint. Networks deliver postbacks from their servers, which are datacenter addresses by design. Verify the postback signature instead.
Sources and terms
Each feed is published by the provider for allow lists, firewalls or geolocation. None of them states a license that forbids redistribution, except Microsoft's Download Center terms, which is why Azure is not in the committed data (see below). Check the terms yourself before you build a commercial product on them.
| Provider | Category | Feed | Documentation | Terms |
|---|---|---|---|---|
| Amazon Web Services | cloud | ip-ranges.json | AWS IP address ranges | Published for filtering and allow lists; no license stated |
| Google Cloud | cloud | cloud.json | Google IP address ranges | Published for allow lists; no license stated |
| Google services | services | goog.json minus cloud.json | Google IP address ranges | Published for allow lists; no license stated |
| Oracle Cloud | cloud | public_ip_ranges.json | OCI IP address ranges | Oracle documentation; no license stated |
| DigitalOcean | cloud | geo/google.csv | Linked from DigitalOcean platform docs | RFC 8805 geofeed; no license stated |
| Akamai Cloud (Linode) | cloud | linode-geofeed.csv | Declared in the ARIN record | RFC 8805 geofeed; no license stated |
| Vultr | cloud | geofeed.constant.com | Declared in the ARIN record | RFC 8805 geofeed of AS20473; no license stated |
| Cloudflare | cdn | ips-v4, ips-v6 | Cloudflare IP ranges | Published for allow lists; no license stated |
| Fastly | cdn | public-ip-list | Fastly API reference | Public API endpoint; no license stated |
| GitHub | services | api.github.com/meta | Meta API, GitHub's IP addresses | GitHub Terms of Service, API terms |
| iCloud Private Relay | privacy-relay | egress-ip-ranges.csv | Prepare your network for iCloud Private Relay | Published for server operators and geolocation; no license stated |
| Tor Project | tor | torbulkexitlist | Tor Metrics | Tor Project data is CC0 |
| Googlebot and other Google common crawlers | crawler | common-crawlers.json | Documentation | Published for crawler verification; no license stated |
| Google special-case crawlers (AdsBot, Mediapartners) | crawler | special-crawlers.json | Documentation | Published for crawler verification; no license stated |
| Google user-triggered fetchers | crawler | user-triggered-fetchers.json | Documentation | Published for crawler verification; no license stated |
| Google user-triggered fetchers (Google products) | crawler | user-triggered-fetchers-google.json | Documentation | Published for crawler verification; no license stated |
| Bingbot | crawler | bingbot.json | Documentation | Published for crawler verification; no license stated |
| Applebot | crawler | applebot.json | Documentation | Published for crawler verification; no license stated |
| DuckDuckBot | crawler | duckduckbot.json | Documentation | Published for crawler verification; no license stated |
| OpenAI GPTBot | crawler | gptbot.json | Documentation | Published for crawler verification; no license stated |
| OpenAI OAI-SearchBot | crawler | searchbot.json | Documentation | Published for crawler verification; no license stated |
| OpenAI ChatGPT-User | crawler | chatgpt-user.json | Documentation | Published for crawler verification; no license stated |
| PerplexityBot | crawler | perplexitybot.json | Documentation | Published for crawler verification; no license stated |
| Perplexity-User | crawler | perplexity-user.json | Documentation | Published for crawler verification; no license stated |
| MistralAI-User | crawler | mistralai-user-ips.json | Documentation | Published for crawler verification; no license stated |
| Common Crawl CCBot | crawler | ccbot.json | Documentation | Published for crawler verification; no license stated |
| AhrefsBot and AhrefsSiteAudit | crawler | crawler-ip-ranges | Documentation | Published for crawler verification; no license stated |
| Microsoft Azure | cloud | Service Tags download page | Azure service tags | Not redistributed, see below |
Microsoft Azure. The weekly Service Tags file comes from the Microsoft Download Center, and the Microsoft Terms of Use restrict reposting its downloads. The script supports it, but only for your own copy:
node scripts/update.js --include azure --out /etc/dcip
That reads the current file name from the download page and adds the AzureCloud tag as providers/azure.txt and to all.txt. data/providers/azure.txt is in .gitignore. Without it, Azure coverage is partial: GitHub's Actions runner ranges, which are Azure addresses, are included through the GitHub feed.
Providers without an official feed are not included. Hetzner, OVHcloud, Scaleway, Contabo, Alibaba Cloud and many smaller hosts do not publish one, and this repo does not fill the gap with ASN or WHOIS guesses. If your provider publishes a feed, open an issue with the link to its documentation.
Updates
.github/workflows/update.yml runs every day at 03:17 UTC and on demand. It fetches all feeds, rebuilds data/, runs the tests, and commits only if a range list changed. A run where only the fetch time moved commits nothing, so the history shows real changes only.
flowchart LR
F["12 provider<br/>feeds"] --> N["Parse, drop private<br/>and too-broad<br/>prefixes, merge"]
N --> G{"Failed, empty<br/>or shrank<br/>by half?"}
G -- no --> D["data/<br/>providers, categories,<br/>all.txt, ranges.json"]
G -- "yes: keep previous<br/>list, mark stale" --> D
Loading
Failures are handled per feed. Each request gets 60 seconds and three attempts. If a feed is down, returns nothing, or suddenly returns less than half of its previous ranges, its previous list is kept and the feed is marked stale in data/sources.json with the reason. Nothing is written if every feed fails. Entries that do not parse, prefixes broader than /8 (IPv4) or /16 (IPv6), and private, loopback, link-local, multicast and documentation ranges are dropped.
A feed that is still stale on the next run, or a run that fails outright, opens a single issue labeled automation with the feed, the error and the source URL. The issue closes itself once every feed is fresh again.
To rebuild by hand:
node scripts/update.js # writes data/ node scripts/update.js --out /etc/dcip # writes somewhere else GITHUB_TOKEN=... node scripts/update.js # optional: authenticated api.github.com/meta request
--out and --include azure are the only arguments. GITHUB_TOKEN is the only environment variable; it is sent to api.github.com and nowhere else, and only matters if your IP has used up the unauthenticated API limit.
Troubleshooting
| What you see | Cause and fix |
|---|---|
| Every request matches | You are checking your proxy or CDN, not the visitor. See Behind a proxy or CDN. |
invalid IP address and exit code 2 from bin/check.js |
The argument is not an IP. Ports (1.2.3.4:443) are not accepted; brackets and IPv6 zone IDs are. |
check.js prints its usage and exits 2 |
--only got a category that does not exist. Valid: cloud, cdn, services, privacy-relay, tor. |
A feed shows "status": "stale" with fetch or parse failed: ... HTTP 403 |
For github, the API rate limit. Set GITHUB_TOKEN. Other feeds recover on the next run; the previous list stays in place. |
Every feed failed. Nothing was written. |
No outbound HTTPS from the machine, or a proxy blocks it. The message lists the reason per feed. |
no ServiceTags_Public link on the download page |
Microsoft changed the Azure download page. Azure is opt-in; run without --include azure and open an issue. |
Limitations
- Residential proxies and most commercial VPNs are not covered. They route traffic through home and mobile connections or through hosts without public feeds. This list catches the cheap, server-based share of bad traffic. For the rest you need an IP intelligence service, device signals and behavior.
- One signal, not a verdict. People do use cloud desktops, corporate proxies and VPNs hosted on rented servers. Prefer soft actions: hold a reward for review, require a verified phone, skip a sign-up bonus. Block outright only where a human has no reason to come from a server, for example on an offer click or a reward claim.
- Tor and Private Relay are real people. Apple asks servers to treat Private Relay addresses like carrier-grade NAT: many users share one address. They are separate lists for that reason.
- Freshness. Lists are at most a day behind the feeds, and the feeds themselves lag new allocations. IP addresses also move between owners.
- Crawler lists cover only operators that publish one. A crawler that is not in the table is not verified either way, and an operator can start crawling from new addresses before its list catches up.
- Provider feeds include their own infrastructure. Google's services list contains crawlers and fetch proxies; GitHub's contains its web and API servers. That is usually what you want for fraud filtering, but not for every use.
Further reading
- OWASP Automated Threats to Web Applications: the abuse patterns this kind of check helps with.
- Moonetix Academy: What is IP reputation? and Ad fraud and bot traffic.
Development
npm test # Node tests; also runs the PHP and Python tests and the cross-checks if php and python3 are installed php php/test.php # PHP tests only python3 python/test_datacenter_ip.py # Python tests only npm run update # fetch all feeds and rebuild data/
The tests cover address parsing (IPv4-mapped, zone IDs, brackets, RFC 5952 output), range arithmetic, the matcher against a brute-force check on random data and range edges, the build step against fake feeds (merging, special ranges, stale feeds, the shrink guard, opt-in sources), the integrity of the committed data, the CLI, the firewall and web server files (including an nginx -t check when nginx is installed), crawler verification, and agreement between the Node, PHP and Python helpers on boundary addresses of every list.
CI runs the same commands on every push and pull request. To propose a new provider, use the Add a provider feed issue form.
Layout: src/ip.js parsing and range math, src/matcher.js lookup engine, src/sources.js feed definitions and parsers, src/bots.js the crawler table and verifyBot(), src/build.js fetch and write, scripts/update.js and bin/check.js command line entry points, php/ the PHP helper, python/ the Python helper.
About Moonetix
Moonetix is an affiliate network. Publishers add its offerwall to apps, games, Discord and Telegram communities and rewards sites, and users earn the publisher's own currency by completing offers. Every click and conversion goes through traffic checks like the ones described in How Moonetix Checks Traffic Quality.
Related open source: offerwall-postback-handlers, postback-tester, awesome-offerwalls, and more at github.com/moonetix.
License
Code: MIT. See LICENSE. The range data belongs to the providers listed in Sources and terms and is subject to their terms.