Edge Server Log File Crawler & Bot Verification Script
Node.js and Python script to parse edge server access logs, verify genuine Googlebot IP addresses via reverse DNS, and calculate crawl frequency by directory.
Why Reverse DNS Verification Matters
Scrapers and rogue bots frequently forge Googlebot User-Agent strings. Validating crawler IPs against reverse DNS ensures that your log file analysis reflects real search engine crawl patterns rather than automated scrapers.
Implementation Code & Script
Verifies whether a crawler IP belongs to genuine Googlebot using reverse and forward DNS lookups.
import dns from 'dns/promises';
export async function isGenuineGooglebot(ip) {
try {
const hostnames = await dns.reverse(ip);
for (const host of hostnames) {
if (host.endsWith('.googlebot.com') || host.endsWith('.google.com')) {
const resolvedIps = await dns.resolve(host);
if (resolvedIps.includes(ip)) {
return true;
}
}
}
return false;
} catch (err) {
return false;
}
}How to cite and attribute this tool
MIT LicenceThis resource is free, open and un-gated under the MIT Open Source Licence. You are encouraged to use, integrate and cite it with attribution:
Geraghty, G. (2026). Edge Server Log File Crawler & Bot Verification Script. Gordon Geraghty Resources Hub. https://gordongeraghty.com/resources/technical-seo/log-file-analysis-script
BibTeX Format
@misc{geraghty_log_file_analysis_script,
author = {Geraghty, Gordon},
title = {Edge Server Log File Crawler & Bot Verification Script},
year = {2026},
url = {https://gordongeraghty.com/resources/technical-seo/log-file-analysis-script},
note = {Head of Performance, Empire Amplify}
}Changelog & Version History
v1.0.0Initial release of edge log parser with reverse DNS bot verification.