PhantomCrawl is a command-line web crawler for developers who need to extract data from the real web - not just simple static pages. Most scrapers break the moment they hit a site with anti-bot protection. PhantomCrawl does not.

It works by trying the fastest, cheapest method first and only escalating to more powerful techniques when necessary. This means it is fast on simple sites and capable on protected ones - all automatically.

How it works

PhantomCrawl uses a 4-layer escalation engine. Every URL starts at Layer 1 and only moves to the next layer if the current one fails or returns no meaningful content. Most of the internet never leaves Layer 1.

Installation

PhantomCrawl ships as a pre-built binary. No Go installation required. Download the binary for your platform and move it to your PATH.

Linux / macOS / Termux

# Linux
chmod +x phantomcrawl-linux-amd64
sudo mv phantomcrawl-linux-amd64 /usr/local/bin/phantomcrawl

# Termux (Android)
chmod +x phantomcrawl-linux-arm64
mv phantomcrawl-linux-arm64 $PREFIX/bin/phantomcrawl

# macOS
chmod +x phantomcrawl-darwin-arm64
sudo mv phantomcrawl-darwin-arm64 /usr/local/bin/phantomcrawl

Build from source

Requires Go 1.21+

git clone https://github.com/var-raphael/PhantomCrawl.git
cd PhantomCrawl
go build -ldflags="-s -w" -o phantomcrawl .

Quickstart

Get your first crawl running in under 2 minutes.

phantomcrawl init

https://example.com
https://news.ycombinator.com
https://cloudflare.com

phantomcrawl start

Crawl Settings

All configuration lives in crawl.json in your working directory. Here is a full example with every field:

{
  "urls_file": "./urls.txt",
  "batch_size": 3,
  "throttle": 5,
  "depth": 0,
  "depth_limit": 10,
  "stay_on_domain": true,
  "output": "~/phantomcrawl/scraped",
  "retry": {
    "max_attempts": 3,
    "backoff": "exponential",
    "respect_retry_after": true
  }
}

Scrape Options

Control what data is extracted from each page.

"scrape": {
  "text": true,        // Extract visible text content
  "links": true,       // Extract all hyperlinks (resolved to absolute URLs)
  "images": "links_only", // Extract image URLs
  "videos": "links_only", // Extract video/iframe URLs
  "documents": "links_only", // Extract PDF, DOC, XLS links
  "emails": false,     // Extract email addresses
  "phone_numbers": false, // Extract phone numbers
  "metadata": true     // Extract meta tags (title, description, OG, Twitter)
}

All extracted links are automatically resolved to absolute URLs. A relative link like /about on https://example.com becomes https://example.com/about.

AI Cleaning

Raw HTML is noisy - navigation menus, cookie banners, footers, ads. AI cleaning strips all of that and extracts only the meaningful content. PhantomCrawl supports Groq (free tier) and OpenAI.

Setup with Groq (recommended, free)

GROQ_KEY_1=gsk_your_key_here
GROQ_KEY_2=gsk_another_key_here

"ai": {
  "enabled": true,
  "save_raw": true,
  "save_cleaned": true,
  "max_concurrent_cleans": 1,
  "model": "llama-3.3-70b-versatile",
  "provider": "groq",
  "key_rotation": "random",
  "keys": ["$GROQ_KEY_1", "$GROQ_KEY_2"],
  "prompt": "You are a web content extractor..."
}

AI config fields

Proxy Setup

Route requests through proxies to avoid IP bans on large crawls. Proxies are tunneled at the TCP level - traffic goes through the proxy before the TLS handshake, making it look like the request originated from the proxy's IP.

PROXY_1=http://user:[email protected]:8080
PROXY_2=http://proxy2.example.com:8080

"anti_bot": {
  "rotate_user_agents": true,
  "request_delay_jitter": true,
  "proxy": {
    "enabled": true,
    "key_rotation": "random",
    "urls": ["$PROXY_1", "$PROXY_2"]
  }
}

Layer 3 - Headless Browser

Layer 3 is used automatically when Layers 1, 2, and 2.5 all fail to return meaningful content. It runs a real browser to fully render the page.

Option A: Local Chrome (automatic)

If Chrome or Chromium is installed on your machine, PhantomCrawl detects it automatically and uses go-rod to control it. No configuration needed.

phantomcrawl start
# You will see:
✓ Chrome detected. Using go-rod for Layer 3.

Option B: Browserless (cloud Chrome)

If Chrome is not installed (common on servers and mobile), you can use Browserless - a cloud Chrome service. Free tier available.

BROWSERLESS_KEY=your_key_here

"layer3": {
  "key_rotation": "random",
  "keys": ["$BROWSERLESS_KEY"]
}

Environment Variables

Never put API keys directly in crawl.json. Use a .env file instead and reference variables with $VAR_NAME in your config.

# AI keys
GROQ_KEY_1=gsk_abc123...
GROQ_KEY_2=gsk_def456...

# Browser
BROWSERLESS_KEY=your_browserless_key

# Proxies
PROXY_1=http://host:port
PROXY_2=http://user:pass@host:port

.env
crawl.json
urls.txt
scraped/

Commands

Resume behavior

PhantomCrawl tracks every crawled URL in a local SQLite database at ~/.phantomcrawl/state.db. If a crawl is interrupted - whether by a network error, token quota, or manual stop - you can resume exactly where you left off by running phantomcrawl start again without resetting. Already-crawled URLs are skipped. Pending AI cleans are retried.

Output Format

Each crawled URL creates a folder under your output directory. The folder name is derived from the page title or URL slug.

~/phantomcrawl/scraped/
  cloudflare.com/
    cloudflare/
      raw.json      # Full extracted data + raw HTML
      cleaned.json  # AI-cleaned text + structured data
    en-gb/
      raw.json
      cleaned.json

raw.json

Contains everything extracted from the page before AI processing.

{
  "url": "https://cloudflare.com",
  "title": "Cloudflare - The Web Performance & Security Company",
  "content": "Extracted text content...",
  "links": ["https://cloudflare.com/products", "..."],
  "images": ["https://cloudflare.com/img/logo.svg"],
  "documents": [],
  "emails": [],
  "metadata": { "description": "...", "og:image": "..." },
  "layer_used": "layer1",
  "crawled_at": "2026-03-14T...",
  "raw": "<!DOCTYPE html>..."
}

cleaned.json

AI-processed version with clean text and all structured data included.

{
  "url": "https://cloudflare.com",
  "title": "Cloudflare - The Web Performance & Security Company",
  "cleaned": "Cloudflare is a global network...\nProducts: CDN, DDoS protection...",
  "links": ["https://cloudflare.com/products", "..."],
  "images": ["https://cloudflare.com/img/logo.svg"],
  "metadata": { "description": "..." },
  "crawled_at": "2026-03-14T..."
}

Converting to other formats

Output is JSON. Convert to any format using standard tools:

import json, csv

with open('scraped/cloudflare.com/cloudflare/cleaned.json') as f:
    data = json.load(f)

print(data['cleaned'])   # AI-cleaned text
print(data['links'])     # All extracted links
print(data['metadata'])  # Page metadata

PhantomClean

While PhantomCrawl handles extraction, PhantomClean handles quality. It processes folders in configurable concurrent batches, watches for new files in real time as your scraper runs, and exports results in JSON, CSV, XML, TXT, or HTML.

Installation

PhantomClean ships as a pre-built binary. Download for your platform from GitHub Releases.

# Linux
chmod +x phantomclean-linux-amd64
sudo mv phantomclean-linux-amd64 /usr/local/bin/phantomclean

# Termux (Android)
chmod +x phantomclean-linux-arm64
mv phantomclean-linux-arm64 $PREFIX/bin/phantomclean

# macOS
chmod +x phantomclean-darwin-arm64
sudo mv phantomclean-darwin-arm64 /usr/local/bin/phantomclean

Input Format Requirements

PhantomClean expects scraped data in a specific folder/file structure. PhantomCrawl produces this automatically — using any other scraper requires matching this format.

scraped/
  site-name/
    page-title/
      raw.json         # required — full extracted data
      cleaned.json     # optional — preferred over raw.json if present

Each raw.json must contain at least one of these fields. PhantomClean checks them in this order:

If none of content, text, or html are present, the file is skipped with reason no text content found.

These fields are optional but are preserved in the cleaned output when present:

Configuration

Run phantomclean init to generate a cleaner.json template. All settings live here.

{
  "concurrent_folders": 3,         // folders processed concurrently per batch
  "folder_to_clean": "./scraped",  // root scraped folder (PhantomCrawl output)
  "output_folder": "./organized",  // where cleaned files are written
  "output_file_name": "organized", // output filename without extension
  "export_format": ["json"],       // json, csv, xml, txt, html, all_formats
  "watch_mode": true,              // keep running after initial scan
  "watch_debounce_seconds": 2,     // seconds of silence before flushing new files
  "min_word_count": 10,            // skip files below this word count
  "boilerplate_threshold": 3,      // sentences seen this many times = boilerplate
  "quality_score_minimum": 0.5,    // skip files scoring below this (0.0-1.0)
  "language": "en",               // only keep english. use "all" to disable
  "content_type": "text",          // text, code, or mixed
  "strip_nav_links": true,         // remove navigation links from output
  "overwrite": false,             // re-process already-cleaned files
  "resume": true,                 // skip already-processed files on restart
  "omit_folders": [],              // folder names to skip entirely
  "output": {
    "zip_on_complete": true,
    "zip_name": "dataset-{date}-{file_count}-files"
  }
}

Cleaning Pipeline

Every file passes through 4 layers in sequence. A file is skipped if it fails the quality or word count threshold after Layer 3.

AI Cascade

PhantomClean supports three AI providers in a cascade — it tries each in order and falls back to the next if one fails or hits a rate limit. If all fail, it falls back to rule-based output and marks the file as rules-only so you can retry with phantomclean clean-ai later.

"ai": {
  "enabled": true,
  "fallback_to_rules": true,      // use rule output if all AI fails
  "only_if_no_cleaned": false,   // skip AI if cleaned.json already exists
  "prompt_file": "prompt.txt",   // path to your prompt file
  "rotate": "random",            // random or sequential key rotation
  "timeout_seconds": 15,          // per-request timeout
  "max_retries": 2,               // retries per provider before trying next
  "providers": [
    { "provider": "groq",      "model": "llama-3.3-70b-versatile", "keys": ["$GROQ_KEY_1"] },
    { "provider": "openai",    "model": "gpt-4o-mini",            "keys": ["$OPENAI_KEY_1"] },
    { "provider": "anthropic", "model": "claude-haiku-4-5-20251001", "keys": ["$ANTHROPIC_KEY_1"] }
  ]
}

GROQ_KEY_1=gsk_your_key_here
GROQ_KEY_2=gsk_another_key_here
OPENAI_KEY_1=sk_your_key_here
ANTHROPIC_KEY_1=your_key_here

Chunking

Documents over 2000 words are automatically split into chunks before being sent to the AI. Each chunk is cleaned independently and the results are joined back together. This means even very large documents (10,000+ words) are handled without hitting token limits or timeouts.

Commands

Output Format

Cleaned files mirror the input folder structure under output_folder.

organized/
  site-name/
    page-title/
      organized.json   # cleaned output
      organized.csv    # if csv in export_format
      organized.txt    # if txt in export_format

organized.json schema

{
  "url": "https://example.com/page",
  "title": "Page Title",
  "content": "Cleaned text content...",
  "word_count": 1842,
  "quality_score": 0.94,
  "language": "en",
  "cleaned_at": "2026-03-18T12:00:00Z",
  "ai_used": "groq/llama-3.3-70b-versatile",
  "layer_used": "layer1",
  "crawled_at": "2026-03-18T11:58:00Z",
  "links": ["https://example.com/other"],
  "images": ["https://example.com/img.jpg"],
  "emails": [],
  "phones": []
}

Running the Full Pipeline

PhantomCrawl and PhantomClean are designed to run simultaneously. Point PhantomClean at PhantomCrawl's output folder and both tools handle their half of the pipeline in real time.

phantomcrawl init
# Edit crawl.json — set output to ./scraped
# Add URLs to urls.txt

phantomclean init
# Edit cleaner.json — set folder_to_clean to ./scraped
# Add AI keys to .env

# Terminal 1
phantomcrawl start

# Terminal 2
phantomclean start

# After the crawl is done, retry any rules-only files
phantomclean clean-ai

Contributing

PhantomCrawl is open source under the BSL license - free for personal and non-commercial use. Contributions are welcome.

git clone https://github.com/var-raphael/PhantomCrawl.git
cd PhantomCrawl

go build ./...
go run main.go init
go run main.go start