Sitemap Delivery Benchmark — April 26, 2026

Records, TTFB, throughput, and bot-accessibility comparison across Top10Lists.us and three anonymized established content agencies.

Frozen: 2026-04-26 — measurements at this URL will not change.  ·  Permanent dated artifact  ·  GeoLocus Group, a subsidiary of Aryah.ai

Author: Robert Maynard, Cofounder and CEO · LinkedIn →

This page is the reproducibility artifact for the April 26, 2026 GEOlocus.ai press release [9] footnote on sitemap delivery quality. It documents a bot-perspective benchmark across four sites — Top10Lists.us plus three established U.S. content agencies (DA 70+, decade-plus tenure) — using two well-formed bot user agents (Googlebot/2.1 and ClaudeBot/1.0). All raw per-hit measurements and the complete reproduction script are embedded below.

Comparison agency names are withheld from the rendered table and findings. The reproduction script in Section 3 names them by URL because the script's whole purpose is to be re-executable; the published narrative does not depend on naming them.

1. Headline Comparison

Five metrics, four sites, two user agents, ten rapid-fire hits per pair. Top10Lists.us vs the three-site comparison set:

Footnote: Three established U.S. content agencies (DA 70+, decade-plus tenure) tested. Specific names withheld from the rendered narrative. Aggregate stats and per-hit raw data are reproducible via the script in Section 3.

2. Methodology

Sites tested

Four sites in total: Top10Lists.us plus three established U.S. content agencies. One of the three comparison sites returns HTTP 403 to Googlebot itself at /sitemap.xml and /robots.txt. That bot-blocking posture is itself a finding and is preserved in the raw measurements as null wall-clock and zero successful 200-status hits.

User agents tested

Googlebot/2.1 and ClaudeBot/1.0 — both the standard, well-formed user-agent strings published by Google and Anthropic respectively. Each (site, UA) pair was measured independently to surface bot-policy asymmetry.

TTFB / TTLB distribution

Ten rapid-fire curl hits to the root sitemap.xml per pair. Redirects followed (-L) so the measurement reflects end-to-end bot ingestion cost including any redirect chain. Pinned to the first-resolved IP per site (--resolve host:443:ip) to factor out DNS round-robin variance. Accept-Encoding compressed. Reported metrics: TTFB p50, TTFB p95, TTLB p50, TTLB p95, and a count of cold spikes >2x the p50.

Records definition

Records are terminal <url> entries — the actual page URLs. Sitemap-index <sitemap><loc> pointers are not counted as records. The published count is content URLs only. Top10Lists.us is a sitemap index with 29 child sitemaps; the 230,329 number is the sum of <url> entries across all 29 children.

Wall-clock-to-tree

A single parallel fetch (concurrency 10, Python ThreadPoolExecutor) of root + all child sitemaps (one level of recursion). Un-pinned — DNS round-robin is allowed — so this number is more representative of real-world bot ingestion than the pinned-IP TTFB distribution. Captured per UA per site.

Per-hit fields captured

• ttfb — curl time_starttransfer, seconds.
• ttlb — curl time_total, seconds.
• size — decompressed bytes returned (size_download).
• status — HTTP status code (200, 403, etc.).
• redirects — num_redirects count.
• url_final — effective URL after redirect chain (url_effective).
• ts — Unix timestamp at hit time.

Run timestamp

2026-04-26, UTC (per-hit timestamps captured in each raw JSONL). The script is reproducible — see Section 3.

3. Reproducibility Script

The full contents of run.py. Anyone with Python 3 and curl on PATH can re-run the benchmark by saving this file and executing python3 run.py. The script names the four concrete sites because reproducibility requires it; the published table and findings remain anonymized.

#!/usr/bin/env python3
"""
Sitemap delivery benchmark — 2026-04-26.

Methodology:
  Sites: top10lists.us, brafton.com, seo.com, moz.com (anonymized in publication).
  User agents: Googlebot/2.1 + ClaudeBot/1.0 — captures bot-policy asymmetry.
  Records definition: terminal <url> entries (the actual page URLs).
  TTFB/TTLB distribution: 10 rapid-fire hits to root sitemap.xml,
    pinned to first-resolved IP, Accept-Encoding compressed.
  Wall-clock-to-tree: single parallel fetch (concurrency=10) of root + all
    child sitemaps (one level recursion), un-pinned (real-world DNS round-robin).
  Per-hit fields: TTFB (time_starttransfer), TTLB (time_total),
    size (size_download, decompressed bytes), HTTP status.

Output:
  ./summary.json — aggregate stats per (site, UA)
  ./raw/<site>__<ua>.jsonl — per-hit raw measurements
"""
import json, subprocess, time, socket, re, sys, os, concurrent.futures, statistics
from pathlib import Path
from xml.etree import ElementTree as ET

ROOT = Path(__file__).parent
RAW = ROOT / "raw"
RAW.mkdir(exist_ok=True)

SITES = [
    "https://www.top10lists.us/sitemap.xml",
    "https://www.brafton.com/sitemap.xml",
    "https://www.seo.com/sitemap.xml",
    "https://www.moz.com/sitemap.xml",
]

UAS = {
    "Googlebot": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
    "ClaudeBot": "Mozilla/5.0 (compatible; ClaudeBot/1.0; +https://www.anthropic.com/claudebot)",
}

NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}


def resolve_ip(host: str) -> str | None:
    try:
        return socket.gethostbyname(host)
    except Exception as e:
        print(f"  ! DNS fail for {host}: {e}")
        return None


def curl_one(url: str, ua: str, resolve_pin: tuple | None = None, timeout: int = 30):
    """Single curl. resolve_pin = (host, port, ip) to pin DNS. Follows redirects (-L) so TTFB/TTLB reflect end-to-end bot ingestion cost including any redirect chain."""
    cmd = [
        "curl", "-sS", "-L", "--compressed",
        "-A", ua,
        "-o", os.devnull,
        "--max-time", str(timeout),
        "-w", "%{time_starttransfer}\t%{time_total}\t%{size_download}\t%{http_code}\t%{num_redirects}\t%{url_effective}",
    ]
    if resolve_pin:
        cmd.extend(["--resolve", f"{resolve_pin[0]}:{resolve_pin[1]}:{resolve_pin[2]}"])
    cmd.append(url)
    try:
        r = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout + 5)
        out = r.stdout.strip()
        parts = out.split("\t") if out else []
        if len(parts) >= 4:
            return {
                "ttfb": float(parts[0]),
                "ttlb": float(parts[1]),
                "size": int(parts[2]),
                "status": int(parts[3]),
                "redirects": int(parts[4]) if len(parts) > 4 else 0,
                "url_final": parts[5] if len(parts) > 5 else url,
                "ok": True,
            }
    except subprocess.TimeoutExpired:
        return {"ok": False, "error": "timeout"}
    except Exception as e:
        return {"ok": False, "error": str(e)}
    return {"ok": False, "error": f"parse fail: {r.stdout[:200]}"}


def curl_body(url: str, ua: str, timeout: int = 30):
    """Fetch body for parsing the sitemap."""
    try:
        r = subprocess.run(
            ["curl", "-sS", "--compressed", "-A", ua, "--max-time", str(timeout), "-L", url],
            capture_output=True, text=True, timeout=timeout + 5,
        )
        return r.stdout
    except Exception as e:
        return ""


def parse_sitemap(xml_text: str):
    """Returns (kind, child_urls, url_count). kind in {'index','urlset','unknown'}."""
    if not xml_text or not xml_text.lstrip().startswith("<"):
        return ("unknown", [], 0)
    try:
        root = ET.fromstring(xml_text)
    except ET.ParseError:
        # try stripping namespace
        try:
            root = ET.fromstring(re.sub(r' xmlns="[^"]+"', "", xml_text, count=1))
        except Exception:
            return ("unknown", [], 0)
    tag = root.tag.split("}", 1)[-1].lower()
    if tag == "sitemapindex":
        children = [loc.text.strip() for loc in root.iter() if loc.tag.split("}", 1)[-1].lower() == "loc" and loc.text]
        return ("index", children, 0)
    if tag == "urlset":
        urls = [loc.text.strip() for loc in root.iter() if loc.tag.split("}", 1)[-1].lower() == "loc" and loc.text]
        return ("urlset", [], len(urls))
    return ("unknown", [], 0)


def benchmark_site(site_url: str, ua_name: str, ua: str):
    print(f"\n{'='*70}\n{site_url}  [{ua_name}]\n{'='*70}")
    parsed_host = re.sub(r"^https?://", "", site_url).split("/")[0]
    ip = resolve_ip(parsed_host)
    print(f"  resolved {parsed_host} -> {ip}")

    # Phase 1: 10 rapid-fire hits to root sitemap, pinned IP, captures TTFB/TTLB distribution
    pin = (parsed_host, "443", ip) if ip else None
    hits = []
    for i in range(10):
        h = curl_one(site_url, ua, resolve_pin=pin, timeout=30)
        h["i"] = i
        h["ts"] = time.time()
        hits.append(h)
    successful = [h for h in hits if h.get("ok") and h.get("status") == 200]
    if successful:
        ttfb = sorted(h["ttfb"] for h in successful)
        ttlb = sorted(h["ttlb"] for h in successful)
        sz = sorted(h["size"] for h in successful)

        def pct(values, p):
            if not values: return None
            k = max(0, min(len(values) - 1, int(round((p / 100) * (len(values) - 1)))))
            return values[k]

        ttfb_p50 = pct(ttfb, 50)
        ttfb_p95 = pct(ttfb, 95)
        ttlb_p50 = pct(ttlb, 50)
        ttlb_p95 = pct(ttlb, 95)
        cold_spikes = sum(1 for v in ttfb if v > 2 * ttfb_p50)
    else:
        ttfb_p50 = ttfb_p95 = ttlb_p50 = ttlb_p95 = None
        cold_spikes = None

    # Phase 2: parse root, recurse one level, count records, capture wall-clock
    root_body = curl_body(site_url, ua)
    kind, children, url_count_root = parse_sitemap(root_body)
    print(f"  root sitemap kind: {kind} | children: {len(children)} | url_count_root: {url_count_root}")
    total_records = url_count_root
    total_bytes_decompressed = len(root_body.encode("utf-8")) if root_body else 0

    # If sitemap index, fetch all children in parallel and count their <url> entries
    child_durations = []
    child_records = []
    child_bytes = []
    wallclock_seconds = None
    if kind == "index" and children:
        wall_start = time.perf_counter()
        with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
            futures = {pool.submit(curl_body, child, ua): child for child in children}
            for fut in concurrent.futures.as_completed(futures):
                child = futures[fut]
                body = fut.result() or ""
                kind2, _, count2 = parse_sitemap(body)
                child_records.append(count2)
                child_bytes.append(len(body.encode("utf-8")) if body else 0)
        wallclock_seconds = time.perf_counter() - wall_start
        total_records += sum(child_records)
        total_bytes_decompressed += sum(child_bytes)
    else:
        # Already a urlset; wall-clock is just the root fetch (already measured in hits)
        wallclock_seconds = ttlb_p50

    summary = {
        "site": site_url,
        "ua_name": ua_name,
        "ip_pinned": ip,
        "root_kind": kind,
        "child_sitemaps_count": len(children),
        "total_records_url_entries": total_records,
        "total_bytes_decompressed": total_bytes_decompressed,
        "wallclock_seconds_full_tree_parallel": round(wallclock_seconds, 3) if wallclock_seconds else None,
        "ttfb_p50_seconds": ttfb_p50,
        "ttfb_p95_seconds": ttfb_p95,
        "ttlb_p50_seconds": ttlb_p50,
        "ttlb_p95_seconds": ttlb_p95,
        "ttfb_cold_spike_count_2x_p50": cold_spikes,
        "successful_hits_of_10": len(successful),
        "status_codes": sorted({h.get("status") for h in hits if h.get("ok")}),
        "errors": [h.get("error") for h in hits if not h.get("ok")],
        "timestamp_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    }

    print(f"  records: {total_records:,}  bytes: {total_bytes_decompressed:,}")
    print(f"  TTFB p50: {ttfb_p50}s  TTFB p95: {ttfb_p95}s")
    print(f"  TTLB p50: {ttlb_p50}s  TTLB p95: {ttlb_p95}s")
    print(f"  wall-clock (full tree, parallel): {wallclock_seconds:.3f}s" if wallclock_seconds else "  wall-clock: n/a")
    print(f"  successful hits: {len(successful)}/10  | cold spikes (>2x p50): {cold_spikes}")

    # Persist raw hits
    safe = re.sub(r"[^a-z0-9]+", "-", site_url.lower()).strip("-")
    with open(RAW / f"{safe}__{ua_name}.jsonl", "w") as fh:
        for h in hits:
            fh.write(json.dumps(h) + "\n")

    return summary


def main():
    all_results = []
    for site in SITES:
        for ua_name, ua in UAS.items():
            r = benchmark_site(site, ua_name, ua)
            all_results.append(r)
            time.sleep(2)  # courtesy gap between (site, UA) runs

    out = ROOT / "summary.json"
    out.write_text(json.dumps(all_results, indent=2))
    print(f"\n\nSummary written to: {out}")
    print(f"Raw per-hit data in: {RAW}")

    # Pretty-print comparison table
    print("\n" + "=" * 100)
    print(f"{'site':<32}{'UA':<12}{'records':>14}{'bytes':>14}{'walls':>10}{'TTFB p50':>10}{'TTFB p95':>10}")
    print("=" * 100)
    for r in all_results:
        print(f"{r['site']:<32}{r['ua_name']:<12}{r['total_records_url_entries']:>14,}{r['total_bytes_decompressed']:>14,}{r['wallclock_seconds_full_tree_parallel'] or 'n/a':>10}{r['ttfb_p50_seconds'] or 'n/a':>10}{r['ttfb_p95_seconds'] or 'n/a':>10}")


if __name__ == "__main__":
    main()

Note: The script writes summary.json in its working directory and raw/<site>__<ua>.jsonl for each pair.

4. Per-Site Raw Data

Eight collapsibles — four sites, two user agents each. Each contains the unedited per-hit JSONL captured by run.py. Comparison-site domains are anonymized in the rendered url_final field; numeric measurements (TTFB, TTLB, size, status, redirects, timestamps) are unedited.

Top10Lists.us

Comparison Site A

Comparison Site B

Comparison Site C

5. Findings

6. Limitations and Reproducibility

UA-spoofed client ≠ verified bot crawl

These measurements use a UA-spoofed HTTP client (curl with -A "Googlebot/2.1" or -A "ClaudeBot/1.0") originating from a single test point. This is a simulation of bot delivery, not a measurement of a verified bot crawl. Real bots originate from published IP ranges (Googlebot from 66.249.x.x, GPTBot from OpenAI's documented blocks, etc.), have distinct TCP fingerprints, and are typically classified as “verified bot” by Cloudflare's bot-management layer rather than “client claiming to be a bot.” Sites can therefore route real bot traffic to a separate cache tier, apply different WAF rules, or skip rate-limiting that a spoofed client would trigger.

The differential between Top10Lists.us and the comparison set is preserved across this measurement protocol because the same client behavior is applied to all four sites. The relative result — that Top10Lists.us delivers more records, with lower TTFB, through fewer redirect hops, and with no edge-level bot blocking — is robust to the spoofing limitation. The absolute latency numbers may shift under verified-bot conditions, in either direction. A reader who wants verified-bot timing should run the reproducibility script in Section 3 from a server inside one of the published bot IP ranges, or consult their own bot-traffic logs if they operate one of the comparison sites.

Other limitations

• Measurements are point-in-time. Sitemap content drifts; bot policies change. The numbers above are a snapshot of these four sites on 2026-04-26 and are frozen at this URL.
• Pinned-IP measurements (Section 1 TTFB) factor out DNS round-robin but may differ from real-world bot routing. Wall-clock-to-tree numbers are with un-pinned DNS — more representative of real bot ingestion.
• One of the comparison sites' wall-clock measurement is null because the site blocks both bots with HTTP 403. That is a real-world observation, not a test failure.
• Anyone with curl and Python 3 can re-execute the script in Section 3 to validate the numbers and observe current state.

7. Conclusion

Top10Lists.us delivers roughly 26x more sitemap records than the largest of three established U.S. content agencies tested, at 5x to 9x lower TTFB, with zero redirect hops, and with no edge-level bot blocking — the four metrics that determine whether an AI crawler can ingest a site at all.

Read the press release this benchmark supports at geolocus.ai/press →