Home  /  Methodology  /  Sitemap Benchmark
Methodology

Sitemap Delivery Benchmark

A reproducible protocol for measuring and comparing XML sitemap delivery quality across competing web properties -- including response time, URL yield, entry format, and AI-legibility signals.

Published: April 25, 2026  ·  GeoLocus Group, a subsidiary of Aryah.ai

1. Purpose

XML sitemaps are among the first files AI crawlers retrieve when indexing a site. A sitemap that is slow, malformed, incomplete, or missing AI-relevant metadata directly reduces the AI citation surface of that domain. This benchmark provides a standardized, reproducible method for evaluating sitemap delivery quality so that publishers, GEO practitioners, and researchers can make data-driven comparisons across domains.

The benchmark does not measure content quality or domain authority. It measures only the technical delivery layer: can an AI crawler fetch the sitemap reliably, quickly, and in a form that maximizes indexing coverage?

2. Measurement Dimensions

D1 -- Existence and Discoverability

15 pts

Whether a sitemap is declared in /robots.txt and resolves at the declared URL. Discovery check: GET /robots.txt then parse Sitemap: directives. Secondary check: GET /sitemap.xml if robots.txt declares nothing. A site with no sitemap scores 0 on this dimension.

D2 -- Response Time (TTFB)

20 pts

Time To First Byte for the sitemap resource, measured as the p50 across 5 cold-fetch requests spaced 60 seconds apart. Thresholds: ≤200 ms = full credit (20 pts), ≤500 ms = partial (12 pts), >500 ms = 0 pts. Timeouts (>10 s) = 0 pts.

D3 -- URL Yield

20 pts

Total distinct URLs extracted from the sitemap (or all referenced sub-sitemaps for sitemap index files). Yield normalized: 10,000+ distinct URLs = full credit (20 pts). Fewer than 100 distinct URLs on a site with obvious page depth = red flag deduction.

D4 -- Entry Format Correctness

20 pts

Validates each <url> entry against the Sitemap Protocol 0.9 schema. Required: <loc> is an absolute HTTPS URL. Scored optional fields: lastmod in ISO-8601, changefreq using standard tokens, priority in [0.0, 1.0]. Malformed entries deduct 2 pts each, capped at -20 pts.

D5 -- AI-Legibility Signals

25 pts

Five sub-signals, 5 pts each:

  • D5a -- lastmod currency: ≥80% of entries have a lastmod within the past 180 days.
  • D5b -- Compression: sitemap served with Content-Encoding: gzip or .xml.gz variant offered.
  • D5c -- HTTPS canonical: all loc values use https://; none use bare HTTP on an HTTPS domain.
  • D5d -- Content-Type header: server returns application/xml or text/xml.
  • D5e -- robots.txt allowance: sitemap path not Disallowed; at least one named AI bot explicitly allowed or unrestricted.

3. Scoring Rubric

Total possible score: 100 points. Bands:

Band Score Range Interpretation
Optimal85 -- 100Sitemap is a strong AI-indexing asset; negligible delivery friction.
Good65 -- 84Minor gaps. Likely indexed well but missing some AI-legibility signals.
Marginal40 -- 64Measurable delivery friction. Crawlers may skip entries or re-crawl inefficiently.
Poor0 -- 39Sitemap is likely a net-negative for AI indexing.

4. Data Collection Procedure

  1. All measurements taken from a single AWS us-east-1 instance to control for network geography.
  2. User-Agent: Mozilla/5.0 (compatible; GeolusBot/1.0; +https://geolocus.ai/methodology/sitemap-benchmark)
  3. Five cold-fetch trials per domain, each separated by 60 seconds. p50 used for TTFB.
  4. Sitemap XML parsed with a strict schema-validating parser (Python lxml + Sitemap 0.9 XSD).
  5. Scoring applied per section 3. All raw measurements stored in a reproducible artifact file.
  6. Benchmarks are point-in-time. Date of measurement is always disclosed.

5. Reproducibility Script

The following bash script replicates the D1, D2, and D5c/D5d checks for any domain. Requires curl, python3, and xmllint. 

#!/usr/bin/env bash
# sitemap-benchmark.sh -- GEO Sitemap Delivery Benchmark
# Usage: bash sitemap-benchmark.sh <domain>

set -euo pipefail
DOMAIN="${1:?Usage: sitemap-benchmark.sh <domain>}"
UA="Mozilla/5.0 (compatible; GeolusBot/1.0; +https://geolocus.ai/methodology/sitemap-benchmark)"

echo "=== Sitemap Benchmark: $DOMAIN ==="

# D1: Discoverability
ROBOTS=$(curl -sS --max-time 10 -A "$UA" "https://$DOMAIN/robots.txt" || echo "")
SITEMAP_URL=$(echo "$ROBOTS" | grep -i "^Sitemap:" | head -1 | awk '{print $2}')
[ -z "$SITEMAP_URL" ] && SITEMAP_URL="https://$DOMAIN/sitemap.xml"
echo "[D1] Sitemap URL: $SITEMAP_URL"

# D2: TTFB p50 across 5 trials
echo "[D2] Measuring TTFB (5 trials, 60s apart)..."
TIMES=()
for i in 1 2 3 4 5; do
  T=$(curl -sS --max-time 10 -A "$UA" -H "Accept-Encoding: gzip, deflate" \
    -o /dev/null -w "%{time_starttransfer}" "$SITEMAP_URL" 2>/dev/null || echo "999")
  TIMES+=("$T")
  [ "$i" -lt 5 ] && sleep 60
done
P50=$(printf '%s
' "${TIMES[@]}" | sort -n | awk 'NR==3')
echo "[D2] TTFB p50: ${P50}s"

# D5c/D5d checks
HEADERS=$(curl -sS --max-time 10 -A "$UA" -I "$SITEMAP_URL" 2>/dev/null)
CT=$(echo "$HEADERS" | grep -i "content-type:" | head -1)
echo "[D5c] HTTPS loc: $(echo "$SITEMAP_URL" | grep -c '^https://')/1"
echo "[D5d] Content-Type: $CT"

# URL yield (shallow)
XML=$(curl -sS --max-time 30 -A "$UA" -H "Accept-Encoding: gzip, deflate" "$SITEMAP_URL" 2>/dev/null || echo "")
COUNT=$(echo "$XML" | grep -c "<loc>" || echo 0)
echo "[D3] URL yield (shallow): $COUNT entries"

echo "=== Done. ==="

6. Caveats and Limitations

  • TTFB measurements reflect delivery from a single AWS region.
  • Dynamic sitemaps that change between runs are noted as non-deterministic.
  • This benchmark does not assess whether URLs in the sitemap resolve or contain substantive content.
  • Sitemap index files with more than 50,000 sub-sitemaps are capped at 50,000 for yield counting.
  • Scores are not weighted for domain size.

7. See Also