Crawler logs getting started

Crawler logs show when AI systems crawl your site, which pages they request, which bot family they belong to, and whether the request succeeded.

Choose a provider

WatchLLM accepts native crawler log payloads from Akamai, Cloudflare Logpush, Cloudflare Workers, AWS CloudFront, Fastly, Google Cloud CDN, Netlify, Vercel, and Manual setup.

Authentication

Create a project-scoped API key in /workspace/api. Use the full key once in your CDN, edge function, or log shipper.

curl -X POST "https://your-watchllm-host.com/api/events" \
  -H "Authorization: Bearer $WATCHLLM_API_KEY" \
  -H "content-type: application/json" \
  --data '[{"timestamp":"2026-05-01T12:00:00Z","status_code":200,"request_method":"GET","request_path":"/","content_type":"text/html","hostname":"example.com","user_agent":"Mozilla/5.0 GPTBot/1.3"}]'

What is stored

WatchLLM stores normalized timestamp, status, method, host, path, query string, content type, user agent, referrer, country, and classified crawler key. Non-HTML assets and non-AI traffic are skipped to keep the table focused.

Verify the connection

Use /workspace/settings/crawler-logs to send a local test event and inspect provider-specific setup cards. Accepted rows appear in /workspace/crawlers and the REST crawler analytics endpoints.