Fastly crawler logs

Create a Fastly HTTPS logging endpoint or log shipper that formats crawler requests as JSON.

Endpoint

POST /api/events/fastly

curl -X POST "https://your-watchllm-host.com/api/events/fastly" \
  -H "Authorization: Bearer $WATCHLLM_API_KEY" \
  -H "content-type: application/json" \
  --data '[{"time":"2026-05-01T12:00:00Z","status":200,"reqMethod":"GET","url":"/","responseContentType":"text/html","requestHost":"example.com","ua":"Mozilla/5.0 Google-Extended"}]'

Fastly fields

Include request time, status, method, URL/path, host, response content type, client IP, referrer, and user agent. WatchLLM classifies known crawler user agents before inserting rows.

Verify

Open /workspace/settings/crawler-logs?provider=fastly, then use /workspace/crawlers to confirm accepted rows and top pages.