Guides

Nightly crawl automation

Use the Developer API to queue a full-site crawl every night, wait for completion, and notify your team when scores drop or critical issues appear.

Overview

A typical nightly workflow looks like this:

1. Resolve the property ID (once) via GET /properties.
2. Queue a scan with POST /properties/:id/scans.
3. Poll GET /jobs/:id until status is completed or failed.
4. Optionally fetch shared reports with reports:read if you publish audit links.

Full endpoint reference: Developer API docs.

Queue & poll

Send an optional body to cap crawl depth for faster nightly runs. Use an Idempotency-Key header so retries do not enqueue duplicate jobs. A 202 response includes data.job.id — poll every 30–60 seconds until the job completes.

curl -s -X POST \
  -H "Authorization: Bearer $XF_API_KEY" \
  -H "Idempotency-Key: nightly-$(date +%Y-%m-%d)-PROPERTY_ID" \
  -H "Content-Type: application/json" \
  -d '{"maxPages": 500, "maxDepth": 5}' \
  https://api.xenonflare.com/api/v1/properties/$XF_PROPERTY_ID/scans

Poll until complete

Most site scans finish within a few minutes depending on size and plan limits. See the Poll job tab above for a bash loop, or register webhooks for job.completed / job.failed.

Cron example

Run at 2:00 AM UTC on a server with curl and jq installed — use the Cron line tab above in your crontab. Store your API key in a secrets manager or environment variable; never commit it to version control.

Alerting on regressions

After the job completes, inspect data.job.result or compare scores in your own database. For stakeholder-facing summaries, create a share link in the web app and list reports via GET /reports with the reports:read scope.

Related

• CI pipeline integration
• Example scan diff
• Scheduled SEO monitoring
• Agency portfolio monitoring
• Developer API reference
• Idempotency guide
• Site audit API
• SEO automation API