Guides

Nightly crawl automation

Use the Developer API to queue a full-site crawl every night, wait for completion, and notify your team when scores drop or critical issues appear.

Overview

A typical nightly workflow looks like this:

1. Resolve the property ID (once) via GET /properties.
2. Queue a scan with POST /properties/:id/scans.
3. Poll GET /jobs/:id until status is completed or failed.
4. Optionally fetch shared reports with reports:read if you publish audit links.

Full endpoint reference: Developer API docs.

Queue a scan

Send an optional body to cap crawl depth for faster nightly runs. Use an Idempotency-Key header so retries do not enqueue duplicate jobs.

curl -X POST \
  -H "Authorization: Bearer xf_live_xxxxxxxx" \
  -H "Idempotency-Key: nightly-$(date +%Y-%m-%d)-PROPERTY_ID" \
  -H "Content-Type: application/json" \
  -d '{"maxPages": 500, "maxDepth": 6}' \
  https://api.xenonflare.com/api/v1/properties/PROPERTY_ID/scans

Poll until complete

A 202 response includes data.job.id. Poll every 30–60 seconds; most site scans finish within a few minutes depending on size and plan limits.

JOB_ID="507f1f77bcf86cd799439014"

until true; do
  STATUS=$(curl -s \
    -H "Authorization: Bearer xf_live_xxxxxxxx" \
    https://api.xenonflare.com/api/v1/jobs/$JOB_ID | jq -r '.data.job.status')

  echo "Job status: $STATUS"

  case "$STATUS" in
    completed|failed|cancelled) break ;;
    *) sleep 45 ;;
  esac
done

Cron example

Run at 2:00 AM UTC on a server with curl and jq installed:

0 2 * * * /opt/xenonflare/nightly-crawl.sh >> /var/log/xenonflare-crawl.log 2>&1

Store your API key in a secrets manager or environment variable — never commit it to version control.

Alerting on regressions

After the job completes, inspect data.job.result or compare scores in your own database. For stakeholder-facing summaries, create a share link in the web app and list reports via GET /reports with the reports:read scope.

Related

• CI pipeline integration
• Scheduled SEO monitoring
• Agency portfolio monitoring
• Developer API reference
• Idempotency guide
• Site audit API
• SEO automation API