Launch and follow a crawl

Pick or create a campaign

The campaigns of a project:

curl "https://api.nessflow.com/v1/projects/$PROJECT_ID/campaigns" \
  -H "Authorization: Bearer $NESSFLOW_TOKEN"

For a weekly audit of 500 pages with JavaScript rendering:

curl -X POST "https://api.nessflow.com/v1/projects/$PROJECT_ID/campaigns" \
  -H "Authorization: Bearer $NESSFLOW_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "Weekly audit", "start_url": "https://exemple.fr/", "max_depth": 5, "max_urls": 500, "javascript_rendering": true, "schedule": {"frequency": "weekly", "day": "monday", "time": "06:00"}}'

name, start_url, max_depth and max_urls are required. An unknown field is refused (422), never ignored: a typo does not go through silently.

Launch the crawl

Launching a crawl consumes one crawl of your plan's monthly quota. The Idempotency-Key header is therefore required: if the response is lost and you retry with the same key, the API returns the first response instead of launching a second crawl.

curl -X POST "https://api.nessflow.com/v1/campaigns/$CAMPAIGN_ID/crawls" \
  -H "Authorization: Bearer $NESSFLOW_TOKEN" \
  -H "Idempotency-Key: $(uuidgen)"

One crawl runs at a time per team. If a crawl is already running, the API answers 409 crawl.already_running, with active_crawl_id.

Follow the progress

curl "https://api.nessflow.com/v1/crawls/$CRAWL_ID" \
  -H "Authorization: Bearer $NESSFLOW_TOKEN"

Three fields are enough:

• active: true while the crawl runs. The server decides it, including for a crawl left without a worker;
• status: queued, running, then completed, failed or stopped;
• stage_plan.settled: true when every report is computed.

status: completed arrives before post-processing ends: wait for settled before reading reports. Rather than polling, subscribe a webhook to crawl.completed.

Stop a crawl

Stopping is synchronous, and never refused for billing reasons:

curl -X POST "https://api.nessflow.com/v1/crawls/$CRAWL_ID/stop" \
  -H "Authorization: Bearer $NESSFLOW_TOKEN"