Scout · Endpoints

DNS, robots & crawl

Network facts and multi-page link discovery. Crawl returns a link graph only — Scout does not archive page HTML.

DNS

POST/v1/dnsAPI key

Lookup records for a hostname. 1 credit.

Name	Type	Required	Description
`hostname`	`string`	required	Hostname to resolve (no scheme).
`types`	`string[]`	optional	Subset of A, AAAA, TXT, CNAME, MX, NS, SOA. Default: common set.

bash

curl -X POST https://scout.macrocontent.dev/v1/dns \
  -H "Authorization: Bearer sk_scout_…" \
  -H "Content-Type: application/json" \
  -d '{"hostname":"example.com"}'

json

{
  "hostname": "example.com",
  "types": ["A", "AAAA", "TXT", "MX"]
}

Example response (truncated):

json

{
  "status": "success",
  "hostname": "example.com",
  "records": {
    "A": ["93.184.216.34"],
    "AAAA": ["2606:2800:220:1:248:1893:25c8:1946"],
    "TXT": ["v=spf1 -all"],
    "MX": []
  },
  "timestamp": "…"
}

robots.txt

POST/v1/robotsAPI key

Whether a URL is allowed for a user-agent, plus sitemap_urls. 1 credit.

Name	Type	Required	Description
`url`	`string`	required	URL to check against robots.txt.
`user_agent`	`string`	optional	UA token for robots matching.

json

{
  "url": "https://example.com/pricing",
  "user_agent": "MacroScout"
}

Example response:

json

{
  "status": "success",
  "url": "https://example.com/pricing",
  "allowed": true,
  "user_agent": "MacroScout",
  "sitemap_urls": ["https://example.com/sitemap.xml"],
  "timestamp": "…"
}

Crawl (link discovery)

POST/v1/crawlAPI key

Depth/page-limited crawl returning pages[], discovered_urls[], truncated. Credits: 15 + 2×max_pages (charged up-front).

Name	Type	Required	Description
`start_url`	`string`	required	Seed URL.
`max_depth`	`number`	optional	Link depth from seed.
`max_pages`	`number`	optional	Hard cap (default 25, max 100). Drives credit cost.
`same_origin`	`boolean`	optional	Stay on start origin (default true).
`allow_hosts`	`string[]`	optional	Extra hosts when same_origin is false.
`seed_from_sitemap`	`boolean`	optional	Seed queue from sitemap URLs in robots.
`respect_robots`	`boolean`	optional	Honor robots Disallow (default true).

json

{
  "start_url": "https://example.com",
  "max_depth": 2,
  "max_pages": 25,
  "same_origin": true,
  "seed_from_sitemap": true,
  "respect_robots": true
}

Example response (truncated):

json

{
  "status": "success",
  "start_url": "https://example.com",
  "pages": [
    {
      "url": "https://example.com/",
      "depth": 0,
      "status": "ok",
      "http_status": 200,
      "final_url": "https://example.com/",
      "links_found": 12,
      "error": null,
      "robots_allowed": true
    }
  ],
  "discovered_urls": [
    "https://example.com/",
    "https://example.com/about"
  ],
  "truncated": false,
  "response_time_ms": 3200
}

Response size

max_pages is capped at 100 (default 25). Each page entry is small (URL + status + link count — no HTML), but discovered_urls[] can still reach hundreds of strings. Prefer tighter max_pages / max_depth in interactive calls; for large batches, crawl once then feed URLs into jobs (job results expire after ~30 minutes — not an archive).

No content archive

Crawl stores link / status facts only. Feed discovered URLs into jobs or call extract/inspect per URL. Full walkthrough: First verification recipe.

Typical follow-up:

json

{
  "requests": [
    { "type": "inspect", "body": { "url": "https://example.com/about", "include": ["meta", "headings"] } },
    { "type": "extract", "body": { "url": "https://example.com/pricing", "fields": [{ "name": "h1", "selector": "h1", "type": "text" }] } }
  ]
}