Skip to main content
POST
Crawl websites to discover internal links

Authorizations

Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json

Scraping configuration

url
string

The URL of the website to crawl and fetch internal links from

Example:

"https://....."

sitemapUrl
string

The sitemap URL to crawl and fetch internal links from

Example:

"https://.....sitemap.xml"

streaming
boolean

Enable streaming of the scraping results

Example:

false

limit
number

Maximum number of links to scrape

Example:

100

Response

Successfully scraped website and retrieved internal links