Skip to main content
POST
/
api
/
aibot
/
scrape
Crawl websites to discover internal links
curl --request POST \
  --url https://app.livechatai.com/api/aibot/scrape \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "url": "https://.....",
  "sitemapUrl": "https://.....sitemap.xml",
  "streaming": false,
  "limit": 100
}
'

Authorizations

​
Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json

Scraping configuration

​
url
string

The URL of the website to crawl and fetch internal links from

Example:

"https://....."

​
sitemapUrl
string

The sitemap URL to crawl and fetch internal links from

Example:

"https://.....sitemap.xml"

​
streaming
boolean

Enable streaming of the scraping results

Example:

false

​
limit
number

Maximum number of links to scrape

Example:

100

Response

Successfully scraped website and retrieved internal links