Crawlers - Auto-Scrape Websites
Use crawlers to automatically build your chatbot's knowledge base from any website.
What is a Crawler?
A crawler automatically scrapes content from a website and converts it into files your chatbot can use as a knowledge base. The more relevant content the crawler gathers, the smarter your chatbot becomes.
Creating a Crawler
- Go to Dashboard - New Crawler (botbyte.in/dashboard/new/crawler)
- Configure the crawler:
| Field | Description |
|---|---|
| Display Name | A label for the crawler in your dashboard |
| Crawling URL | The starting URL where the crawler begins |
| URL Match | A string pattern the crawler must match in every URL it visits |
| Selector | A CSS selector to extract specific content, e.g. [id='main-content'] |
- Click Create and the crawler will start processing.
How It Works
- The crawler visits the Crawling URL
- It follows links that match the URL Match pattern
- On each page, it extracts content from the element matching your Selector
- Extracted content is saved as files that can be attached to your chatbot
Tips for Better Crawling
- Be specific with URL Match - Use a path prefix like
/docs/to avoid crawling unrelated pages - Use precise selectors - Target the main content area to avoid scraping navigation, footers, and ads
- Test your selector - Open your browser DevTools (F12) and run
document.querySelector("[id='main-content']")to verify it returns the right element - Multiple crawlers - Create separate crawlers for different sections of a large website
After Crawling
Once the crawler finishes:
- Go to your chatbot's settings
- Select the crawled files in the Knowledge Files section
- The chatbot will now use that content to answer questions