The Crawler is a method of extracting data from websites to bring into Yext. Data is scraped from specified web pages and then ingested via a connector. Crawlers can be configured to run on a one-time or ongoing basis.

This doc covers crawler configuration settings, how to use a crawler as a connector source, and system limits.

Crawler Configuration

Duplicate URLs

If a URL is encountered more than once in a single crawl, only the first instance will be crawled.

Schedule

Inactivity reset. Crawlers set to Daily or Weekly will automatically revert to Once after 14 days of inactivity. A crawler is considered inactive if:

• It is not linked to a connector
• It is linked to a manually-run connector that has not been run in the past 14 days
• It is linked to a manually-run connector whose configuration has not been viewed in the past 14 days

Crawlers reset to Once remain in the platform and can still be viewed and run manually. You can re-add a daily or weekly schedule, but it will reset again if the crawler becomes inactive.

Supported File Types

Note that the crawler can still detect and spider to URLs found in unselected file types without actually crawling them. For example, if you select PDF only but your PDFs are linked from HTML pages, the crawler will spider through the HTML to find the PDFs — but will only crawl the PDFs themselves.

Rate Limit

The maximum number of concurrent tasks the crawler can execute on a site at one time.

Set this based on the maximum number of concurrent requests your site can handle without impacting performance. If too many requests are received, the site will return a 429 error, which will appear in the Status column on the Crawl Details page. If this happens, lower the rate limit. If your site can handle a higher throughput, increase it to speed up the crawl.

Blacklisted URLs

Specify URLs to exclude from crawling, even if they fall within the chosen crawl strategy. You can provide exact URLs or use regex notation to match URL patterns.

To learn more about regex notation, see Mozilla's developer cheat sheet.

Source Type

The crawler supports two source types: Sitemap or Domain.

Sitemap Source Type

Requires a sitemap URL (.xml format). You can also choose whether to check the lastmod tag on URLs in the sitemap.

Domain Source Type

Requires a crawl strategy, one or more start URLs, and optionally a sub-page URL structure.

Pages or Domains to Crawl

The start URL is the first URL crawled and the starting point for all subsequent spidering. You can configure more than one start URL. Start URLs can be as broad or specific as needed — all of the following are valid:

• yext.com
• blog.yext.com
• https://www.yext.com
• https://www.yext.com/blog/2023/05/why-tech-leaders-are-embracing-custom-built-dxp

Including https:// is optional — the crawler resolves the HTTPS protocol automatically.

Crawl Strategy

Determines which pages are crawled from the start URL. The crawler detects pages via URLs in href tags within the HTML (e.g., <a href="www.yext.com">), then spiders to each and repeats, up to 100,000 URLs (see System Limits below).

All Pages example (start URL: www.yext.com/blog):

Sub-Pages example (start URL: www.yext.com/blog):

Sub-Page URL Structure

Allows the crawler to spider to sub-pages with a different URL structure than the start URL. Use this to crawl sub-pages across multiple subdirectories on one domain using a single crawler.

For example, to capture sub-pages under both www.yext.com/blog and www.yext.com/faq but not www.yext.com/products, set the start URL to www.yext.com/blog with the Sub-Pages strategy, and add www.yext.com/faq/* as a sub-page URL structure.

Query Parameter Settings

Designate whether query parameters should be ignored when differentiating between crawled URLs. This prevents the crawler from treating the same page with different query strings as distinct URLs.

All query parameters ignored (start URL: www.yext.com/blog):

Specific parameter language ignored:

No query parameters ignored:

Max Depth (Domain Source Only)

The number of levels past the start URL that the crawler will spider to.

For example, if the start URL is www.yext.com/blog (depth = 0) and Max Depth is set to 1, the crawler will spider to pages directly linked from www.yext.com/blog (e.g., www.yext.com/blog/2023), but will not continue to pages linked from those results.

If a URL appears at multiple depths during a crawl, the system uses the depth of the first instance encountered.

Crawler Status and Performance

Using the Crawler as a Connector Source

The connector pulls data from the most recently completed crawl of the selected crawler.

File Types

Select the types of files to bring in via the connector: HTML, PDF, or both.

This setting does not have to match the file type setting of the crawler itself, but the connector is limited to only the file types that were actually crawled.

Example: If a crawler is configured to crawl both HTML and PDF files, a connector using that crawler can be set to ingest only HTML files. However, if the crawler was set to scrape only HTML files, a connector configured to scrape only PDFs would return no data.

URLs

Page Type

PDF Support

If the connector's File Type setting includes PDFs (i.e., PDF Only or PDF and HTML), the Page Type must be set to Detail Page.

When data is extracted from a PDF, each PDF is treated as a single entity. The data is read as field metadata, and the entire body of the PDF is ingested as unstructured content.

HTML-specific selectors (CSS or XPath) are not compatible with PDFs. If a connector is configured to support both HTML and PDF file types, these selectors can still be added — but the corresponding fields will be blank for any PDFs.

Selectors

If both HTML and PDF file types are selected, all HTML-specific selectors will be available but will return blank values for PDFs. The Author and Created Date selectors will not be available in this combined mode.

System Limits