High Credit Consumption with Web Content (Crawler) Connector in Data 360

This article provides workarounds to reduce credit consumption by the Web Content (Crawler) connector. By adjusting data stream settings or changing the connection method based on your site's update frequency and structure, you can effectively manage processing frequency and the number of ingested pages.

Consider the following methods to reduce credit consumption:

1. Change the schedule frequency

Modify the execution frequency of the schedule to lower the processing frequency of the search index. This is effective when the website is updated infrequently.
・The default frequency is every 24 hours.
・You can change the setting to weekly or monthly depending on your operational needs.

2. Lower the crawl depth 

When using the Web Content (Crawler), setting a lower crawl depth reduces the number of ingested pages, thereby cutting down consumption.

・A deeper crawl depth increases the number of pages ingested. 
・Lowering the depth reduces data volume, so it should be set considering the balance with answer accuracy.

3. Filter included page elements or target URLs 

Adjust the settings for included page elements or included web pages to reduce unnecessary data processing and crawling.

・Specifying [Included Page Elements] : Specify a jQuery selector to limit the HTML elements to be ingested. By default, the entire page is ingested. Narrowing it down to specific elements reduces unnecessary data processing and lowers consumption.

・Specifying [Included Web Pages] : Use a regular expression to limit the target URLs. If left unconfigured (blank), all pages under the domain are crawled. Restricting the crawl to a specific path (e.g., .*/content/.*) helps control consumption.

4. Turn off image ingestion 

Ingesting and processing image data along with text information increases the volume of data processed, leading to higher credit consumption. Turning off image ingestion can help lower credit consumption.

5. Switch to Web Content (Sitemap) integration

If a sitemap is available for the website, using the sitemap integration is recommended. Since the URLs to be retrieved are predetermined, it prevents crawling unnecessary pages and effectively limits the processing volume.

Important Notes

• You cannot edit an existing Web Content (Crawler) data stream configuration to change it to a sitemap integration.

• To use the sitemap integration, you must configure a new connection using the Web Content (Sitemap) connector and create a new data stream.

Steps to Change Settings

1. Navigate to [App Launcher] | [Data Cloud].

2. Click the Data Streams tab.

3. Locate the data stream row for the target Web Content connection.

4. Click [Edit Data Stream] from the ▼ icon in the upper right.

5. Modify the necessary settings, such as Crawl Depth, Included Page Elements, Included Web Pages, Included File Types, or Schedule.

6. Save the changes.

For more details, please refer to the following:

• Create an Unstructured Data Connection from the Web Content Crawler

• Web Content (Sitemap) Connector