Prompt Template Batch Processing

Overview

Batch processing generates large numbers of prompt responses asynchronously in a single request. This approach helps you avoid hitting LLM rate limits and efficiently handles high-volume content generation.

The system queues your batch job, processes all items, and stores the results. You can monitor the job's progress and retrieve completed responses when ready. The actual completion time depends on the model provider and can vary.

Common use cases include:

• Creating personalized email campaigns for thousands of customers.
• Generating product descriptions from catalog data.
• Summarizing large volumes of case notes or customer feedback.
• Translating content across multiple records.
• Triggering bulk content generation on a schedule or by record changes.

Implementation Options

Salesforce provides two primary methods for implementing batch processing:

Flow

This approach works for most use cases and is best for Salesforce admins building no-code solutions. Use the Prompt Template (Batch Generation) action in a Schedule-Triggered or Record-Triggered Flow to create batch jobs that run automatically. These Flows include Completed and Timed out paths, allowing you to handle successful completion and timeout scenarios differently.

Limits:

• Flow batch jobs typically begin processing within two hours of submission.
• Up to 10,000 generations per day for standard models.
• If using models that support native batch processing, up to 50,000 generations per day.
• Up to 1,000 items per hour when using WebSearch Retrievers.

Apex

This approach offers maximum flexibility for custom integration and automation. Use the AiJobRun and AiJobRunItem objects to create and manage batch jobs programmatically. Apex is recommended when you want to:

• Build a custom UI. For example: a button that starts a job, or a dashboard listing batch jobs.
• Control batch size dynamically (accumulate items until a count or time threshold, then start).
• Trigger from external events that aren't record-triggered or schedule-based.

Limits:

• Up to 1,000 AiJobRunItems for each AiJobRun.
• If using models that support native batch processing, there’s a limit of 10,000 AiJobRunItems for each AiJobRun.
• Up to 1,000 items per hour when using WebSearch Retrievers.
• Up to 5 AiJobRun objects in 24 hours. If your requests exceed the daily limit, they’re placed in a queued status until the limit resets.

Best Practices

Select the right model:

• Use a model labeled Batch Supported: Yes in Prompt Builder for fastest processing.
• Avoid mini models for batch as they're notably slower.

Size your batches:

• Bigger is better. Number of jobs is the limiting factor, not items in those jobs. Batches of 200 and 2,000 finish in roughly the same amount of time.
• One or more failed items don't block the rest of the batch.

Select the right implementation: Flow vs. Apex:

• Use Flow unless you need a custom UI to start jobs, dynamic batch sizing, or non-record and non-schedule triggers.

Design your Flow:

• Only perform one batch action per Flow. Chained batch actions queue up and significantly increase runtime.
• Combine prompts into a single template when possible.
• Set the timeout to 1 day.

Monitor job progress:

• Query AiJobRun and AiJobRunItem for status, item counts, errors, and responses.
• Subscribe to AiJobRunStatusEvent for event-driven workflows.

Limitations and Considerations

Prerequisites

• The specified prompt template must have an active version.
• For Flow: A Default Workflow User must be configured in Process Automation Settings.

Processing Behavior

• Jobs process in chronological order based on creation date.
• Jobs created within seconds of each other may not follow a strict sequence.
• After processing begins, input values and certain fields become immutable.
• Job status changes are system-managed and can’t be manually altered.
• Completion time depends on the model provider and can vary.

Platform Considerations

• Batch processing is asynchronous and results aren't immediate.
• Set appropriate timeout lengths. One day is recommended for Flow batch jobs.
• Monitor job status through platform events or by querying job records.
• Plan for processing time when designing time-sensitive workflows.
• Consider fallback handling for failed items within a batch.