When using the MuleSoft Amazon S3 Connector's "On New Object" source operation, new objects may be returned across multiple polling cycles rather than all at once.
When 11 files are uploaded to an S3 bucket within 1 second with a 10-second polling interval configured in MuleSoft, objects may be returned as follows:
---
13:06:17 --- 3 objects
13:06:27 --- 2 objects
13:06:37 --- 2 objects
13:06:47 --- 2 objects
13:06:57 --- 1 object
13:07:07 --- 1 object
---
As shown above, all 11 objects are not returned in the first polling cycle.
The MuleSoft Amazon S3 Connector's "On New Object" source operation relies on AWS SQS (Simple Queue Service) to receive S3 event notifications. Due to its distributed architecture, AWS SQS does not guarantee that all available messages are returned in a single retrieval. This is a known characteristic of AWS SQS and does not result in message loss — all events are eventually delivered and processed (see [1-2]).
This behavior is currently by design.
Mitigation is to adjust the poll interval (see [3]) in your MuleSoft application to check the queue more frequently.
[1] https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html
[2] https://repost.aws/questions/QUSFyO7uCNQ3a2FLyepHoJqg/some-messages-are-being-polled-others-are-missed-in-aws-sqs
[3] On New Object - Scheduling Strategy
If retrieving all events in a single polling cycle is a critical requirement, please submit an Idea Exchange (https://ideas.salesforce.com/s/) request asking for the MuleSoft S3 Connector to support returning all messages in one polling cycle.
005386545

We use three kinds of cookies on our websites: required, functional, and advertising. You can choose whether functional and advertising cookies apply. Click on the different cookie categories to find out more about each category and to change the default settings.
Privacy Statement
Required cookies are necessary for basic website functionality. Some examples include: session cookies needed to transmit the website, authentication cookies, and security cookies.
Functional cookies enhance functions, performance, and services on the website. Some examples include: cookies used to analyze site traffic, cookies used for market research, and cookies used to display advertising that is not directed to a particular individual.
Advertising cookies track activity across websites in order to understand a viewer’s interests, and direct them specific marketing. Some examples include: cookies used for remarketing, or interest-based advertising.