Prepare Existing Files for Ingestion

Create metadata sidecar files so that the search service can associate existing Amazon S3 objects with Salesforce files.

Note Complete this task only for files that existed before External Storage Search and its required permissions were enabled. Salesforce automatically creates the required metadata for new files after the feature and permissions are enabled.

Before you begin, make sure that you have:

• A configured Amazon Bedrock knowledge base or Amazon Kendra index.
• An Amazon S3 inventory report or manifest that identifies the existing objects to process.
• Permission to create Lambda functions and Amazon S3 Batch Operations jobs.
• Permission to read the mapping file, source objects, and existing sidecars.
• Permission to write metadata sidecars to the source bucket.

1. Identify the existing S3 objects to process.
2. Generate a mapping file that associates each existing file with its Salesforce ContentDocumentId and Amazon S3 location.
   See Generate a Batch Mapping File for Externally Stored Files.
3. Create a sidecar metadata file for each existing file.
   The metadata associates the file with its Salesforce record. For metadata requirements, see Metadata and filtering for a knowledge base in the AWS documentation.
4. Add the associated Salesforce ContentDocumentId values to each sidecar.
5. Verify that each sidecar follows the Amazon Bedrock or Kendra metadata format, as applicable.
   For more information, see Add metadata to files for a knowledge base in the AWS documentation, and see S3 document metadata in the AWS documentation.
6. Store the metadata sidecars:
   • Amazon Kendra: Store under __sfdcroot__/__metadata__/ in the source S3 bucket.
   • Amazon Bedrock: Store the metadata sidecar alongside the source object.
7. Invoke a Lambda function with Amazon S3 batch events.

   For instructions, see Invoke a Lambda function with Amazon S3 batch events.

   

   Note Amazon Kendra doesn't generate transcripts or summaries from audio, video, or image content. Use the appropriate model to create searchable text content for these files before Amazon Kendra ingests them. For example, Amazon Bedrock Data Automation. See Amazon Bedrock Data Automation.

   The following example illustrates the sidecar update logic:

   1. Load mapping file
   2. For each file in S3:
      a. Look up the salesforce content document Ids from mapping file
      b. Create sidecar metadata json file
      c. Populate the content document Ids in the json file according the correct Bedrock / Kendra format
      d. Make sure the metadata attribute key is "contentDocumentIds" and is an array of 15 digit Id strings

8. Grant the Lambda function access to the source bucket.
9. Create and run an Amazon S3 Batch Operations job that invokes the Lambda function for the existing objects.

• Generate a Batch Mapping File for Externally Stored Files
  Prepare existing files for Amazon Bedrock or Amazon Kendra by generating a mapping file that associates Salesforce files with their Amazon S3 locations. You must complete this task only if you have existing files in Amazon S3 that you'd like to search.