Engagement SFTP Performance & Reliability Guidelines

When building applications that send and receive data from MCE Enhanced Secure FTP Service, follow these best practices to optimize performance and reliability.

Guidelines for Measuring Performance

For applications with throughput requirements, it is essential to measure throughput, latency, and other critical aspects of the system. Many factors can affect the observed performance including the client Internet connection speed, the network over which the data must travel, and the destination FTP server performance. Following the best practices in this guide can help you achieve the best possible throughput. In addition to these recommendations specific to the Enhanced Secure FTP Service, Marketing Cloud also supports integration with AWS S3, Azure Blob Storage, and GCP Cloud Storage.

The protocols used for these services are more efficient and can often yield superior performance to the Enhanced Secure FTP Service. However, FTP is a widely supported protocol that allows integration with many systems. A decision must be uniquely made for each system with respect to using the Enhanced Secure FTP Service versus the other cloud-native options, and ideally that decision must be made based on real measurement of system performance.

TCP/IP Settings

Optimizing FTP performance can involve tuning several TCP/IP settings.   Below we have listed a number of settings that have been broadly seen to influence performance. 

1. TCP Window Size: Increasing the TCP window size can help improve throughput, especially in high-latency or high-bandwidth networks. This parameter controls the amount of data that can be sent before  waiting for an acknowledgement from the receiver. A larger window allows more data in transit at any given time.
2. Maximum Transmission Unit (MTU) Size:  The MTU size defines the largest chunk of data that can be sent individually. A larger MTU can reduce fragmentation and improve performance.
3. TCP Fast Open: The TCP Fast Open option allows data to be sent with the initial SYN packet of a TCP connection. This can allow data to start flowing earlier in the connection, which is especially helpful for smaller file transfers.
4. Firewall and Security Settings: Ensure that firewalls and other network devices are configured to allow FTP connections with trusted hosts without unnecessary inspection or delays.

Compression

Compression can be used to reduce the number of bytes that must be transferred to represent the entire dataset.   If this is not being used, this is something that can be done to have a significant and immediate impact on the amount of time that it takes to transfer a file from location to location. The Enhanced Secure FTP service supports compression in the SFTP connection for SFTP clients that have enabled compression. Alternatively, files can be compressed prior to transfer for clients that can’t support compression in the SFTP connection.

Expectations Around File Size

In general, it is preferable to prefer upload a smaller number of larger files than uploading many small files.   Fewer sessions uploading larger files will experience less overhead of initializing connections, and the SFTP compression is more efficient for larger files. If many small files are required, consider combining the files into a larger *.ZIP file that can be uncompressed using File Transfer Activity before usage.

SFTP Client 

The SFTP client being used can have a considerable impact on performance. Keep the SFTP client up to date to ensure the best performance. Updates to SFTP clients commonly include optimized algorithms, bug fixes, and enhanced protocol support that can increase the reliability and performance of transfers.

Many SFTP clients offer features to throttle transfers to a particular amount of bandwidth. To optimize the performance and reliability of SFTP transfers, ensure that these limits are disabled.

Data Retention

The performance and reliability of SFTP operations can be impacted by the number of files and directories contained within the same parent directory. A high number of files and directories increases the latency of listing operations, which can result in errors and difficulty navigating directories. Always maintain the minimum set of data necessary on the Enhanced Secure FTP Service and consider using subdirectories to partition files when a large number of files is required. The Enhanced Secure FTP Service will automatically clean up files older than 21 days to help maintain high performance of the system.

Network Access

In general, any source is able to connect to the Enhanced Secure FTP Service. An IP address can be temporarily banned due to a rapid succession of failed login attempts. Additionally, some IP  addresses can be permanently banned due to a longer history of malicious failed login attempts.

The FTP Accounts section of setup includes options to configure an IP allowlist. In cases where a particular IP address is unable to connect to the Enhanced Secure FTP Service, the issue is most often that an allowlist is being enforced for the account, and the source IP address is not included in the allowlist.

If an SFTP client is unable to connect to the Enhanced Secure FTP Service, please contact MCE Support for assistance to confirm that the IP can connect and to clear any possible ban.