Amazon S3 5GB Limit

Amazon S3 and compatible services used to have a 5GB object (file size) limit. Although Amazon Changed the total object limit in 2010 to 5TB it is still true that the largest object that can be uploaded in a single PUT command is 5 gigabytes.  They  require that the software uploading large files upload it in smaller parts using their Multipart upload API.

Split Large Files into 2GB files on destination

High-Sync fully supports Multipart upload to Amazon S3 so the option discussed below “Split Large FIles” is not necessary when using Amazon S3.  However, some Amazon Compatible Services may not support multipart uploads.  Also some users may have Internet connections that may be slow or unreliable in ways that cause large file uploads (greater than 5GB) to fail intermittently.  To overcome these challenges, High-Sync includes an option to  break files into smaller files during the upload process.

When this option is selected, rather than using Amazon’s multi-part upload, the files are broken into individual files and stored up on the cloud as separate files  with filenames that allow High-Sync to understand the various pieces and reassemble them when they’re brought back down later.  Refer to the nearby Graphic showing the option to Split Large Files.  The default option is 2GB file size.  Adjust this size as desired to create reliable uploads while staying under the file size limit of your cloud provider.

Screenshot from Windows File Explorer showing Original 6.7GB file

To see how this feature works, refer to the screenshot showing a 6.7GB inbox.pst file as shown in Windows file explorer.  After uploading to Wasabi using the Split Large Files option you can see in the Wasabi screen shot the file has been broken into 4 pieces.  Note the first 3 are 2GB each with the remainder in the 4th file.

Notice also that the original filename “inbox.pst” has been renamed on the destination with a code number and part number that will allow High-Sync to automatically identify it as a segmented file and reassemble it back to it’s original file name if it is pulled back down from the cloud.  Obviously, High-Sync must be used to retrieve the file to automatically reassemble.  If you use another tool such as the web interface you will simply end up with the 4 pieces named as shown.

Screenshot from Wasabi browser window showing file has been broken into 4 pieces