If you want to store multiple copies of objects in your S3 buckets in different regions, S3 can be configured to automatically replicate objects from a source bucket to replica buckets across the world, increasing performance and access latency. of your applications.
It should be noted that if the objects you are replicating are publicly available on the Internet, such as a media bucket for downloaded content, you can simply use AWS CloudFront CDN to cache content in edge locations everywhere.
How does replication work?
Replication basically runs every time you download or update an object in a bucket. You create a replication rule, which defines the source and destination bucket. Objects uploaded to the source bucket will be uploaded to the destination bucket. For replication to work, you must enable versioning. This will slightly increase the storage costs, depending on how much you overwrite the data.
There are some limitations. AWS does not replicate deleted objects, so you will need to delete each copy manually. And, while replication usually occurs fairly quickly, it can take up to 15 minutes or more if you are replicating large objects.
When an object is replicated, a metadata tag is defined in the source object. All replicas will have the REPLICA tag, and the source object will be tagged COMPLETED. This means that you can have two-way replication – two compartments with replication rules pointing to each other. S3 will not replicate objects imported with the replica tag, otherwise you will be stuck in an infinite replication loop.
This is called multi-master replication, but there is no system in place to ensure that objects downloaded later are replicated later. In this configuration, it is possible for data to get out of sync if you update two objects simultaneously, and it is possible to mine stale data when synchronizing from one master to another.
Replication configuration
Of S3 management console, navigate to the source bucket. You will need to enable versioning under “Properties”:
Once this option is enabled, you can add a new replication rule under Management> Replication> Add rule.
By default it will replicate all objects, but you can set a filter if you want.
Under Destination, select the destination bucket. You can also configure the storage class for replica objects, as well as set the owner if you are transferring between accounts. There is also an option for “S3 Replication Time Control”, which ensures that objects will be replicated within 15 minutes, backed up by a service level agreement.
On the next screen, you will need to select or create an IAM role for replication. Permissions are handled automatically, just enter a name.
Click “Next”, then click “Save.” The rule must be active immediately; you can test uploading an object, and you should see it replicated in the destination bucket, then you will see the replication status tag change to COMPLETED.