S3 Replication with Example
S3 Replication is a feature in Amazon S3 that automatically copies objects and their metadata from one bucket (the source) to another bucket (the destination). Replication helps with data backup, disaster recovery, compliance, and reducing latency for users in different regions.
Types of S3 Replication
- Cross-Region Replication (CRR): Replicates objects between buckets in different AWS regions.
- Same-Region Replication (SRR): Replicates objects between buckets within the same AWS region.
How S3 Replication Works
- Replication is configured at the bucket level using a replication rule.
- You can specify which objects to replicate (all objects or those with specific prefixes/tags).
- Both source and destination buckets must have versioning enabled.
- Replication can copy new objects, object updates, and (optionally) delete markers.
Example Scenario
Suppose you have two buckets:
- Source bucket: source-bucket (in the US East region)
- Destination bucket: destination-bucket (in the EU West region)
You want to replicate all objects from source-bucket to destination-bucket for disaster recovery.
Steps to Set Up Cross-Region Replication
Enable Versioning on both buckets.
Set Up IAM Role
Create an IAM role that allows S3 to replicate objects on your behalf.Create a Replication Rule
In the S3 console or using the AWS CLI, create a rule to replicate all objects.
Example: S3 Replication Using AWS CLI
# Enable versioning on both buckets
aws s3api put-bucket-versioning --bucket source-bucket --versioning-configuration Status=Enabled
aws s3api put-bucket-versioning --bucket destination-bucket --versioning-configuration Status=Enabled
# Create a replication configuration JSON file (replication.json)
# Example content:
# {
# "Role": "arn:aws:iam::123456789012:role/s3-replication-role",
# "Rules": [
# {
# "Status": "Enabled",
# "Priority": 1,
# "DeleteMarkerReplication": { "Status": "Disabled" },
# "Filter": {},
# "Destination": {
# "Bucket": "arn:aws:s3:::destination-bucket"
# }
# }
# ]
# }
# Apply the replication configuration
aws s3api put-bucket-replication --bucket source-bucket --replication-configuration file://replication.json
# Upload an object to the source bucket
aws s3 cp example.txt s3://source-bucket/example.txt
# The object will be automatically replicated to the destination bucket
What Happens Next
- When you upload
example.txttosource-bucket, S3 automatically copies it todestination-bucket. - The replicated object retains its metadata and version ID (if versioning is enabled).
- You can verify replication by listing objects in the destination bucket.
Benefits
- Disaster Recovery: Data is available in another region if the primary region fails.
- Compliance: Meet regulatory requirements for data redundancy and geographic distribution.
- Low Latency: Serve users from the region closest to them by replicating data.
Considerations
- Replication is not retroactive: Only objects uploaded or modified after the replication rule is enabled are replicated.
- Storage Costs: You pay for storage in both source and destination buckets, and for inter-region data transfer in CRR.
