AMAZON SIMPLE STORAGE SERVICE REPLICATION

AMAZON SIMPLE STORAGE SERVICE REPLICATION

Abstract - Customers often have operational requirements or company policies that require additional copies of their existing or new Amazon S3 items for Disaster Recovery or to reduce latency. Amazon S3 Replication is used extensively to replicate newly uploaded objects between S3 buckets. With the help of Amazon S3 Batch Replication, we can replicate existing objects which were created or uploaded before enabling S3 replication. Whereas, Amazon S3 Same Region Replication and Amazon S3 Cross Region Replication will automatically replicate newly created objects in the bucket.

Key Words: S3 Replication, S3 Same Region Replication (SRR), S3 Cross Region Replication(CRR),S3 Batch Replication,S3ReplicationTimeControl(RTC)

1. INTRODUCTION

Replication is a process of automatically copying objects between buckets in the same or different AWS Regions. This copying happens when you create new objects or updateexistingones.

Amazon Simple Storage Service (S3) replication allows you to have an exact copy of your objects stored in other buckets. This is a very flexible and cost-effective way to store your data in the cloud. It can be used to help you shareyourdataacrossdifferentaccounts,ortostorelarge amounts of data where you need it to be available anywhere.

1.1 PURPOSE OF S3 REPLICATION

 ReduceLatency

 Enhanceavailability

 DisasterRecovery

 ObjectReplicationwithin15minutes

 StoreobjectsinMultipleRegion

 CopyObjectstocosteffectivestorageclass

 Dataredundancy

 Meetcompliancerequirements

 CopyingObjectswhilepreservingthemetadata

 SustainobjectcopiesundervariousOwners

2. REPLICATION OPTIONS:

2.1 AMAZON S3 SAME REGION REPLICATION

S3 Same Region Replication(SRR) will automatically replicate objects from a source bucket to destination bucketwithinsameAZordifferentAZinsameregion.We caneasilyaggregatethelogsfromdifferentS3bucketsfor intra-region processing or for configuration of live replication between developer and test environment. It usesasynchronous replication,thatmeansobjectsarenot copied to destination bucket as soon as it is created or modified.

Fig -1: S3SameRegionReplication(SRR)

We can configure SRR using AWS Console, AWS CLI and AWSSDKs.

Replicationcan bedoneat bucket-level,for certain prefix, for specific tags or object level within same account or differentaccountinsameregion.

It allow us to change account ownership to protect replicatedobjectsfromaccidentaldeletion

2.2 AMAZON

S3 CROSS REGION REPLICATION

S3 Cross Region Replication(CRR) will automatically replicateobjectsordatafromsourcebuckettodestination bucket in different region. It minimizes latency for data access in different geographic regions. Similar to SRR, we can configure CRR using AWS Console, AWS CLI and AWS SDKs.

Replicationcan bedoneat bucket-level,for certain prefix, for specific tags or object level within same account or differentaccountindifferentregion.

It allow us to change account ownership to protect replicatedobjectsfromaccidentaldeletion.

WecanconfigureS3CRRtocopyobjectsintooneormore bucketsinadifferentregiontoimproveresilience Ithelps us in maintaining compliance, disaster recovery and securityenforcement.

2.3

With CRR and SRR, Amazon S3 protects your data by automatically replicating new objects that you upload. S3 Batch Replication, on the other hand, lets you replicate existing objects using S3 Batch Operations, which are managedjobs.

Weneedto payS3chargesforstoringdata intheselected storage class in the primary copy, for PUT requests to replicatedata acrossAWSRegions,and forretrieving data from storage in the selected storage class. If something already exists in the destination bucket, we will check to see if the data is in sync. If it's not in sync, we have to replicate the data using a PUT request, but you won't pay theinter-RegionDataTransferOUTcharge.Ifthedataisin sync, Batch Replication won't do anything and we will incurnocharge.

Ifwewanttodoa batchreplicationjob,wecanchoose to haveitonlyreplicateobjectsthatwerecreatedrecentlyor thatareinacertainstatus.

 Pending:Theobjectispendingreplication.

 InProgress:Theobjectisbeingreplicated.

 Available:Theobjectisavailableforreplication.

 Error: An error occurred while replicating the object.

3. STEPS TO SETUP S3 REPLICATION:

1. NavigatetoS3console.

2. Createtwobucketsbyenablingbucketversioning -Source bucket in Region A and Destination bucket in Region A(S3 SRR) or in Region B (S3 CRR).

3. In source bucket, in Management tab, Click on createreplicationrule.

Fig -5: Replicationrulecreation

4. Inreplicationruleconfiguration,enterReplication rule name, choose Status of replication rule as Enable/Disable after creating bucket, set Priority value.InSourcebucket,choosescoperuleasLimit the scope of this rule using one or more filters (if wewanttolimittospecifictagorprefixorobject) with Filter type as specific prefix or Apply to all objects in the bucket (if wewanttoallobjectsto bereplicated).

5. In Destination, Select Choose a bucket in this account if destination bucket exist in same account or Specify a bucket in another account if destinationbucketispresentinotheraccountand also enable check box if you want to change ownershiptodestinationbucketowner.

8. We can also enable Additional replication options like Replication Time Control(RTC) to replicate new objects within 15 minutes, Replication metricsandnotificationstogetmetricstomonitor replication rule, Delete marker replication (delete markers are replicated on destination bucket object) and Replica modification sync to replicate changes made in source bucket to destination bucket.

9. AfterweclickonSave.Wewillgettwooptionsas shownasbelowfigure.

6. ForIAMrole,ifyoualreadycreatedanyrole,then select Choose from existing IAM roles or select EnterIAMroleandchooseCreatenewrole.

7. To replicate encrypted, we can enable Replicate objects encrypted with AWS KMS and provide KMS Key details. To change storage class at destination, enable Change the storage class for the replicated objects and choose the storage class.

a. We can NO, if we don’t want to replicate existingobjectsinsourcebucket andClickon Submit.

b. Select Yes, if you want to replicate existing objects and Submit. It will redirect to Batch OperationJobCreationpage.

Fig -10: BatchJobConfiguration

c. ChooseJobrunoptions.

d. If we want to completion report for all tasks or for failed tasks, Enable Generate completionreportandprovidepathtoimport report.

4. RESULTS

a. Replication

Fig -11: ReportGenerationandIamRoleConfiguration

e. We can select existing permission for batch operations or create new role and click on Save.

Fig -13: SourceBucket

Fig -14: DestinationBucketafterReplication

b. Batch Replication

Fig -12: BatchOperation

f. Batch operation will be created. After batch operation is completed, a folder will be created for completion report in provided destinationbucket.

Fig -15: BatchJob

Fig -16: SourceBucketbeforeReplication

Fig -17: DestinationBucketafterBatchReplication

Fig -18: CompletionReport

ThemainobjectiveofthisS3Replicationistocopyobjects ordata to same region(sameaccount or differentregion) for intra-testing, log aggregation or to different region (same account or different region) for disaster recovery, compliance requirement. We have implemented S3 batch replication to replicate for existing object which was createdbefore enabling replicationasS3 SRR and S3CRR will only replicate newly uploaded or created object after creatingreplicationrule.

REFERENCES

[1] https://docs.aws.amazon.com/AmazonS3/latest/user guide/replication.html

[2] https://docs.aws.amazon.com/AmazonS3/latest/user guide/replication.html

[3] https://www.geeksforgeeks.org/amazon-s3-crossregion-replication/

[4] https://medium.com/@maxy_ermayank/aws-s3bucket-replication-56dcfec2f424

[5] https://aws.plainenglish.io/what-is-amazon-s3replication-fbb9c45507b3

[6] https://medium.com/cloud-techies/s3-same-regionreplication-srr-and-cross-region-replication-crr34d446806bab

[7] https://aws.amazon.com/s3/pricing/

[8] https://aws.plainenglish.io/aws-s3-replication60c4663efc2f

[9] https://www.workfall.com/learning/blog/replicateexisting-s3-objects-using-s3-batch-replication/

[10] https://docs.aws.amazon.com/AmazonS3/latest/user guide/s3-batch-replication-batch.html

[11] https://aws.amazon.com/blogs/aws/new-crossregion-replication-for-amazon-s3/

[12] https://aws.amazon.com/blogs/storage/replicatingexisting-objects-between-s3-buckets/

[13] https://www.workfall.com/learning/blog/replicateexisting-s3-objects-using-s3-batch-replication/