S3 API too slow? xargs to the rescue.
My days as a BigData engineer grew darker in the past few days. Now almost all critical data that our aggregations, partners and customers want to access is located in a bucket on Amazon’s S3. Since that gets horribly expensive for the vast amounts of data we create on a daily basis, we lean towards not keeping much more than a month of active objects in the bucket, to meet our SLA’s regarding backup and recovery, we keep them for about a year in Glacier afterwards....