2

I've recently been working to create a CentOS 6.4 Amazon machine image (AMI) from scratch to be used as a base image for running Postfix email services on EC2. In my quest to learn the process, I started by using the EC2 API's ec2-import-volume command to import the disk images I've built.

After having several attempts fail at registering and running AMIs from my uploaded images, I had deleted several GB of files from these uploads that I no longer needed in order to reduce usage costs.

Throughout this process, I've been watching my AWS costs closely and noticed recently that I've been accumulating charges daily for S3 requests that I can't account for.

Today, I found my bill to be about $30 higher than I expected and downloaded a usage report to investigate. It shows hundreds of megabytes worth of requests for today alone to an empty bucket that I used more than a day ago with the import command.

Not understanding where these requests are coming from, I enabled S3 logging for that bucket and waited for logs to appear. And they did...megabytes worth, all showing HEAD requests for parts uploaded to S3 by the import command issued days ago and coming from ec2-54-244-xxx-xxx.us-west-2.compute.amazonaws.com with a user agent like JetS3t/0.8.0 (Linux amsz-x.x.x.x; amd64; JVM 1.6.0_20).

Some context:

  • I'm the only one with access to the problem bucket
  • I had no running EC2 instances or anything else related to S3 when the requests occurred
  • The user agent on the logged requests doesn't describe an instance I recognize

Where are all these requests being generated?

5
  • 2
    Welcome to SuperUser! It's better to convert your edit to an answer, rather than marking it as Solved in the title :-)
    – happy_soil
    Oct 19, 2013 at 10:07
  • Thanks! I don't have enough rep yet. As soon as the time limit passes, I'll put things where they belong. ;D
    – mossymaker
    Oct 19, 2013 at 10:22
  • Sorry, I wasn't aware that you need rep to do that!
    – happy_soil
    Oct 19, 2013 at 10:24
  • Yep—no worries! I'm glad to see people encouraging good Q&A form.
    – mossymaker
    Oct 19, 2013 at 10:26
  • If you already had the solution (which is perfectly permissible) you could have posted your answer at the same time as the question.
    – ChrisF
    Oct 19, 2013 at 10:51

1 Answer 1

1

Seeing nothing in the AWS Console, I suspected that the ec2-import-volume command from the EC2 API tools may be the culprit and found a related ec2-describe-conversion-tasks.

Executing this showed two pending and two expired tasks that apparently have been attempting conversion of disk image data that I'd since deleted and doing so over and over again.

Issuing ec2-cancel-conversion-task killed the process that was generating $30 in HEAD requests. The conversion system seems to lack any logic for intelligently handling 404s caused by the deleted disk image data. Perhaps some improvement and better documentation is due.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .