Current time: 12-12-2017, 12:55 PM Hello There, Guest! (LoginRegister)

Post Reply 
 
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
AWS Private instance: Agents not being stopped/terminated
01-06-2015, 11:49 PM
Post: #1
AWS Private instance: Agents not being stopped/terminated
Have a private instance on AWS using AMI ami-24199c4c, which has been working for some time. However, recently the agent instances don't seem to be being terminated when the work queue is empty.

The user data for the server instance:

Code:
ec2_key=XXX
ec2_secret=XXX
api_key=XXX
headless=0

EC2.ScaleFactor=50
EC2.us-east-1.min=0
EC2.us-east-1.max=2

Have there been recent changes that might affect this behaviour?
Find all posts by this user
Quote this message in a reply
01-07-2015, 12:09 AM
Post: #2
RE: AWS Private instance: Agents not being stopped/terminated
There were changes made back on 12/18-19 where the AWS SDK was updated (which fixed using Frankfurt) and to better scale down. It used to only turn off instances that were idle in the last hour but if any jobs were submitted they would still be spread across instances so it would keep all of the instances active. The change would mark instances as being offline based on the scale factor which would let them go idle and be terminated.

The termination is only checked on hourly increments from when the instances started so they won't go away right away (since EC2 is billed hourly).

Is there any chance that individual jobs are getting submitted and processed in between batches?

Looking at the /getTesters.php page should tell you how long it has been since the last work for a given agent. Are those all showing long times (and are all of the live instances showing up)?
Visit this user's website Find all posts by this user
Quote this message in a reply
01-07-2015, 02:57 AM (This post was last modified: 01-07-2015 02:58 AM by kevinrdixon.)
Post: #3
RE: AWS Private instance: Agents not being stopped/terminated
(01-07-2015 12:09 AM)pmeenan Wrote:  Is there any chance that individual jobs are getting submitted and processed in between batches?

Looking at the /getTesters.php page should tell you how long it has been since the last work for a given agent. Are those all showing long times (and are all of the live instances showing up)?

The instance isn't currently shared, and the test history suggests no jobs are getting submitted between batches.
/getTesters.php shows the one and only agent that is up at the moment, and that it has been idle 186 minutes.....

Is the change of AWS SDK significant - do we need to update that in some way given we're on the old AMI and WPT code is auto-updated?
Find all posts by this user
Quote this message in a reply
01-07-2015, 03:36 AM
Post: #4
RE: AWS Private instance: Agents not being stopped/terminated
I'll take a look and see if something broke with going all the way back down to zero instances. The code is running on the public WPT as well but I never have it drop below 1 so I may have missed something.

The AWS SDK change should be pulled in automatically since the AMI updates itself (the SDK is bundled in with the WPT code).

Taking a look now to see what I can find.
Visit this user's website Find all posts by this user
Quote this message in a reply
01-07-2015, 03:54 AM
Post: #5
RE: AWS Private instance: Agents not being stopped/terminated
btw, if any errors happen they should get logged to /var/www/webpagetest/www/log/error.log.<date>

You may need to 'sudo su' to be able to see the actual log but there may be hints in there.
Visit this user's website Find all posts by this user
Quote this message in a reply
01-07-2015, 05:41 AM
Post: #6
RE: AWS Private instance: Agents not being stopped/terminated
I just tried out the same AMI with the same options and it spun up a test machine to run the tests I needed and then shut it down at the end of the hour (pretty much as expected) so I'm at a bit of a loss.

Can you check the AWS IAM permissions for the key you assigned to make sure that it has permission to terminate instances?
Visit this user's website Find all posts by this user
Quote this message in a reply
01-08-2015, 06:45 PM (This post was last modified: 01-08-2015 06:48 PM by kevinrdixon.)
Post: #7
RE: AWS Private instance: Agents not being stopped/terminated
(01-07-2015 05:41 AM)pmeenan Wrote:  Can you check the AWS IAM permissions for the key you assigned to make sure that it has permission to terminate instances?

Key has full EC2 permissions:
Code:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "ec2:*",
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "elasticloadbalancing:*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "cloudwatch:*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "autoscaling:*",
      "Resource": "*"
    }
  ]
}

(01-07-2015 05:41 AM)pmeenan Wrote:  btw, if any errors happen they should get logged to /var/www/webpagetest/www/log/error.log.<date>

The only error log has one line:
Code:
01:00:02 - Error launching EC2 instance. Region: us-east-1, AMI: ami-561cb13e, error: The instance ID 'i-eba59b15' does not exist
...and that is because I terminated the agent (because it was hanging around idle). So no clue there.

I don't know if this is significant, but we have a bunch of benchmarks running overnight.
Find all posts by this user
Quote this message in a reply
01-08-2015, 11:42 PM
Post: #8
RE: AWS Private instance: Agents not being stopped/terminated
OK, what I'm seeing is a discrepancy between AWS and WPT

My AWS Agent fired up and then shutdown after a period of non-use, but both getLocations and getTesters report it as still being present - getTesters reports it's not checked in for 40 mins at this point.

I seem to remember that when the agent hasn't check-in for a while it does eventually get removed but can't remember whether I've just made that up or what the time out it if not!

Andy

Using WebPageTest - http://usingwpt.com/
Visit this user's website Find all posts by this user
Quote this message in a reply
01-09-2015, 01:19 AM
Post: #9
RE: AWS Private instance: Agents not being stopped/terminated
Oh, yes. getTesters.php will keep agents in the list for up to 60 minutes but the list is only purged when an agent actually polls for work from that location so the last tester will show forever, just with a REALLY long time since last check.

If it's causing grief I can see about having both of those only show active agents and have a separate flag that includes offline agents.
Visit this user's website Find all posts by this user
Quote this message in a reply
01-10-2015, 10:28 PM
Post: #10
RE: AWS Private instance: Agents not being stopped/terminated
Thats definitely not the problem I was having - AWS console confirmed the instances weren't being terminated.
Last nights runs appear to have terminated correctly, however, so I'll keep a tab on when this does happen next time I see it.
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: 1 Guest(s)