Current time: 12-19-2017, 12:23 AM Hello There, Guest! (LoginRegister)

Post Reply 
 
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
AWS Private instance: Agents not being stopped/terminated
01-13-2015, 07:15 AM (This post was last modified: 01-13-2015 08:07 AM by andydavies.)
Post: #11
RE: AWS Private instance: Agents not being stopped/terminated
I think this is the full set of permissions needed - think there may be one or two extra in there.

Code:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1420829356000",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags",
                "ec2:DescribeRegions",
                "ec2:DescribeVolumes",
                "ec2:DeleteVolume",
                "ec2:DescribeInstances",
                "ec2:RunInstances",
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:TerminateInstances"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Andy

Using WebPageTest - http://usingwpt.com/
Visit this user's website Find all posts by this user
Quote this message in a reply
01-26-2015, 06:07 PM (This post was last modified: 01-26-2015 06:07 PM by kevinrdixon.)
Post: #12
RE: AWS Private instance: Agents not being stopped/terminated
I already have
Code:
{
      "Action": "ec2:*",
      "Effect": "Allow",
      "Resource": "*"
    },
...which allows everything on EC2.

This is still a problem. This morning, I have one agent instance which is not stopped, and isn't listed in /getTesters.php?f=html.
Looks like it failed to be stopped - from error.log.20150125:
Code:
00:00:02 - EC2:Launching EC2 instance. Region: us-east-1, AMI: ami-XXXXX, error: The instance ID 'i-YYYYY' does not exist
00:00:05 - EC2:Listing running EC2 instances: The instance ID 'i-YYYYY' does not exist
In fact, the instance DOES exist, and it looks like the call to terminate instance intermittently fails. This is a repeated pattern which has happened in the past (similar entries in the error logs).
Perhaps a retry mechanism for terminating instances?
Find all posts by this user
Quote this message in a reply
01-29-2015, 01:10 AM
Post: #13
RE: AWS Private instance: Agents not being stopped/terminated
I'll double-check the logic but it should always pull the full list of running instances and if it sees one that shouldn't be running and is tagged with the WPT tags it should try to terminate it every time it checks (every 5 minutes but the terminations are done only close to hourly increments on the run time).
Visit this user's website Find all posts by this user
Quote this message in a reply
01-29-2015, 06:08 PM
Post: #14
RE: AWS Private instance: Agents not being stopped/terminated
Same deal this morning:

Code:
00:00:05 - EC2:Launching EC2 instance. Region: us-east-1, AMI: ami-561cb13e, error: The instance ID 'i-ZZZZZ' does not exist

i-ZZZZZ is still running 8 hours later.
Find all posts by this user
Quote this message in a reply
02-09-2015, 06:50 PM
Post: #15
RE: AWS Private instance: Agents not being stopped/terminated
I've been travelling for around 3 days, and this morning I've 4 agent instances which are not terminated, and not listed in /getTesters.php. All have the log error shown above.
I've raised this as an issue now, since it is certainly happening with a regularity on my private instance: https://github.com/WPO-Foundation/webpag...issues/397
Find all posts by this user
Quote this message in a reply
02-10-2015, 04:50 AM
Post: #16
RE: AWS Private instance: Agents not being stopped/terminated
I updated the bug and have a theory with what might be causing it and a fix in the works.
Visit this user's website Find all posts by this user
Quote this message in a reply
03-03-2015, 07:48 PM
Post: #17
RE: AWS Private instance: Agents not being stopped/terminated
Hi,

I have also from time to time an instance which aren't terminated.

As I see in AWS console they are not tagged like the other instances with "Name=WebPagetest Agent" but have empty names.
In the log I have for these instances always startup messages like:

Code:
09:05:10 - Instance i-5959c1bd started: m3.medium ami ami-d0c76fa7 in eu-west-1 for eu-west-1,eu-west-1_IE11 with user data: wpt_server=xx.xx.xx.xx wpt_loc=eu-west-1,eu-west-1_IE11 wpt_key=xxxxx
09:05:10 - Error: Launching EC2 instance. Region: eu-west-1, AMI: ami-d0c76fa7, error: The instance ID 'i-5959c1bd' does not exist

Bests

Reiner
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: 1 Guest(s)