AWS Private instance: Agents not being stopped/terminated

[andydavies]

I think this is the full set of permissions needed - think there may be one or two extra in there.

Code:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1420829356000",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags",
                "ec2:DescribeRegions",
                "ec2:DescribeVolumes",
                "ec2:DeleteVolume",
                "ec2:DescribeInstances",
                "ec2:RunInstances",
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:TerminateInstances"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

[kevinrdixon]

I already have

Code:

{
      "Action": "ec2:*",
      "Effect": "Allow",
      "Resource": "*"
    },

...which allows everything on EC2.

This is still a problem. This morning, I have one agent instance which is not stopped, and isn't listed in /getTesters.php?f=html.
Looks like it failed to be stopped - from error.log.20150125:

Code:

00:00:02 - EC2:Launching EC2 instance. Region: us-east-1, AMI: ami-XXXXX, error: The instance ID 'i-YYYYY' does not exist
00:00:05 - EC2:Listing running EC2 instances: The instance ID 'i-YYYYY' does not exist

In fact, the instance DOES exist, and it looks like the call to terminate instance intermittently fails. This is a repeated pattern which has happened in the past (similar entries in the error logs).
Perhaps a retry mechanism for terminating instances?

[pmeenan]

I'll double-check the logic but it should always pull the full list of running instances and if it sees one that shouldn't be running and is tagged with the WPT tags it should try to terminate it every time it checks (every 5 minutes but the terminations are done only close to hourly increments on the run time).

[kevinrdixon]

Same deal this morning:

Code:

00:00:05 - EC2:Launching EC2 instance. Region: us-east-1, AMI: ami-561cb13e, error: The instance ID 'i-ZZZZZ' does not exist

i-ZZZZZ is still running 8 hours later.

[kevinrdixon]

I've been travelling for around 3 days, and this morning I've 4 agent instances which are not terminated, and not listed in /getTesters.php. All have the log error shown above.
I've raised this as an issue now, since it is certainly happening with a regularity on my private instance: https://github.com/WPO-Foundation/webpag...issues/397

[pmeenan]

I updated the bug and have a theory with what might be causing it and a fix in the works.

[Reiner030]

Hi,

I have also from time to time an instance which aren't terminated.

As I see in AWS console they are not tagged like the other instances with "Name=WebPagetest Agent" but have empty names.
In the log I have for these instances always startup messages like:

Code:

09:05:10 - Instance i-5959c1bd started: m3.medium ami ami-d0c76fa7 in eu-west-1 for eu-west-1,eu-west-1_IE11 with user data: wpt_server=xx.xx.xx.xx wpt_loc=eu-west-1,eu-west-1_IE11 wpt_key=xxxxx
09:05:10 - Error: Launching EC2 instance. Region: eu-west-1, AMI: ami-d0c76fa7, error: The instance ID 'i-5959c1bd' does not exist

Bests

Reiner