WebPagetest Forums
Linux EC2 AMI test agents - Printable Version

+- WebPagetest Forums (https://www.webpagetest.org/forums)
+-- Forum: WebPagetest (/forumdisplay.php?fid=7)
+--- Forum: Private Instances (/forumdisplay.php?fid=12)
+--- Thread: Linux EC2 AMI test agents (/showthread.php?tid=15088)



Linux EC2 AMI test agents - josephscott - 10-20-2017 04:28 AM

I've been using the Windows EC2 AMI test agents just fine for some time. Now I'm trying out the Linux version ( specifically ami-c4a968be in us-east-1, Virginia ). Oddly, I'm not seeing any attempt from the agent to connect to our private WPT server.

I confirmed that the instance can reach the WPT server. Pings come back fine, as well as a curl request to the WPT server. The WPT agent appears to be running ok:

python wptagent.py -vvvv --ec2 --xvfb --throttle --fps 30 --exit 60 --alive /tmp/wptagent

I've been poking around the image looking for agent logs some where that could give me a clue on what is happening ( or not happening ). Does it write logs out some where?

Any other suggestions on what to look at?

Ultimately I'm looking to have the WPT server manage spinning up / down the EC2 test agents, but one step at a time :-)


RE: Linux EC2 AMI test agents - pmeenan - 10-20-2017 06:27 AM

Possibly a difference in user data for the location string? The new agents only support wpt_loc and not the wpt_location field that had some baggage with it.


RE: Linux EC2 AMI test agents - josephscott - 10-20-2017 06:41 AM

The user data string I have looks like this:

wpt_server=https://wpt.example.com/ wpt_loc=EC2-US-EAST-N-VIRGINIA

I used the real WPT server URL instead of example.com. I confirmed the EC2 instance has this same data in /var/lib/cloud/instance/user-data.txt.

Fortunately I noticed that in www/ec2/ec2.inc.php.

Perhaps the agent doesn't support a full URL value for wpt_server?

Hunting around, I came across - https://github.com/WPO-Foundation/wptagent/blob/master/internal/webpagetest.py#L201


RE: Linux EC2 AMI test agents - josephscott - 10-20-2017 08:46 AM

Confirmed, I replaced the wpt_server line with:

self.url = '{0}work/'.format(value)

I happened to include the trailing slash in the wpt_server value. After making that change I was able to get the agent to connect and run a test.


RE: Linux EC2 AMI test agents - josephscott - 10-24-2017 03:35 AM

I've created a pull request for wptagent to support full URLs and HTTPS in the wpt_server option.

As noted in the request, my Python is still very basic, so I'm happy to have this done differently if there is a better option.

https://github.com/WPO-Foundation/wptagent/pull/63


RE: Linux EC2 AMI test agents - pmeenan - 10-24-2017 08:01 AM

PR merged. To be fair, the agent is my first real python project so it's probably not a role model for python but it works :-)

The AMI's auto-update the OS and agent code before launching the agents so they should be evergreen and automatically "just work".

FWIW, I may not have documented it yet but there is also wpt_url as a param that takes the full url.


RE: Linux EC2 AMI test agents - josephscott - 10-24-2017 09:06 AM

Thank you for getting that merged! My private WPT server has been spinning EC2 Linux agents up and down and I've confirmed that things are working as expected.


RE: Linux EC2 AMI test agents - pmeenan - 10-25-2017 01:18 AM

Awesome. Let me know if you bump into any issues or missing features. It should be pretty complete but there are a couple of settings that I'm still working on supporting (like fullSizeVideo) and some of the request manipulation options may not work in Firefox yet (adding headers, blocking requests, etc).