Current time: 12-16-2017, 08:45 PM Hello There, Guest! (LoginRegister)

Post Reply 
 
Thread Rating:
  • 1 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Feature Request: Urlblast auto registration
04-28-2010, 12:21 AM
Post: #1
Feature Request: Urlblast auto registration
Not sure if this is the correct place for this, but....

How difficult would it be to have an auto registering urlblast agent?

The idea would be that you simply point your urlblast application to a host running the web page test server and it would auto register that agent and add it to the list of available locations. You could either have the user input a label and location in urlblast, or use the geo location of the requesting agent when adding to the location list.

Also, if the remote agent doesn't ping the server periodically that location would be removed from the list.

We are setting up a server that will likely have many remote agents. This would be most helpful from the perspective of managing the server configuration.

Thanks,

Tony
Find all posts by this user
Quote this message in a reply
04-28-2010, 12:40 AM
Post: #2
RE: Feature Request: Urlblast auto registration
It should be doable without too much work. I obviously wouldn't want to do that for the public instance but it does make some sense for private agents (actually, auto-hiding agents that haven't checked in in a while is on my list to do even for the public instance as we sometimes have boxes go down where there is only a single test machine). On the public instance it might also make sense to allow for auto-registration but with a secret key exchange of some kind.

I wouldn't want to use the geo-location info though. For it to be useful the agent will also need to advertise it's connectivity which should probably be configured, not detected (though detecting the browser version would be possible).
Visit this user's website Find all posts by this user
Quote this message in a reply
04-28-2010, 12:52 AM
Post: #3
RE: Feature Request: Urlblast auto registration
Additionally it would be handy if the agent would pass in some basic info upon registering.

Like...

CPU:
RAM:
OS:
Browser:
...
Find all posts by this user
Quote this message in a reply
04-29-2010, 01:06 AM
Post: #4
RE: Feature Request: Urlblast auto registration
I'm thinking of modifying the code to support wild cards in specifying where the locations are configured.

This would be used in conjunction with the auto registering agents. The idea would be that when an agent registers instead of modifying the locations.ini it would create a new file like locations_keller.ini.

If the remote agent doesn't check in periodically this file gets deleted, which would result in it no longer showing up on the "test" page.

Can you think of any issues I might run into?

Thanks.
Find all posts by this user
Quote this message in a reply
04-29-2010, 03:59 AM
Post: #5
RE: Feature Request: Urlblast auto registration
There are a couple of issue that come to mind (when it comes to implementing anyway):

- When a location checks for work, the code checks to make sure the location is valid. This is where I'd probably recommend doing the registration
- When the test submission page is loaded it pulls the list of locations (other than adding the logic to pull in the other locations this can pretty much stay as is)
- When work is submitted the location is checked to make sure it is valid and to decide where to store the job file. This is where things get a bit tricky because technically you could have a location disappear between when the test page is loaded and when the request is submitted

The other thing that needs to be considered is finding a good place to add the code that removes idle locations. There is support for a cron-job based request that checks all of the locations and sends a mail alert if a location hasn't checked in in a while - that might be a good place to put it. You could also do it on every visit to the test page but that seems like an awful lot of checking (maybe would work well for a private install that doesn't get a lot of traffic).
Visit this user's website Find all posts by this user
Quote this message in a reply
06-04-2010, 07:30 AM (This post was last modified: 06-04-2010 08:02 AM by asperkins.)
Post: #6
RE: Feature Request: Urlblast auto registration
Ok. I have auto registration working. It needs some more tweaking, but so far it works quite nicely.

The server settings no longer have to be updated for each remote agent. That process is handled automagically now. Smile


I discovered that you can have several params in the location= param in urlblast.ini.

My example...

urlblast.ini
---
...
location=Keller_TX&connectivity=FIOS&down=25000000&up=15000000&latency=5&browser=IE+8
...
---

I added some settings to settings.ini

// Used to enable agent auto registration
enableAutoRegister=1
// The key expected from the agent
autoRegisterKey=akey
// The base location for the jobs
autoRegisterLocalDir=/var/www/webpagetest/work/jobs/

So, in getwork, if autoregister is on, I grab all the params and create a file like...

locations_Keller_TX.ini
---
[Keller_TX]
browser=IE 8
down=25000000
up=15000000
connectivity=FIOS
latency=5
label=Keller_TX
localDir=/var/www/webpagetest/work/jobs/Keller_TX
---

I moved the retrieval of locations to a function in common.inc

function GetLocations()

This function will find all files that match locations_*.ini and process them. It builds and returns the expected array.

Also, on the "New Test" tab/page, I check to see if the agent has checked in in the last 5 minutes. If not, the ini file gets deleted and it won't show up. As soon as the agent hits getwork again the file is recreated and he shows up.

If two agents come in with the same location, they will collide. With a little more work I could add support for multiple agents from the same location, but I'm not sure yet if that's needed. I'll just have my users confirm that they didn't use an already used location name.

I hope to get this code back into the main repo. Do you prefer that I send a patch file for your review when I'm done?

btw. Is latency just an estimate, or is it calculated by some means?

Thanks,

Tony
If you'd like to test it, do the following...

In you urlblast.ini

Url Files Url=http://71.170.186.242/work/

Set the location param to something like.
*** Replace the items like Keller_TX, FIOS, etc. with your info.

Location=Keller_TX&connectivity=FIOS&down=25000000&up=15000000&latency=5&browser=IE+8
Location Key=br549

Fire it up and go to http://71.170.186.242 and try it.
Find all posts by this user
Quote this message in a reply
06-04-2010, 09:49 AM
Post: #7
RE: Feature Request: Urlblast auto registration
Yes, please send a patch and I can get it merged into the mainline. I'll probably want to enhance it to detect Dummynet and support dynamic bandwidth as well (a recent addition).

The latency is the measured RTT for the first hop and configured manually (tracert from the test machine out and checking the latency to the ISP).

I can do the multiple agent/same location work when I merge if you'd like. I actually have 8 test systems running in Dulles and they basically get work in a first-come first-served basis. I'll also see if I can eliminate some of the absolute paths that are in the configs and generate them dynamically (that's one of the nagging things I wanted to clean up anyway).

Thanks for working on this!
Visit this user's website Find all posts by this user
Quote this message in a reply
06-04-2010, 09:53 AM
Post: #8
RE: Feature Request: Urlblast auto registration
One thing just came to mind....

You might want to update the "last checked time" every time results are posted back as well. If someone does 10 test runs first & repeat view of a slow page it could take as long as 20 minutes to go through them and the location would hide itself while working on the test.
Visit this user's website Find all posts by this user
Quote this message in a reply
06-04-2010, 09:59 AM (This post was last modified: 06-04-2010 10:00 AM by asperkins.)
Post: #9
RE: Feature Request: Urlblast auto registration
(06-04-2010 09:53 AM)pmeenan Wrote:  One thing just came to mind....

You might want to update the "last checked time" every time results are posted back as well. If someone does 10 test runs first & repeat view of a slow page it could take as long as 20 minutes to go through them and the location would hide itself while working on the test.

That's a good point. I was thinking of making the timeout a settings.ini configurable.
(06-04-2010 09:49 AM)pmeenan Wrote:  Yes, please send a patch and I can get it merged into the mainline. I'll probably want to enhance it to detect Dummynet and support dynamic bandwidth as well (a recent addition).

The latency is the measured RTT for the first hop and configured manually (tracert from the test machine out and checking the latency to the ISP).

I can do the multiple agent/same location work when I merge if you'd like. I actually have 8 test systems running in Dulles and they basically get work in a first-come first-served basis. I'll also see if I can eliminate some of the absolute paths that are in the configs and generate them dynamically (that's one of the nagging things I wanted to clean up anyway).

Thanks for working on this!

Yeah, I guess as long as the multiple agents are all from the same area it's fine for them to share the work.
Find all posts by this user
Quote this message in a reply
04-18-2012, 12:48 AM
Post: #10
RE: Feature Request: Urlblast auto registration
This is just what i was looking for! Smile

Has this feature been implemented yet, or can someone send over a patch to me also?

I'm just in the process of setting up agents, and these changes would help now and in the future! Smile

Is there a tool or script that auto cleans jobs older than a month for example from a private wpt instance?

Thanks

Tom
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: 1 Guest(s)