WebPagetest Forums
urlblast crawl mode - Printable Version

+- WebPagetest Forums (https://www.webpagetest.org/forums)
+-- Forum: Web Performance (/forumdisplay.php?fid=3)
+--- Forum: Testing Tools/Services (/forumdisplay.php?fid=6)
+--- Thread: urlblast crawl mode (/showthread.php?tid=26)



urlblast crawl mode - hsiboy - 07-08-2009 11:13 PM

Hi Patrick,

i have been running urlblast over our portfolio 24/7 and imported the page log into a mysql db.

Now i'm starting to mine the data i've noticed that some of the urls may have been tested 446 times, and some only 311 (where 446 is the highest count), i have noticed when watching the monitor that the urls appear to picked randomly despite me having put them in alphabetical order.

is this a feature or expected behaviour?

Cheers,


Stuart.


RE: urlblast crawl mode - pmeenan - 07-08-2009 11:30 PM

They do run randomly but it should be consistent. The behavior is supposed to be that the list is loaded and then the urls are picked off of the list in a random order until all of the urls have been tested and then it repeats. This was done so that when we have testing running across a large number of machines they aren't all hitting the same urls at the same time.

Should be an easy enough configuration change to have them optionally run in order.

I have seen issues where the random selecting seemed to not hit all of the urls but every time I went back to test it through a debugger or with logging it always checked out Undecided


RE: urlblast crawl mode - hsiboy - 07-12-2009 07:24 AM

Hi Patrick,

that makes sense.


Cheers,

Stuart.