urlblast crawl mode

[pmeenan]

They do run randomly but it should be consistent. The behavior is supposed to be that the list is loaded and then the urls are picked off of the list in a random order until all of the urls have been tested and then it repeats. This was done so that when we have testing running across a large number of machines they aren't all hitting the same urls at the same time.

Should be an easy enough configuration change to have them optionally run in order.

I have seen issues where the random selecting seemed to not hit all of the urls but every time I went back to test it through a debugger or with logging it always checked out