Current time: 12-12-2017, 01:06 AM Hello There, Guest! (LoginRegister)

Post Reply 
 
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
urlblast crawl mode
07-08-2009, 11:13 PM
Post: #1
urlblast crawl mode
Hi Patrick,

i have been running urlblast over our portfolio 24/7 and imported the page log into a mysql db.

Now i'm starting to mine the data i've noticed that some of the urls may have been tested 446 times, and some only 311 (where 446 is the highest count), i have noticed when watching the monitor that the urls appear to picked randomly despite me having put them in alphabetical order.

is this a feature or expected behaviour?

Cheers,


Stuart.
Find all posts by this user
Quote this message in a reply
07-08-2009, 11:30 PM
Post: #2
RE: urlblast crawl mode
They do run randomly but it should be consistent. The behavior is supposed to be that the list is loaded and then the urls are picked off of the list in a random order until all of the urls have been tested and then it repeats. This was done so that when we have testing running across a large number of machines they aren't all hitting the same urls at the same time.

Should be an easy enough configuration change to have them optionally run in order.

I have seen issues where the random selecting seemed to not hit all of the urls but every time I went back to test it through a debugger or with logging it always checked out Undecided
Visit this user's website Find all posts by this user
Quote this message in a reply
07-12-2009, 07:24 AM
Post: #3
RE: urlblast crawl mode
Hi Patrick,

that makes sense.


Cheers,

Stuart.
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: 1 Guest(s)