04-17-2011, 12:36 AM
Have you a bot name like w3c_validator or googlebot ?
Some sites must make two redirects to check some ability with javascript and write internal vars in php.
you detect this and follow, but you should do not - that was only for real users.
Bots like
$config['bots']=array('w3c_validator','googlebot','msnbot','slurp','gigabot','scrubby','robozila','nutch','ia_archiver','baiduspider','yahoo-mmcrawler','psbot','asterias','yahoo-blogs','SeitenReport');
are disabled automatically for redirects and so it was a good idea to set your name is this method.
Some sites must make two redirects to check some ability with javascript and write internal vars in php.
you detect this and follow, but you should do not - that was only for real users.
Bots like
$config['bots']=array('w3c_validator','googlebot','msnbot','slurp','gigabot','scrubby','robozila','nutch','ia_archiver','baiduspider','yahoo-mmcrawler','psbot','asterias','yahoo-blogs','SeitenReport');
are disabled automatically for redirects and so it was a good idea to set your name is this method.