WebPagetest Forums
Comparison of WPT.org and Pingdom testing method - Printable Version

+- WebPagetest Forums (https://www.webpagetest.org/forums)
+-- Forum: Web Performance (/forumdisplay.php?fid=3)
+--- Forum: Testing Tools/Services (/forumdisplay.php?fid=6)
+--- Thread: Comparison of WPT.org and Pingdom testing method (/showthread.php?tid=11696)

Comparison of WPT.org and Pingdom testing method - thompsonpaul - 09-23-2012 12:38 PM

I'm sure I saw a description on the boards here at one time by Patrick of the difference between the way WPT performs it's tests and how Pingdom does theirs.

I know WPT uses an actual browser session and closely imitates a user's Internet connection.

Given that Pingdom's tests are always so much faster, I'm assuming their using some "imitation' of a user Internet connection that is actually a faster, more direct-access pipe.

Can anyone describe (or point me to a source for the info on) the difference between the way these two systems measure?



RE: Comparison of WPT.org and Pingdom testing method - pmeenan - 09-24-2012 11:01 PM

The biggest differences are going to be in the connectivity, machine configuration and browser. Pingdom runs their tests from a backbone-connected data center which basically has zero latency for the last mile and effectively unlimited bandwidth.

They claim to be using Chrome for testing (on their tools.pingdom.com page) but it's not clear what the browser settings are or what OS they are running on (Linux would be my guess but it would just be a guess).

I usually use http://www.engadget.com as a test to see if someone is using a real browser because it's complex enough that the results differ drastically if they aren't and it looks like the results from Pingdom (requests and bytes) are pretty close to what WPT returns.

That said, looking at their waterfall, it doesn't look like it loads anything like Chrome should (at least on Windows). My guess is that they have a bug in how they record the "Connect" time because it looks like there are hundreds of requests in flight in parallel but that "connect" time appears to be "blocked" time where the browser is waiting for an available connection to use for the request.

It looks like requests are fulfilled in 4-7ms for just about everything served from a CDN and there doesn't appear to be much/any wait for js processing so a combination of fast hardware and fast connectivity are both driving the insanely fast times.