WebPagetest Forums
"requests.csv" vs "Request Details" - Printable Version

+- WebPagetest Forums (https://www.webpagetest.org/forums)
+-- Forum: Web Performance (/forumdisplay.php?fid=3)
+--- Forum: Testing Tools/Services (/forumdisplay.php?fid=6)
+--- Thread: "requests.csv" vs "Request Details" (/showthread.php?tid=774)



"requests.csv" vs "Request Details" - agraan - 05-21-2011 04:35 AM

I conducted a test on http://www.cnn.com on May 19, 2011 at 13:30 CST and compared the data in the "Request Details" table (via the WPT GUI) with the data in the "requests.csv" file for the same test.

Observations:
1) There are only 172 rows of data in the former (GUI table), but 176 rows of data corresponding to the "Cleared Cache-Run" case in the latter (CSV file).
2) Apart from the "Time to First Byte" column, I was not able to match the data in many of the other columns.

Am I misreading/misinterpreting something or is the data from these 2 sources in fact inconsistent?

Assuming the 2 sources are NOT compatible, what would be the best way to download the former (i.e. GUI table) via scripts? (Hoping to avoid having to wade through HTML code to extract the data...)

Thanks,
Anshul


RE: "requests.csv" vs "Request Details" - pmeenan - 05-21-2011 04:59 AM

Well, since the CSV is actually used to draw the UI I don't think it's a mismatch problem, possibly just a processing difference.

The first thing you need to do is only look at "Transaction Type" == 3 - those are the requests (type 0 is the summary page data and 8 shouldn't be in the data but looks like it's sneaking through). That will get you to the 172 rows.

After that there is a bunch of adjusting and calculating that is done to get the individual times (mostly because some of the component times are included in the "load time").

The "DNS Lookup" in the GUI table is the same as the "DNS Time" column
The "Initial Connection" in the GUI table is the same as the "Connect Time" column
The TTFB you already have matching
The "Content Download" in the GUI is "Time to Load" - "Time to First Byte"

The tricky one is Offset. The "Start Time" in the CSV is the offset of the request, not including the DNS, socket connect or SSL. To get the offset you need to subtract the DNS, Connect and SSL times from "Start Time"


RE: "requests.csv" vs "Request Details" - agraan - 05-21-2011 06:34 AM

Indeed, based on some random checks, the data from the CSV does match up to the GUI table as per your instructions. *Relieved!*

Thanks - I would have never figured out those mappings myself...

Regards,
Anshul
Just to get a quick confirmation from you:

Although there is no equivalent GUI table for the Repeat View on WPT (right?), if I look at the CSV file, I can follow your instructions (i.e. look at only rows with "Transaction Type" == 3, etc, etc) for the rows corresponding to "Cached-Run" in column C, to derive similar data (Start Offset, TTFB, etc) for the Repeat View, right?

Thanks,
Anshul


RE: "requests.csv" vs "Request Details" - pmeenan - 05-21-2011 07:23 AM

If you click on a repeat view waterfall you should be taken to the equivalent page that has a repeat view data table. You can either use the name or the explicit "cached" column to distinguish.


RE: "requests.csv" vs "Request Details" - agraan - 05-24-2011 02:09 AM

How can I determine the "Start Render" and "Document Complete" times (green and blue lines respectively in the GUI's Waterfall View) from the CSV files?

Thanks,
Anshul


RE: "requests.csv" vs "Request Details" - pmeenan - 05-24-2011 04:24 AM

Those are in the page data csv file (should have columns for both of them).