WebPagetest Forums
Combination of edge cases triggers ~200ms delay in TTFB - Printable Version

+- WebPagetest Forums (https://www.webpagetest.org/forums)
+-- Forum: WebPagetest (/forumdisplay.php?fid=7)
+--- Forum: Bugs/Issues (/forumdisplay.php?fid=10)
+--- Thread: Combination of edge cases triggers ~200ms delay in TTFB (/showthread.php?tid=13322)

Pages: 1 2

RE: Combination of edge cases triggers ~200ms delay in TTFB - pmeenan - 10-24-2014 07:30 AM

I'd recommend turning off the tcp_nopush setting. It defaults to off and push is what avoids Nagle interacting with the 200ms delayed acks. tcp_nodelay defaults to on which is where you want it as well.

RE: Combination of edge cases triggers ~200ms delay in TTFB - josephscott - 10-24-2014 07:40 AM

That would still be an odd interaction with SPDY.

Turned it off, ran another test - http://www.webpagetest.org/result/141023_4C_12EZ/ - still getting the ~200ms delay.

I've turned it back on, to preserve the original testing conditions.

RE: Combination of edge cases triggers ~200ms delay in TTFB - josephscott - 10-24-2014 08:44 AM

I've running more tests and I think I've got another useful data point.

A couple of weeks ago I was experimenting with 'gzip_static on;' in Nginx. I noticed that when that was turned on, and there was a pre-compressed file available, that the HTTP responses from Nginx looked more like a traditional static file ( which makes sense, it really is at that point ).

Specifically, look at the 'Response' section of Request #1 for the test that @igrigorik ran earlier in this thread - http://www.webpagetest.org/result/141023_KN_6fcd65a19900d95376fc023b977e6a18/1/details/

What is missing from that response is the "Content-Length:" header. I didn't think much of this originally. I figured that when Nginx was doing compression on the fly it simply didn't do the extra work to include that value in the response.

Then I remembered that with "gzip_static on;" it would include it in the length in the response. So I combined that with this experiment, turning gzip_static on and creating a pre-compressed test-slow.html.gz for Nginx to use ( gzip -6 test-slow.html ). A quick call with curl confirmed that with all of this in place "Content-Length: 1019" was included in the response.

Now for the moment of truth, I ran a new WPT test: http://www.webpagetest.org/result/141023_Z6_13WH/

The TTFB is back down to 35ms again!

So perhaps the issue really is with Nginx and the cross sections of SPDY and compression.

I should have looked closer, when compressing on the fly Nginx is doing "Transfer-Encoding: chunked".

This does a good job of explaining the situation with Nginx, HTTP compression, and content length - https://github.com/CPAN-API/cpan-api/issues/240#issuecomment-10258159

I think I've figured this out. Doing more tests to confirm.

RE: Combination of edge cases triggers ~200ms delay in TTFB - josephscott - 10-24-2014 03:13 PM

More edge cases turning up as I do more tests. I've found options to deal with the situation where a single file is requested over SPDY. Unfortunately those options don't appear to work when multiple requests come across a SPDY connection, even though it is still just the first one that experiences the delay.

Still digging.

RE: Combination of edge cases triggers ~200ms delay in TTFB - josephscott - 10-28-2014 05:39 AM

I've been running ( LOTS ) more tests against our private instance to narrow this down more.

After trying several different options I settled with adding "gzip_min_length 1603;" to the Nginx config. I was able to get a better TTFB for both of my initial tests:

- http://www.webpagetest.org/result/141027_AX_3e0ace72de9d454a55afae713bc3570d/
- http://www.webpagetest.org/result/141027_64_6a563343da1b5c188e4eb2a9d1c88b53/

I've got another test that looks like requesting two resources. That ran into the same ~200ms TTFB delay as well, even with "gzip_min_length 1603;" set. The second request was not delayed, even though it was also for a very small file.

In the two request case it came down to the first request being less than 1,009 bytes triggering the delay:

- 2 req, 1008 byte 1st (delay): http://www.webpagetest.org/result/141027_TR_e5b8e506030dafb183b345ff274fbf14/
- 2 req, 1009 byte 1st (no delay): http://www.webpagetest.org/result/141027_M0_67c890cc447e87a84e60fef2c90c83ae/

I tried shrinking small.html by a few bytes and increasing small.css by the same amount. Still triggered the delay: http://www.webpagetest.org/result/141027_NN_d0dcaeba5c073a939030594b181a2fb8/

The first request still appears to be the key here. And the 1603 limit isn't sufficient, because files below 1,009 bytes also triggered the delay.

At this point it looks like we could avoid the delay by doing the following:

- Add "gzip_min_length 1603;" to Nginx config
- Make sure all files are 1,009 bytes or more

At best these are hacky work arounds. And I'm not even 100% sure if 1,603 and 1,009 are consistent points, or just artifacts of my particular test setup.

RE: Combination of edge cases triggers ~200ms delay in TTFB - josephscott - 11-22-2014 07:18 AM

Good news!

A patch was provided by the folks at Nginx, which I've tested and confirmed that it removed the ~200ms delay that was happening under these specific conditions. This morning that patch was committed to the Nginx repo - http://trac.nginx.org/nginx/changeset/2c10db908b8c4a9c0532c58830275d5ad84ae686/nginx/src

RE: Combination of edge cases triggers ~200ms delay in TTFB - pmeenan - 11-25-2014 01:20 AM

Very cool. I expect that will actually happen a lot in the wild under all sorts of different conditions so it's probably quite a win for nginx perf. In your tests it was pretty clear because it was showing in TTFB but from the looks of the patch it might commonly just show as an extra 200ms delivering the last bit of data which might be less obvious but still pretty nasty.

RE: Combination of edge cases triggers ~200ms delay in TTFB - josephscott - 12-04-2014 04:07 AM

The latest release of Nginx, 1.7.8 ( released 02 Dec 2014 ) includes this fix - http://nginx.org/en/CHANGES - "Feature: now the "tcp_nodelay" directive works with SPDY connections."

I've written up a summary of the background and fix at https://josephscott.org/archives/2014/12/nginx-1-7-8-fixes-200ms-delay-with-spdy/