I'd recommend turning off the tcp_nopush setting. It defaults to off and push is what avoids Nagle interacting with the 200ms delayed acks. tcp_nodelay defaults to on which is where you want it as well.
I've running more tests and I think I've got another useful data point.
A couple of weeks ago I was experimenting with 'gzip_static on;' in Nginx. I noticed that when that was turned on, and there was a pre-compressed file available, that the HTTP responses from Nginx looked more like a traditional static file ( which makes sense, it really is at that point ).
What is missing from that response is the "Content-Length:" header. I didn't think much of this originally. I figured that when Nginx was doing compression on the fly it simply didn't do the extra work to include that value in the response.
Then I remembered that with "gzip_static on;" it would include it in the length in the response. So I combined that with this experiment, turning gzip_static on and creating a pre-compressed test-slow.html.gz for Nginx to use ( gzip -6 test-slow.html ). A quick call with curl confirmed that with all of this in place "Content-Length: 1019" was included in the response.
So perhaps the issue really is with Nginx and the cross sections of SPDY and compression.
I should have looked closer, when compressing on the fly Nginx is doing "Transfer-Encoding: chunked".
This does a good job of explaining the situation with Nginx, HTTP compression, and content length - https://github.com/CPAN-API/cpan-api/iss...t-10258159
I think I've figured this out. Doing more tests to confirm.
More edge cases turning up as I do more tests. I've found options to deal with the situation where a single file is requested over SPDY. Unfortunately those options don't appear to work when multiple requests come across a SPDY connection, even though it is still just the first one that experiences the delay.
I've got another test that looks like requesting two resources. That ran into the same ~200ms TTFB delay as well, even with "gzip_min_length 1603;" set. The second request was not delayed, even though it was also for a very small file.
In the two request case it came down to the first request being less than 1,009 bytes triggering the delay:
A patch was provided by the folks at Nginx, which I've tested and confirmed that it removed the ~200ms delay that was happening under these specific conditions. This morning that patch was committed to the Nginx repo - http://trac.nginx.org/nginx/changeset/2c.../nginx/src
Very cool. I expect that will actually happen a lot in the wild under all sorts of different conditions so it's probably quite a win for nginx perf. In your tests it was pretty clear because it was showing in TTFB but from the looks of the patch it might commonly just show as an extra 200ms delivering the last bit of data which might be less obvious but still pretty nasty.