Gaps in waterfall specifically in IE

[quarterdome]

Thanks, Sundeep.

So you have a support link / number about the OSCP stapling not being supported for custom domains on CloudFront? SSL Labs showing OSCP stapling enabled on cdn0.apartmentlist.com (which is powered by cloudfront). Perhaps the issue is already fixed?

[SundeepPundamale]

Hello Matt,

Yes we have a ticket number and the latest update says AWS is investigating the issue and their product teams are on the case. It is an internal support case so you might not be able to view the ticket history/details.

What you say is correct. For apartmentlist.com OSCP stapling works even for custom domains. In our case OSCP stapling doesn't work for custom domains. We cross checked both the URL's in SSL Labs.

We also shared this finding with AWS(for apartmentlist.com OSCP stapling works).

[alien_]

I work with Sundeep, and we finally figured out what was wrong.

The AWS support team checked on their side and confirmed that everything worked well, but the way we did our tests for OCSP stapling was actually flawed.

The OCSP stapling happens at the CloudFront edge level so every node from an edge location needs to do it. The first request returns immediately a non-stapled answer and the node fires an OCSP request to our CA and then caches the answer which is then used for the subsequent requests hitting that edge node. (The nodes currently don't share these caches, but I filed a feature request so that they try to use a shared storage for that content, like they do for static content.)

During our test we only executed a relatively small number of requests, so we never hit the same edge node twice and that's why it appeared the stapling was broken. When testing you need to fire a few hundreds of requests until consistently getting stapled responses.

As for explaining the rest of the gap from our waterfall graph, we saw that it often happens that the CPU is maxed out while loading our app, for multiple reasons:

• sometimes due to multiple SSL connections started at the same time, since they are quite heavy during the negotiation phase
• while other gaps can be explained due to heavy CPU use during the parsing of Javascript and CSS content (our webapp code is around 400K when minified and compressed)
  

When these overlap, the gap can grow even up to a few seconds, and it's exacerbated if the testing machine is not powerful enough.

[alien_]

(09-30-2014 08:59 AM)quarterdome Wrote:  The results from a Thinkpad look way more reasonable.

The gap for the http://www.apartmentlist.com is about 10x smaller than in the test I linked about, and is about 200ms. I would expect that to be normal OCSP delay, since http://www.apartmentlist.com does not support OSCP stapling (due to Heroku not supporting it at the moment). The gaps for cdnX.apartmentlist.com are even shorter (presumably because CloudFront supports OSCP stapling).

However, what I see from Denver location looks terrible. We have RUM (Real User Monitoring) enabled via NewRelic on our app, and we see that there is a large number of users that are experiencing similar performance performance in real life (20+ second page loads).

At first I thought there is something wrong with out certificate (we had one from RapidSSL), but I replaced it with the one from Comodo (total shot in the dark!) and absolutely nothing has changed in the waterfall and these gaps.

Any idea what I can do to get to the bottom of it?

The CPU usage is the bottleneck. I think you have too many SSL connections being set up in parallel, those are quite heavy on the CPU.

If possible, try to switch to a CDN provider that supports SPDY, so you'll only have an SSL connection to each of your domains.