WebPagetest Forums
Random failures from Taiwan WPT instance - Printable Version

+- WebPagetest Forums (https://www.webpagetest.org/forums)
+-- Forum: Web Performance (/forumdisplay.php?fid=3)
+--- Forum: Discuss Test Results (/forumdisplay.php?fid=4)
+--- Thread: Random failures from Taiwan WPT instance (/showthread.php?tid=15455)

Random failures from Taiwan WPT instance - jakub - 07-26-2018 06:47 PM


I've been testing using various locations, and I've noticed that when I use "Taiwan - GCE" instance (any browser - Chrome, Firefox, Opera), sometimes it results in 503 response.

It is happening for any resource on Dailymotion.com domain (the example is a static JSON file to make things less complicated).

See here:
- (HTTP request) https://www.webpagetest.org/result/180725_GW_6b3d7a1f82ce8f8d4c9b32a6c7df8f30/
- (HTTPS request) https://www.webpagetest.org/result/180726_25_2550ce5437534804ff650420da260523/

My question is, how can I be sure the issue is not on WPT Taiwan instance, but on DM server side?
I asked my infra team and they say they don't see any 503s in the graphs/logs.

Can those 503s come from WPT itself? If so, what could be the reasons?

Does WPT do Man-in-the-Middle and other things like that?

Thanks in advance for any suggestions.

RE: Random failures from Taiwan WPT instance - pmeenan - 07-26-2018 11:56 PM

No, WPT doesn't proxy the outbound requests. A 503 would be generated by a reverse-proxy somewhere in the path, not at the server itself. If you are using a CDN then it is likely coming from the CDN edge, possibly because it can't reach the origin.

It's also possible that some security layers somewhere in front of the server are filtering/blocking the traffic intermittently intentionally (particularly since it is coming from Google Cloud).

You can capture a tcpdump which will show the 503's coming back from the edge ( in the case of the test you sent). From the looks of it that's not a CDN edge but may still be a load balancer or security device somewhere upstream of the server logs.

RE: Random failures from Taiwan WPT instance - pmeenan - 07-26-2018 11:58 PM

btw, the tcpdump option is in the "advanced settings" tab of the advanced settings (capture packet dump). Sharing the actual tcpdump with the ops team will tend to help move the conversation along to "yes, it's really happening"

RE: Random failures from Taiwan WPT instance - jakub - 07-27-2018 07:59 PM

Hi Patrick, thanks for you input and the pcap suggestion!

Some things I forgot to mention / did not emphasise enough:

- the issues happen both for static and dynamic resources, I just used a static JSON URL for simplicity
- the IP from the WPT logs is which is the expected IP for the domain, resolved in the same way from everywhere (it's not geodistributed / not a CDN)
- the issue happens both for http and https requests with equal chance
- so far I only reproduced on WPT Taiwan instance, not others (that's a bit weird honestly)

My reasoning is: since the IP matches, and issue happens for HTTPS request, and there's no MitM etc. (hence the integrity is guaranteed, no third-party proxies messing with the output), the problem indeed must be on the server side (maybe some anti-bot solution as you mentioned).

This is an interesting run where the initial http request is fine (redirects to https) but https gets 503 response:
pcap is here:

I checked the PCAP for the https request and it looks like it contains the proper cert etc.

I will check with the devops again. Thanks!