Current time: 10-24-2020, 10:57 AM Hello There, Guest! (LoginRegister)

Post Reply 
 
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Tests not found or going to incorrect results
10-07-2020, 07:12 AM
Post: #1
Question Tests not found or going to incorrect results
Over the course of the last 2-3 weeks I have noticed more and more of my Tests in my test history are starting to fail with "Test not found" when I click on the results, for example the below test https://www.webpagetest.org/result/20092...965a691c6/
What's even stranger other tests in my history now link to a Google Maps test made from Tokyo in July when the url shows it was a test from September
https://www.webpagetest.org/result/20092...3845673c3/

Am I hitting some kind of limits in my account or are there storage issues?

Thank you,
Andreas
Find all posts by this user
Quote this message in a reply
10-08-2020, 12:58 AM
Post: #2
RE: Tests not found or going to incorrect results
Not an account problem or a limits problem, sorry. I transitioned to a different way of archiving tests (http instead of NFS mount) and there were a few hiccups where tests were lost but it shouldn't be an ongoing problem. I may have broken something more than I thought though if it is still happening from tests from the last few days.

The maps test is also my fault. Some user's automated test scripts were stuck in a loop constantly re-requesting tests that were not found and not handling the error condition so I copied an old valid test to backfill some of the missing tests and make their scripts stop hammering the server. Unfortunately it looks like I was probably a little too aggressive with the backfill.
Visit this user's website Find all posts by this user
Quote this message in a reply
10-08-2020, 03:01 AM
Post: #3
Thumbs Up RE: Tests not found or going to incorrect results
Thank you Patrick,
I will run my usual battery of tests today and see if everything is working again. If I see something not working I will let you know.
Find all posts by this user
Quote this message in a reply
10-15-2020, 04:41 AM (This post was last modified: 10-15-2020 04:52 AM by andreasg.)
Post: #4
Thumbs Down RE: Tests not found or going to incorrect results
Hello Patrick,
well it looks like the tests from last week are gone again.

Link to October 9th Test https://www.webpagetest.org/result/20100...2a2a83c49/
October 7th Test
https://www.webpagetest.org/result/20100...6d56c5d68/
September 29th
https://www.webpagetest.org/result/20092...80d7d4f83/
September 28th
https://www.webpagetest.org/result/20092...5023b548e/

I looks like I have lost all test history data since mid September and none of the test have saved for longer than maybe a day. Going to run again my Wednesday test suite hoping it will save this time.

I just noticed something. One of the tests from last week did not get lost

https://www.webpagetest.org/result/20100...daf93993d/

The biggest difference between that test and the other tests is that I did that one fresh through the Form on the Home page for all the other tests I used the "Re-run the test" button for example rerunning this old test
https://www.webpagetest.org/result/20091...71b5070bd/

Could the re-run have something to do with the storage issue?
Find all posts by this user
Quote this message in a reply
10-15-2020, 06:21 AM
Post: #5
RE: Tests not found or going to incorrect results
That's bizarre. The test infra did change where the long-term storage is archived to and there were a few days of hiccups along the way but it shouldn't be persistent and there is around the same volume of data being archived so there is some edge condition that isn't being handled properly.

A few questions if you don't mind that might help me isolate it:
- Were the tests run manually or through the API?
- What location?
- Do you know if the tests completed quickly or were in the queue for a really long time?
- Did the tests complete and then disappear or were they queued and then disappeared?
Visit this user's website Find all posts by this user
Quote this message in a reply
10-15-2020, 06:48 AM
Post: #6
RE: Tests not found or going to incorrect results
I significantly simplified the archiving logic, particularly the part that deletes tests to make sure it never deletes a test that hasn't been archived. Hopefully that helps:

https://github.com/WPO-Foundation/webpag...3a373a09d6
Visit this user's website Find all posts by this user
Quote this message in a reply
10-15-2020, 06:52 AM
Post: #7
RE: Tests not found or going to incorrect results
To answer your questions.
- Were the tests run manually or through the API?
The tests were all run manually through the webpagetest.org UI.
- What location?
I am assuming you mean Geography. They ran from "Toronto, Canada - Chrome - Cable"
This is todays test currently accessible https://www.webpagetest.org/result/20101...2456b331e/
- Do you know if the tests completed quickly or were in the queue for a really long time?
They were queued up just as expetced 10 tests with 9 first and repeat runs each
- Did the tests complete and then disappear or were they queued and then disappeared?
The tests completed, I saw the results and when I checked a week later they were gone.

Hope this info sheds more light on it.
Find all posts by this user
Quote this message in a reply
10-20-2020, 02:03 AM
Post: #8
Thumbs Down RE: Tests not found or going to incorrect results
Good Morning Patrick,
yet another set of tests got lost https://www.webpagetest.org/result/201014_DiG6_ce58de30a15cfdf49df2bcc013fcf501/
and
https://www.webpagetest.org/result/20101...336e283aa/

yet this one
https://www.webpagetest.org/result/20101...631cf4942/
and this one
https://www.webpagetest.org/result/20101...2456b331e/
Seem to have survived. So 2 test survived 9 tests vanished.

Let me know what I can do to help debug this further.
Find all posts by this user
Quote this message in a reply
10-20-2020, 06:01 AM
Post: #9
RE: Tests not found or going to incorrect results
Do you know if those were after my change or before (they were on the same day)?

That said, I just added another level of protection to have it verify that the archived copy exists in the long-term storage by actively polling for it before allowing the local copy to be deleted. If it still gets deleted after this change then it is going to be from a different chunk of code somewhere (not the archiving code).
Visit this user's website Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: 1 Guest(s)