wpt_batch.py - xml failed to retrieve

[sjwill23]

We are submitting requests via wpt_batch.py on a private instance with a local agent and cant find a way around the error "The xml failed to retrieve."

c:\WPT>wpt_batch.py --location=EG_Win7:Chrome
WARNING:root:The XML failed to retrieve: 141027_9F_9afe69b7c16b47ecf4b6d81135d73d23

We can see that the test runs fine and appears to complete successfully. We just cant seem to get the results xml to be sent back to the client who ran wpt_batch.py successfully (though the result directory does get created successfully). Here is the entry the script--

result_test_ids = set(test_results.keys())
for test_id in completed_test_ids:
if test_id not in result_test_ids:
logging.warn('The XML failed to retrieve: %s', test_id)

[pmeenan]

What happens if you try to retrieve the results manually in a browser (to see if it is a server or batch code problem)?

i.e., open a browser and load:
http://<server>/results.php?test=141027_9F_9afe69b7c16b47ecf4b6d81135d73d23

and
http://<server>/xmlResult.php?test=141027_9F_9afe69b7c16b47ecf4b6d81135d73d23

[sjwill23]

Yep, both of those do indeed work. It appears everything on the server side is ok and the tests run successfully:

xxx@devwptserver01:/var/www/html/results/14/10/29/9R/fd7d83efc174eb580a5002b89f07a115$ ls
1.0.visual.dat.gz 1_Cached_screen.jpg 1_screen.jpg test.lock
1.1.visual.dat.gz 1_IEWPG.txt.gz breakdown1.json test.log
1_Cached_IEWPG.txt.gz 1_IEWTR.txt.gz breakdown1.json.gz video_1
1_Cached_IEWTR.txt.gz 1_progress.csv.gz testinfo.ini
1_Cached_progress.csv.gz 1_report.txt.gz testinfo.json.gz
1_Cached_report.txt.gz 1_screen_doc.jpg test.job
xxx@devwptserver01:/var/www/html/results/14/10/29/9R/fd7d83efc174eb580a5002b89f07a115$ more test.log
2014/10/29 17:07:13 - Test Created
2014/10/29 17:10:26 - Starting test (initiated by tester DEVWPTAGENT02-10.130.253.13)
2014/10/29 17:10:58 - Test Run Complete. Run: 1, Cached: 0, Done: , Tester: DEVWPTAGE
NT02-10.130.253.13
2014/10/29 17:11:24 - Test Run Complete. Run: 1, Cached: 1, Done: , Tester: DEVWPTAGE
NT02-10.130.253.13
2014/10/29 17:11:24 - Test Run Complete. Run: 1, Cached: 1, Done: 1, Tester: DEVWPTAG
ENT02-10.130.253.13
2014/10/29 17:11:24 - Test Complete
2014/10/29 17:11:24 - Test post-processing complete

Since the tests above worked I figured the next thing to try is running the same wpt_batch.py test locally on the server itself. I get the same error. I guess that rules out permissions or any other connection type issue. So it must be something up with the GetXMLResult function? Only thing I can think of is maybe I have an old version of the wpt_batch or wpt_batch_lib? If you happen to have a link to these files where you are sure they are the latest versions I can do a quick diff to rule that out (and I attached the ones we are using).

[pmeenan]

The code is with the project in github: https://github.com/WPO-Foundation/webpag.../batchtool

I don't think the code has changed in years but to be fair I'm not all that familiar with it (someone else authored that script). If you're still having issues after updating let me know and I'll see if I can do any debugging on it.

[sjwill23]

Thanks Patrick, yeah I did a diff and it seems my files are the exact same except I have my server ip hard coded in. I shouldn't have said I have automatically eliminated permissions by running the script on the server itself. I was thinking there still could be a permission missing on a folder or file where the test/id or something else cannot be called or accessed. With that said, I used the install check script on the server and everything looks ok there too as far as permissions (and everything else).


PHP version at least 5.3: 5.5.9-1ubuntu4.4
GD Module Installed: yes
zip Module Installed: yes
zlib Module Installed: yes
curl Module Installed: yes
php.ini allow_url_fopen enabled: yes
APC Installed: NO (optional)
php.ini upload_max_filesize > 10MB: 20M
php.ini post_max_size > 10MB: 80M
System Utilities

ffmpeg Installed (required for video): yes
ffmpeg 1.x Installed with fps, scale and decimate filters(required for mobile video): 1.2.6-7:1.2.6-1~trusty1,fps,scale,decimate
imagemagick compare Installed (required for mobile video): yes
jpegtran Installed (required for JPEG Analysis): yes
exiftool Installed (required for JPEG Analysis): yes
Filesystem Permissions

{docroot}/tmp writable: yes
{docroot}/results writable: yes
{docroot}/work/jobs writable: yes
{docroot}/work/video writable: yes
{docroot}/logs writable: yes

[sjwill23]

Here's some more findings:

In the file wpt_batch_lib.py, there's a function called GetXMLResult, I see that it is constructing a URL of:
http://<server>/xmlResult/<test_id>/

It doesn't seem to work, we get a 404. What do I need to change on the server end to get this to work? I tried changing the code to the following which does return an XML but it's expecting a DOM object instead.

http://<server>/xmlResult.php?test=<test_id>

That resulted in other errors.

[sjwill23]

I ended up slightly modifying the code to get it working. I don't know if this is correct but it worked. I changed the URL being built to pull the xml results to this:
http://<server>/xmlResult.php?test=<test_id>

Then I had to modify how the filename was being built since the modified URL breaks that portion of the code.

Thanks