Current time: 05-29-2020, 03:01 PM Hello There, Guest! (LoginRegister)

Post Reply 
 
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
OSError: [Errno 12] Cannot allocate memory
12-20-2019, 03:01 AM
Post: #1
OSError: [Errno 12] Cannot allocate memory
Our test agents have started to fail after successfully running for a few hours. The error is: OSError: [Errno 12] Cannot allocate memory

We use docker images on c4.large instances (4GB RAM) but have also seen this failure using on-prem instances with 8GB of ram. Rolling back to an image from early December did not fix the issue, so I suspect this could be due to a browser change.

To get agents working again, we can restart the container. Adding swap also seems to fix the issue, but we have not run tests long enough to see if this is a permanent fix.

Anyone else see this issue and have a permanent fix?

We run via docker with these options:
docker run -d --shm-size=1g -e "SERVER_URL=http://<fqdn>/work/" -e "LOCATION=${region}_Linux" --cap-add="NET_ADMIN" --restart unless-stopped --name <custom-repo>/webpagetest/agent:latest
Find all posts by this user
Quote this message in a reply
01-16-2020, 11:21 PM
Post: #2
RE: OSError: [Errno 12] Cannot allocate memory
Any news here? Our 2 agents exhaust 8GB of memory within less than 8 hours.
Find all posts by this user
Quote this message in a reply
01-17-2020, 09:19 AM
Post: #3
RE: OSError: [Errno 12] Cannot allocate memory
We are working around the issue by doing an hourly container restart. If you do a "ps" on the host, you will see a lot of defunct chrome processes.

From our user-data script:
"# Restart once/hour for zombie procs\n",
"sudo echo '#!/bin/bash\n",
"sleep $(( ${RANDOM:0:1}*60 ))\n",
"docker restart wpt' > /etc/cron.hourly/wpt-restart\n",
"sudo chmod 755 /etc/cron.hourly/wpt-restart\n"
Find all posts by this user
Quote this message in a reply
01-20-2020, 09:53 PM
Post: #4
RE: OSError: [Errno 12] Cannot allocate memory
Thank you, jedmichnowicz. So do we restart them with some frequency. Weird, why just two of us are here. I believe it's a common issue.
The only reason for no answers, I guess, is that wpt docker is not maintained here.
Find all posts by this user
Quote this message in a reply
02-05-2020, 10:33 PM
Post: #5
RE: OSError: [Errno 12] Cannot allocate memory
You're not alone! I'm also experiencing this problem.
Find all posts by this user
Quote this message in a reply
03-26-2020, 06:38 PM
Post: #6
RE: OSError: [Errno 12] Cannot allocate memory
Same problem here. I just upgraded a few of my agents to the latest docker image (published on mar-20, 2020) and started seeing this error. Agents are running version 20.01 ... anybody found a solution for this yet?
Find all posts by this user
Quote this message in a reply
03-26-2020, 07:37 PM
Post: #7
RE: OSError: [Errno 12] Cannot allocate memory
We do as @jedmichnowicz suggested, restart them periodically. In our case (8G of memory for two agent containers) every 8 hours is good enough.
Find all posts by this user
Quote this message in a reply
03-27-2020, 08:15 AM
Post: #8
RE: OSError: [Errno 12] Cannot allocate memory
yea, that's a good idea and i was already restarting twice a day. I just rolled back my agents to v18.10 and those are working fine. When i build a fresh new container with a v20.01 agent, i get these memory errors immediately, on the first test i run. So it's not happening over time. There is definitely a difference between the 18.10 and 20.01 images (I haven't tried the 19.04 image yet) ... i don't have the exact log lines anymore but i recall seeing the memory error occur in python's subprocess module ... i'll post here if i see it again
Find all posts by this user
Quote this message in a reply
04-18-2020, 08:31 PM (This post was last modified: 04-21-2020 07:16 PM by maximet.)
Post: #9
RE: OSError: [Errno 12] Cannot allocate memory
Hello,

I have also this memory problem. I first thought it was because of the defunct process too.
Passing the --init flag to docker run fixes the defunct chrome processes problem but not the memory problem.

The defunct chrome processes are not well managed by the container. What you can do to fix that correctly is to init the container with https://github.com/krallin/tini as the init (PID 1) of the container instead of the /wptagent/entrypoint.sh command. tini is designed to manage defunct processes
But it doesn't fix the meory problem :/
Find all posts by this user
Quote this message in a reply
05-21-2020, 12:18 AM
Post: #10
RE: OSError: [Errno 12] Cannot allocate memory
we are also seeing "can not allocate memory issue "
wpt agentt 20.01
max memory : 4gb, cpu =2
total consumued mem is 95%
consumed memory : python 65 %, aws 6.7 , firefox 5.9 ,
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: 1 Guest(s)