You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Ferrum::Browser#restart causes "Errno::EMFILE: Too many open files - /usr/bin/chromium" #637
To Reproduce
reduce the number of maximum open files and produce fetching pages periodically calling #restart
Though I expect that the bug may be fixed just reading the code
Expected behavior
I expect that #quit will close all open files
Screenshots
See backtrace above
Desktop (please complete the following information):
OS: Linux
Browser [Chrome Version 153.0.8010.36 (Official Build) (x86_64)
Thanks for the report. I have a suspicion but need a couple of numbers from your side to confirm it, since the fds Ferrum could be stranding on #restart should be sockets, not regular files.
Could you run this in the worker process (or from a shell against its pid) right after boot, and again after N restarts:
ls -l /proc/<pid>/fd | awk '{print $NF}'| sed 's/:\[[0-9]*\]//'| sort | uniq -c | sort -rn | head -20
grep 'Max open files' /proc/<pid>/limits
What I'm after: does the growing count consist of socket:[...] entries, and does it grow by a fixed amount per #restart (1 or 2)?
Also helpful:
Does the process pass :url/:ws_url (connecting to an already-running Chrome), or does Ferrum spawn Chrome itself?
Any Ferrum::Proxy, or :xvfb, in the options?
Roughly how many #restart calls before EMFILE, and what's the soft nofile limit you lowered it to? A near-exact ratio would nail it down.
Diagnostic: if you call GC.start right after browser.quit/before #restart, does the growth stop? If yes, that points at fds that are only reclaimed when the connection object is garbage-collected, which is exactly the bug I suspect.
Is it one Browser instance being restarted, or one per job?
A minimal loop that reproduces it (even without the job framework) would let me verify a fix directly.
Thanks for the data, but it doesn't answer any of the questions I asked, and those are still what I need to confirm the cause. Unanswered, in particular:
The fd breakdown in the worker process (ls -l /proc/<pid>/fd | awk '{print $NF}' | sed 's/:\[[0-9]*\]//' | sort | uniq -c | sort -rn) right after boot and again after N restarts — are the growing entries socket:[...], and does the count grow by a fixed amount per #restart?
Does your code pass :url/:ws_url, or does Ferrum spawn Chromium itself?
Any Ferrum::Proxy or :xvfb in the options?
How many #restart calls before EMFILE, and what soft nofile limit did you lower it to?
Does GC.start right before #restart stop the growth?
One Browser instance being restarted, or one per job?
A minimal loop that reproduces it, without the job framework.
And I don't understand the init part of your report at all. In SpaceCleaner, roots = @ppids.keys - @ppids.values.flatten are the parents of chromium trees that are not themselves chromium — i.e. your worker processes, not browsers. So please spell out what the 1 you see after a worker restart actually is. My guess is pid 1, meaning chromium trees outlived the worker and were reparented to init, but that's a guess and the screenshot doesn't tell me. What are the column names in that table, and what exactly does the top value count?
One thing I did notice in SpaceCleaner: it collects only the descendants of each root into @to_kill and never the roots themselves, so it kills every chromium below them — including the browser of a job that is still running. That alone can make Ferrum hit a dead browser for reasons unrelated to this issue, so I'd separate the two before drawing conclusions from those counts.
Independently of all that, there is a real leak in 0.18.0 that matches "#quit leaves chromium processes running" and the leftover ferrum_user_data_dir_*:
Process#stop in 0.18.0 sends SIGUSR1 to the pid Ferrum spawned only, never to the process group, and the wait loop ends as soon as that leader is reaped — so Chromium's zygote/renderer/GPU children are never signaled and stay alive.
The user data dir was removed by its own separate finalizer, which could fire while Chromium was still running; Chromium then recreates the directory on the way out and it leaks permanently.
Both are fixed on main: TERM now goes to the whole process group, escalating to KILL after 2s, and the loop only stops once nothing in the group is alive; the directory is removed after the kill, with retries, from a single finalizer. Please test main:
gem"ferrum",github: "rubycdp/ferrum"
With 10 #restart calls in a row on main I see a flat fd count, no extra chromium processes and no new ferrum_user_data_dir_* — including when the browser tree is SIGKILLed from outside between restarts. If it still grows for you there, the fd breakdown from question 1 is what will tell us whether the leftovers are sockets or files.
Thanks for the data, but it doesn't answer any of the questions I asked, and those are still what I need to confirm the cause. Unanswered, in particular:
1. The fd breakdown in the worker process (`ls -l /proc/<pid>/fd | awk '{print $NF}' | sed 's/:\[[0-9]*\]//' | sort | uniq -c | sort -rn`) right after boot and again after N restarts — are the growing entries `socket:[...]`, and does the count grow by a fixed amount per `#restart`?
$ ll /proc/7825/fd | awk '{print $NF}' | sed 's/:\[[0-9]*\]//' | sort | uniq -c | sort -rn | head -20
876 socket=
3 /dev/pts/12
1 /home/dima/devel/www/1tv/log/development.log
1 /dev/null
1 anon_inode:[eventpoll]
1 anon_inode:[eventfd]
1 0
$ grep 'Max open files' /proc/7825/limits
Max open files 1024 4096 files
2. Does your code pass `:url`/`:ws_url`, or does Ferrum spawn Chromium itself?
4. How many `#restart` calls before `EMFILE`, and what soft `nofile` limit did you lower it to?
I think about the number of open files
5. Does `GC.start` right before `#restart` stop the growth?
I call #restart after each @page.go_to
6. One `Browser` instance being restarted, or one per job?
in a job I instantiate a new @ferrum and after each @page.go_to I send #reset
7. A minimal loop that reproduces it, without the job framework.
it is difficult
And I don't understand the init part of your report at all. In SpaceCleaner, roots = @ppids.keys - @ppids.values.flatten are the parents of chromium trees that are not themselves chromium — i.e. your worker processes, not browsers. So please spell out what the 1 you see after a worker restart actually is. My guess is pid 1, meaning chromium trees outlived the worker and were reparented to init, but that's a guess and the screenshot doesn't tell me. What are the column names in that table, and what exactly does the top value count?
root contains the pid of the job runner and 1 that corresponds to init process of the OS. After a job runner is killed their children are assigned to it.
One thing I did notice in SpaceCleaner: it collects only the descendants of each root into @to_kill and never the roots themselves, so it kills every chromium below them — including the browser of a job that is still running. That alone can make Ferrum hit a dead browser for reasons unrelated to this issue, so I'd separate the two before drawing conclusions from those counts.
the job runner is not to be killed
Independently of all that, there is a real leak in 0.18.0 that matches "#quit leaves chromium processes running" and the leftover ferrum_user_data_dir_*:
* `Process#stop` in 0.18.0 sends `SIGUSR1` to the pid Ferrum spawned only, never to the process group, and the wait loop ends as soon as that leader is reaped — so Chromium's zygote/renderer/GPU children are never signaled and stay alive.
* The user data dir was removed by its own separate finalizer, which could fire while Chromium was still running; Chromium then recreates the directory on the way out and it leaks permanently.
Both are fixed on main: TERM now goes to the whole process group, escalating to KILL after 2s, and the loop only stops once nothing in the group is alive; the directory is removed after the kill, with retries, from a single finalizer. Please test main:
gem "ferrum", github: "rubycdp/ferrum"
With 10 #restart calls in a row on main I see a flat fd count, no extra chromium processes and no new ferrum_user_data_dir_* — including when the browser tree is SIGKILLed from outside between restarts. If it still grows for you there, the fd breakdown from question 1 is what will tell us whether the leftovers are sockets or files.
I have upgraded the gem to master and if a problem remains will report
in a job I instantiate a new @ferrum and after each @page.go_to I send #reset
That is the leak. #reset is not a teardown — it disposes browser contexts and nothing else. It does not close the connection and it does not stop the browser. A Ferrum::Browser you create and never #quit leaks by construction, and not just until GC: against main, creating one per iteration, calling #go_to, dropping it and forcing GC.start each time gives
boot: fds=7 threads=1
after drop+GC #1: fds=8 threads=4
after drop+GC #2: fds=9 threads=6
after drop+GC #3: fds=10 threads=8
after drop+GC #4: fds=11 threads=10
after drop+GC #5: fds=12 threads=12
after drop+GC #6: fds=13 threads=14
settled: fds=13 threads=14
One socket and two threads per browser, permanently — the fd never comes back no matter how much GC runs, because the subscriber threads are still alive and keep the socket reachable, so Ruby never finalizes it. Chromium itself does get killed by the finalizer, which is why your process counts move around while the fd count only goes up. The same loop with #quit is flat.
So #quit every browser you create:
begin# ...ensure@ferrum&.quitend
Please add that and post the fd breakdown again. Note also that your answers describe two different flows — the issue body and your answer to (5) say you call #restart after each #go_to, (6) says a new browser per job with #reset — so tell me which one the 490 sockets came from.
I do have two further findings, one a real bug in #quit itself and one in your browser_options, but they are both smaller than this and impossible to measure underneath it. Let's get the #quit in first.
in a job I instantiate a new @ferrum and after each @page.go_to I send #reset
I have added #reset to my code after I have noticed that even when the instance of ferrum goes to the garbage collector, the files and the running instances of chromium remains
I do have rescue Exception => error block that does not have ensure but creates a log record abut the exception. And during my execution no such records are created.
So, I am sure that all instances of ferrum have received #restart.
Please add that and post the fd breakdown again. Note also that your answers describe two different flows — the issue body and your answer to (5) say you call #restart after each #go_to, (6) says a new browser per job with #reset — so tell me which one the 490 sockets came from.
I got an idea. I will call #quit instead of #restart and let us see what happened.
I do have two further findings, one a real bug in #quit itself and one in your browser_options, but they are both smaller than this and impossible to measure underneath it. Let's get the #quit in first.
But in any case, you have to implement the proper finalisation in your code for the case the application that uses the gem did not call #quit. Additional information about how to do that I send in the attachment
in a job I instantiate a new @ferrum and after each @page.go_to I send #reset
Then the accounting is expected, not a bug. #reset is not a teardown: it disposes browser contexts and nothing else. It does not close the connection and it does not stop the browser. A Ferrum::Browser you create and never #quit holds its socket and its threads for the life of the process — and your process is a Que worker with a thread pool, so "the life of the process" is thousands of jobs.
Running exactly that flow against main — your browser_options, create_page, go_to, reset, drop the browser, GC.start, six times:
One socket and two threads per job, and they never come back — not on the next GC, not ever. The leftovers are the client's two subscriber threads, still parked on Queue#pop; a live thread is a GC root, and through its registered callbacks it keeps Client -> WebSocket -> @sock reachable, so the socket is never finalized. Chromium itself is killed, by the process finalizer — that is the chrome=0 column. So your Chromium processes come and go while the socket count only ever climbs, which is exactly the shape of your 876 socket= and 490 socket=. Six jobs cost six fds here; a pooled worker chewing through a queue gets you to 876 without any leak on our side.
Quit every browser you create:
begin# ...ensure@ferrum&.quitend
The same loop with #quit is flat.
The thread pool breaks SpaceCleaner too. It walks the Chromium trees under $$ and kills them — but $$ is the whole worker process, shared by every job thread in the pool, so it kills the browsers of jobs that are still running in sibling threads, and rm -rfs their /tmp/ferrum_user_data_dir* underneath them. That is where killed: 385 and ferrum: 55 come from: your own cleaner, not a Ferrum leak. Those jobs then hit a dead browser, drop the Browser object without quitting, and leak another socket each.
Also worth noting your last report is evidence the process leak is fixed: top: ["22574"] has no 1 in it, where your earlier one did — no Chromium tree outlived the worker and got reparented to init.
So: add the #quit, turn SpaceCleaner off.
#quit what kills the browser process meaning that it's no longer usable #restart is quit + start new pid
You are in a thread pool you have to kill the browser, where instead you were doing either reset or restart none of which applicable in a pool. You can't pass browser instance like that between threads.
#reset is used in capybara tests, when you need to re-use your browser for next test, this essentially wipes all the pages but browser stays #quit what kills the browser process meaning that it's no longer usable #restart is quit + start new pid
You are in a thread pool you have to kill the browser, where instead you were doing either reset or restart none of which applicable in a thread pool. You can't pass browser instance like that between threads.
Initially I did not call #restart nor #quit and got a lot of open files and running chromium instances. I was sure that when I release my instance of ferrum it cleans all but it turned out to be wrong. And this is from my point of view the bug in the gem. So I ask you to fix it. How to do it I explained. When an instance is released it should clean the space. Tomorrow I'll tell you if #quit works, but you should not require all the users to do that and take care to remove not more required staff the gem created.
Oh please! Kick your LLM to make it work hard and shed your light on us with an open PR.
I don't have time to explain both simple reset/restart/quit and complex things like finalizers to someone if they can't grasp even the simple ones.
I was sure that when I release my instance of ferrum it cleans all but it turned out to be wrong
If you stated this from the beginning I would close it immediately w/o spending so much time on it.
Activity
route commentedon Sep 26, 2026
Thanks for the report. I have a suspicion but need a couple of numbers from your side to confirm it, since the fds Ferrum could be stranding on
#restartshould be sockets, not regular files.Could you run this in the worker process (or from a shell against its pid) right after boot, and again after N restarts:
What I'm after: does the growing count consist of
socket:[...]entries, and does it grow by a fixed amount per#restart(1 or 2)?Also helpful:
:url/:ws_url(connecting to an already-running Chrome), or does Ferrum spawn Chrome itself?Ferrum::Proxy, or:xvfb, in the options?#restartcalls beforeEMFILE, and what's the softnofilelimit you lowered it to? A near-exact ratio would nail it down.GC.startright afterbrowser.quit/before#restart, does the growth stop? If yes, that points at fds that are only reclaimed when the connection object is garbage-collected, which is exactly the bug I suspect.Browserinstance being restarted, or one per job?A minimal loop that reproduces it (even without the job framework) would let me verify a fix directly.
route commentedon Sep 26, 2026
Closing for now until you reply
dima4p commentedon Sep 26, 2026
I have the similar information received with my CleanUpJob that performs
SpaceCleaner.new($$).callhere are the reports:
This all shows that #quit leave the chromium processes running. And probably the files open. At least not deleted.
dima4p commentedon Sep 26, 2026
I have to add that the problem is being cured with restart of the worker. After that in the list of
topappears "1".route commentedon Sep 27, 2026
Thanks for the data, but it doesn't answer any of the questions I asked, and those are still what I need to confirm the cause. Unanswered, in particular:
ls -l /proc/<pid>/fd | awk '{print $NF}' | sed 's/:\[[0-9]*\]//' | sort | uniq -c | sort -rn) right after boot and again after N restarts — are the growing entriessocket:[...], and does the count grow by a fixed amount per#restart?:url/:ws_url, or does Ferrum spawn Chromium itself?Ferrum::Proxyor:xvfbin the options?#restartcalls beforeEMFILE, and what softnofilelimit did you lower it to?GC.startright before#restartstop the growth?Browserinstance being restarted, or one per job?And I don't understand the
initpart of your report at all. InSpaceCleaner,roots = @ppids.keys - @ppids.values.flattenare the parents of chromium trees that are not themselves chromium — i.e. your worker processes, not browsers. So please spell out what the1you see after a worker restart actually is. My guess is pid 1, meaning chromium trees outlived the worker and were reparented to init, but that's a guess and the screenshot doesn't tell me. What are the column names in that table, and what exactly does thetopvalue count?One thing I did notice in
SpaceCleaner: it collects only the descendants of each root into@to_killand never the roots themselves, so it kills every chromium below them — including the browser of a job that is still running. That alone can make Ferrum hit a dead browser for reasons unrelated to this issue, so I'd separate the two before drawing conclusions from those counts.Independently of all that, there is a real leak in 0.18.0 that matches "
#quitleaves chromium processes running" and the leftoverferrum_user_data_dir_*:Process#stopin 0.18.0 sendsSIGUSR1to the pid Ferrum spawned only, never to the process group, and the wait loop ends as soon as that leader is reaped — so Chromium's zygote/renderer/GPU children are never signaled and stay alive.Both are fixed on main: TERM now goes to the whole process group, escalating to KILL after 2s, and the loop only stops once nothing in the group is alive; the directory is removed after the kill, with retries, from a single finalizer. Please test main:
With 10
#restartcalls in a row on main I see a flat fd count, no extra chromium processes and no newferrum_user_data_dir_*— including when the browser tree isSIGKILLed from outside between restarts. If it still grows for you there, the fd breakdown from question 1 is what will tell us whether the leftovers are sockets or files.dima4p commentedon Sep 27, 2026
NO,
in initialize
and then
@page.go_to urlsee above
I think about the number of open files
I call
#restartafter each@page.go_toin a job I instantiate a new
@ferrumand after each@page.go_toI send#resetit is difficult
rootcontains the pid of the job runner and1that corresponds toinitprocess of the OS. After a job runner is killed their children are assigned to it.the job runner is not to be killed
I have upgraded the gem to master and if a problem remains will report
dima4p commentedon Sep 28, 2026
With the commit ea93308 I see the following:
So, the bug is not fixed.
route commentedon Sep 28, 2026
That is the leak.
#resetis not a teardown — it disposes browser contexts and nothing else. It does not close the connection and it does not stop the browser. AFerrum::Browseryou create and never#quitleaks by construction, and not just until GC: againstmain, creating one per iteration, calling#go_to, dropping it and forcingGC.starteach time givesOne socket and two threads per browser, permanently — the fd never comes back no matter how much GC runs, because the subscriber threads are still alive and keep the socket reachable, so Ruby never finalizes it. Chromium itself does get killed by the finalizer, which is why your process counts move around while the fd count only goes up. The same loop with
#quitis flat.So
#quitevery browser you create:Please add that and post the fd breakdown again. Note also that your answers describe two different flows — the issue body and your answer to (5) say you call
#restartafter each#go_to, (6) says a new browser per job with#reset— so tell me which one the 490 sockets came from.I do have two further findings, one a real bug in
#quititself and one in yourbrowser_options, but they are both smaller than this and impossible to measure underneath it. Let's get the#quitin first.dima4p commentedon Sep 28, 2026
I have added
#resetto my code after I have noticed that even when the instance of ferrum goes to the garbage collector, the files and the running instances of chromium remainsI do have
rescue Exception => errorblock that does not haveensurebut creates a log record abut the exception. And during my execution no such records are created.So, I am sure that all instances of ferrum have received
#restart.I got an idea. I will call
#quitinstead of#restartand let us see what happened.But in any case, you have to implement the proper finalisation in your code for the case the application that uses the gem did not call
#quit. Additional information about how to do that I send in the attachmentruby-gem-finalizer-cleanup-summary.txt
route commentedon Sep 28, 2026
I think there's a confusion here on your side!
Then the accounting is expected, not a bug.
#resetis not a teardown: it disposes browser contexts and nothing else. It does not close the connection and it does not stop the browser. AFerrum::Browseryou create and never#quitholds its socket and its threads for the life of the process — and your process is a Que worker with a thread pool, so "the life of the process" is thousands of jobs.Running exactly that flow against
main— yourbrowser_options,create_page,go_to,reset, drop the browser,GC.start, six times:One socket and two threads per job, and they never come back — not on the next GC, not ever. The leftovers are the client's two subscriber threads, still parked on
Queue#pop; a live thread is a GC root, and through its registered callbacks it keepsClient -> WebSocket -> @sockreachable, so the socket is never finalized. Chromium itself is killed, by the process finalizer — that is thechrome=0column. So your Chromium processes come and go while the socket count only ever climbs, which is exactly the shape of your876 socket=and490 socket=. Six jobs cost six fds here; a pooled worker chewing through a queue gets you to 876 without any leak on our side.Quit every browser you create:
The same loop with
#quitis flat.The thread pool breaks
SpaceCleanertoo. It walks the Chromium trees under$$and kills them — but$$is the whole worker process, shared by every job thread in the pool, so it kills the browsers of jobs that are still running in sibling threads, andrm -rfs their/tmp/ferrum_user_data_dir*underneath them. That is wherekilled: 385andferrum: 55come from: your own cleaner, not a Ferrum leak. Those jobs then hit a dead browser, drop theBrowserobject without quitting, and leak another socket each.Also worth noting your last report is evidence the process leak is fixed:
top: ["22574"]has no1in it, where your earlier one did — no Chromium tree outlived the worker and got reparented to init.So: add the
#quit, turnSpaceCleaneroff.#quitwhat kills the browser process meaning that it's no longer usable#restartis quit + start new pidYou are in a thread pool you have to kill the browser, where instead you were doing either
resetorrestartnone of which applicable in a pool. You can't pass browser instance like that between threads.dima4p commentedon Sep 28, 2026
Initially I did not call
#restartnor#quitand got a lot of open files and running chromium instances. I was sure that when I release my instance of ferrum it cleans all but it turned out to be wrong. And this is from my point of view the bug in the gem. So I ask you to fix it. How to do it I explained. When an instance is released it should clean the space. Tomorrow I'll tell you if#quitworks, but you should not require all the users to do that and take care to remove not more required staff the gem created.route commentedon Sep 28, 2026
Oh please! Kick your LLM to make it work hard and shed your light on us with an open PR.
I don't have time to explain both simple reset/restart/quit and complex things like finalizers to someone if they can't grasp even the simple ones.
If you stated this from the beginning I would close it immediately w/o spending so much time on it.