Skip to content

Ferrum::Browser#restart causes "Errno::EMFILE: Too many open files - /usr/bin/chromium" #637

Closed

Description

@dima4p

Describe the bug
I have an ActiveJob using QueAdapter. After a while it crashes with the message in title
the backtrace is here:

/usr/local/rvm/gems/ruby-4.0.5@tv/gems/ferrum-0.18.0/lib/ferrum/browser/process.rb:137:in 'Process.spawn'
/usr/local/rvm/gems/ruby-4.0.5@tv/gems/ferrum-0.18.0/lib/ferrum/browser/process.rb:137:in 'Ferrum::Browser::Process#start'
/usr/local/rvm/gems/ruby-4.0.5@tv/gems/ferrum-0.18.0/lib/ferrum/browser.rb:303:in 'Ferrum::Browser#start'

To Reproduce
reduce the number of maximum open files and produce fetching pages periodically calling #restart
Though I expect that the bug may be fixed just reading the code

Expected behavior
I expect that #quit will close all open files

Screenshots
See backtrace above

Desktop (please complete the following information):

  • OS: Linux
  • Browser [Chrome Version 153.0.8010.36 (Official Build) (x86_64)
  • Version [ferrum-0.18.0] as you can see ;-)

Activity

  1. route commented on Sep 26, 2026

    @route
    Member

    Thanks for the report. I have a suspicion but need a couple of numbers from your side to confirm it, since the fds Ferrum could be stranding on #restart should be sockets, not regular files.

    Could you run this in the worker process (or from a shell against its pid) right after boot, and again after N restarts:

    ls -l /proc/<pid>/fd | awk '{print $NF}' | sed 's/:\[[0-9]*\]//' | sort | uniq -c | sort -rn | head -20
    grep 'Max open files' /proc/<pid>/limits

    What I'm after: does the growing count consist of socket:[...] entries, and does it grow by a fixed amount per #restart (1 or 2)?

    Also helpful:

    1. Does the process pass :url/:ws_url (connecting to an already-running Chrome), or does Ferrum spawn Chrome itself?
    2. Any Ferrum::Proxy, or :xvfb, in the options?
    3. Roughly how many #restart calls before EMFILE, and what's the soft nofile limit you lowered it to? A near-exact ratio would nail it down.
    4. Diagnostic: if you call GC.start right after browser.quit/before #restart, does the growth stop? If yes, that points at fds that are only reclaimed when the connection object is garbage-collected, which is exactly the bug I suspect.
    5. Is it one Browser instance being restarted, or one per job?

    A minimal loop that reproduces it (even without the job framework) would let me verify a fix directly.

  2. route commented on Sep 26, 2026

    @route
    Member

    Closing for now until you reply

  3. dima4p commented on Sep 26, 2026

    @dima4p
    Author

    I have the similar information received with my CleanUpJob that performs SpaceCleaner.new($$).call

    class SpaceCleaner
      attr_reader :pid
    
      def initialize(pid)
        @pid = pid
        @ppids = {}
        @to_kill = []
      end
    
      def call
        if pids = get_pids
          get_ppids pids
          roots = @ppids.keys - @ppids.values.flatten
          roots.each do |pid|
            command = `ps -o cmd #{pid}`.split("\n")[1].split(' ').first.chomp
            next if command.split('/').last == 'plasmashell'
            process_children pid
          end
            logger.debug {%%#{self.class.name}@#{__LINE__}#call: #{@pid} #{(roots).inspect} @to_kill: #{@to_kill}%}
          stdout, stderr, status = Open3.capture3 "kill #{@to_kill.join ' '}"
        end
        killed_size = @to_kill.size
        ferrum_size = FileUtils.rm_rf(Dir.glob("/tmp/ferrum_user_data_dir*")).size
        chromium_size = FileUtils.rm_rf(Dir.glob("/tmp/org.chromium.Chromium.*")).size
        AdminLog.record self, "call @#{__LINE__}",
            "pid: #{@pid}, top: #{roots}, killed: #{killed_size}, ferrum: #{ferrum_size}, chromium: #{chromium_size}"
      end
    
      private
    
      def get_pids
        stdout, stderr, status = Open3.capture3 'pgrep -f "chromium "'
        stdout.presence
      end
    
      def get_ppids(pids)
        stdout, stderr, status = Open3.capture3 "ps -o pid,ppid #{pids.gsub(/\n/, ' ')}"
        stdout.split("\n")[1..-1].each do |line|
          pid, ppid = line.sub(/^\s*/, '').split(/\s+/)
          @ppids[ppid] ||= []
          @ppids[ppid] << pid
        end
      end
    
      def logger
        Rails.logger
      end
    
      def process_children(pids)
        Array.wrap(pids).each do |pid|
          if @ppids[pid]
            @to_kill += @ppids[pid]
            process_children @ppids[pid]
          end
        end
      end
    end

    here are the reports:

    Image

    This all shows that #quit leave the chromium processes running. And probably the files open. At least not deleted.

  4. dima4p commented on Sep 26, 2026

    @dima4p
    Author

    I have to add that the problem is being cured with restart of the worker. After that in the list of top appears "1".

  5. reopened this on Sep 27, 2026
  6. route commented on Sep 27, 2026

    @route
    Member

    Thanks for the data, but it doesn't answer any of the questions I asked, and those are still what I need to confirm the cause. Unanswered, in particular:

    1. The fd breakdown in the worker process (ls -l /proc/<pid>/fd | awk '{print $NF}' | sed 's/:\[[0-9]*\]//' | sort | uniq -c | sort -rn) right after boot and again after N restarts — are the growing entries socket:[...], and does the count grow by a fixed amount per #restart?
    2. Does your code pass :url/:ws_url, or does Ferrum spawn Chromium itself?
    3. Any Ferrum::Proxy or :xvfb in the options?
    4. How many #restart calls before EMFILE, and what soft nofile limit did you lower it to?
    5. Does GC.start right before #restart stop the growth?
    6. One Browser instance being restarted, or one per job?
    7. A minimal loop that reproduces it, without the job framework.

    And I don't understand the init part of your report at all. In SpaceCleaner, roots = @ppids.keys - @ppids.values.flatten are the parents of chromium trees that are not themselves chromium — i.e. your worker processes, not browsers. So please spell out what the 1 you see after a worker restart actually is. My guess is pid 1, meaning chromium trees outlived the worker and were reparented to init, but that's a guess and the screenshot doesn't tell me. What are the column names in that table, and what exactly does the top value count?

    One thing I did notice in SpaceCleaner: it collects only the descendants of each root into @to_kill and never the roots themselves, so it kills every chromium below them — including the browser of a job that is still running. That alone can make Ferrum hit a dead browser for reasons unrelated to this issue, so I'd separate the two before drawing conclusions from those counts.

    Independently of all that, there is a real leak in 0.18.0 that matches "#quit leaves chromium processes running" and the leftover ferrum_user_data_dir_*:

    • Process#stop in 0.18.0 sends SIGUSR1 to the pid Ferrum spawned only, never to the process group, and the wait loop ends as soon as that leader is reaped — so Chromium's zygote/renderer/GPU children are never signaled and stay alive.
    • The user data dir was removed by its own separate finalizer, which could fire while Chromium was still running; Chromium then recreates the directory on the way out and it leaks permanently.

    Both are fixed on main: TERM now goes to the whole process group, escalating to KILL after 2s, and the loop only stops once nothing in the group is alive; the directory is removed after the kill, with retries, from a single finalizer. Please test main:

    gem "ferrum", github: "rubycdp/ferrum"

    With 10 #restart calls in a row on main I see a flat fd count, no extra chromium processes and no new ferrum_user_data_dir_* — including when the browser tree is SIGKILLed from outside between restarts. If it still grows for you there, the fd breakdown from question 1 is what will tell us whether the leftovers are sockets or files.

  7. dima4p commented on Sep 27, 2026

    @dima4p
    Author

    Thanks for the data, but it doesn't answer any of the questions I asked, and those are still what I need to confirm the cause. Unanswered, in particular:

    1. The fd breakdown in the worker process (`ls -l /proc/<pid>/fd | awk '{print $NF}' | sed 's/:\[[0-9]*\]//' | sort | uniq -c | sort -rn`) right after boot and again after N restarts — are the growing entries `socket:[...]`, and does the count grow by a fixed amount per `#restart`?
    
    $ ll /proc/7825/fd | awk '{print $NF}' | sed 's/:\[[0-9]*\]//' | sort | uniq -c | sort -rn | head -20
        876 socket=
          3 /dev/pts/12
          1 /home/dima/devel/www/1tv/log/development.log
          1 /dev/null
          1 anon_inode:[eventpoll]
          1 anon_inode:[eventfd]
          1 0
    $ grep 'Max open files' /proc/7825/limits
    Max open files            1024                 4096                 files     
    
    2. Does your code pass `:url`/`:ws_url`, or does Ferrum spawn Chromium itself?
    

    NO,
    in initialize

        @ferrum = Ferrum::Browser.new(
          browser_options: {
            'user-agent' => @mechanize.user_agent,
            "no-crashpad" => false,
            "crash-dumps-dir" => "/tmp/crash_dumps",
          },
          pending_connection_errors: true,
          timeout: 10,
        )
        @page = @ferrum.create_page
    

    and then @page.go_to url

    3. Any `Ferrum::Proxy` or `:xvfb` in the options?
    

    see above

    4. How many `#restart` calls before `EMFILE`, and what soft `nofile` limit did you lower it to?
    

    I think about the number of open files

    5. Does `GC.start` right before `#restart` stop the growth?
    

    I call #restart after each @page.go_to

    6. One `Browser` instance being restarted, or one per job?
    

    in a job I instantiate a new @ferrum and after each @page.go_to I send #reset

    7. A minimal loop that reproduces it, without the job framework.
    

    it is difficult

    And I don't understand the init part of your report at all. In SpaceCleaner, roots = @ppids.keys - @ppids.values.flatten are the parents of chromium trees that are not themselves chromium — i.e. your worker processes, not browsers. So please spell out what the 1 you see after a worker restart actually is. My guess is pid 1, meaning chromium trees outlived the worker and were reparented to init, but that's a guess and the screenshot doesn't tell me. What are the column names in that table, and what exactly does the top value count?

    root contains the pid of the job runner and 1 that corresponds to init process of the OS. After a job runner is killed their children are assigned to it.

    One thing I did notice in SpaceCleaner: it collects only the descendants of each root into @to_kill and never the roots themselves, so it kills every chromium below them — including the browser of a job that is still running. That alone can make Ferrum hit a dead browser for reasons unrelated to this issue, so I'd separate the two before drawing conclusions from those counts.

    the job runner is not to be killed

    Independently of all that, there is a real leak in 0.18.0 that matches "#quit leaves chromium processes running" and the leftover ferrum_user_data_dir_*:

    * `Process#stop` in 0.18.0 sends `SIGUSR1` to the pid Ferrum spawned only, never to the process group, and the wait loop ends as soon as that leader is reaped — so Chromium's zygote/renderer/GPU children are never signaled and stay alive.
    
    * The user data dir was removed by its own separate finalizer, which could fire while Chromium was still running; Chromium then recreates the directory on the way out and it leaks permanently.
    

    Both are fixed on main: TERM now goes to the whole process group, escalating to KILL after 2s, and the loop only stops once nothing in the group is alive; the directory is removed after the kill, with retries, from a single finalizer. Please test main:

    gem "ferrum", github: "rubycdp/ferrum"

    With 10 #restart calls in a row on main I see a flat fd count, no extra chromium processes and no new ferrum_user_data_dir_* — including when the browser tree is SIGKILLed from outside between restarts. If it still grows for you there, the fd breakdown from question 1 is what will tell us whether the leftovers are sockets or files.

    I have upgraded the gem to master and if a problem remains will report

  8. dima4p commented on Sep 28, 2026

    @dima4p
    Author

    With the commit ea93308 I see the following:

    $ ll /proc/22574/fd | awk '{print $NF}' | sed 's/:\[[0-9]*\]//' | sort | uniq -c | sort -rn | head -20
        490 socket=
          3 /dev/pts/12
          1 /home/dima/devel/www/1tv/log/development.log
          1 /dev/null
          1 anon_inode:[eventpoll]
          1 anon_inode:[eventfd]
          1 0
    
    pid: 22574, top: ["22574"], killed: 385, ferrum: 55, chromium: 1272
    

    So, the bug is not fixed.

  9. reopened this on Sep 28, 2026
  10. route commented on Sep 28, 2026

    @route
    Member

    in a job I instantiate a new @ferrum and after each @page.go_to I send #reset

    That is the leak. #reset is not a teardown — it disposes browser contexts and nothing else. It does not close the connection and it does not stop the browser. A Ferrum::Browser you create and never #quit leaks by construction, and not just until GC: against main, creating one per iteration, calling #go_to, dropping it and forcing GC.start each time gives

    boot:              fds=7  threads=1
    after drop+GC #1:  fds=8  threads=4
    after drop+GC #2:  fds=9  threads=6
    after drop+GC #3:  fds=10 threads=8
    after drop+GC #4:  fds=11 threads=10
    after drop+GC #5:  fds=12 threads=12
    after drop+GC #6:  fds=13 threads=14
    settled:           fds=13 threads=14
    

    One socket and two threads per browser, permanently — the fd never comes back no matter how much GC runs, because the subscriber threads are still alive and keep the socket reachable, so Ruby never finalizes it. Chromium itself does get killed by the finalizer, which is why your process counts move around while the fd count only goes up. The same loop with #quit is flat.

    So #quit every browser you create:

    begin
      # ...
    ensure
      @ferrum&.quit
    end

    Please add that and post the fd breakdown again. Note also that your answers describe two different flows — the issue body and your answer to (5) say you call #restart after each #go_to, (6) says a new browser per job with #reset — so tell me which one the 490 sockets came from.

    I do have two further findings, one a real bug in #quit itself and one in your browser_options, but they are both smaller than this and impossible to measure underneath it. Let's get the #quit in first.

  11. dima4p commented on Sep 28, 2026

    @dima4p
    Author

    in a job I instantiate a new @ferrum and after each @page.go_to I send #reset

    I have added #reset to my code after I have noticed that even when the instance of ferrum goes to the garbage collector, the files and the running instances of chromium remains

    So #quit every browser you create:

    begin

    ...

    ensure
    @ferrum&.quit
    end

    I do have rescue Exception => error block that does not have ensure but creates a log record abut the exception. And during my execution no such records are created.
    So, I am sure that all instances of ferrum have received #restart.

    Please add that and post the fd breakdown again. Note also that your answers describe two different flows — the issue body and your answer to (5) say you call #restart after each #go_to, (6) says a new browser per job with #reset — so tell me which one the 490 sockets came from.

    I got an idea. I will call #quit instead of #restart and let us see what happened.

    I do have two further findings, one a real bug in #quit itself and one in your browser_options, but they are both smaller than this and impossible to measure underneath it. Let's get the #quit in first.

    But in any case, you have to implement the proper finalisation in your code for the case the application that uses the gem did not call #quit. Additional information about how to do that I send in the attachment

    ruby-gem-finalizer-cleanup-summary.txt

  12. route commented on Sep 28, 2026

    @route
    Member

    I think there's a confusion here on your side!

    in a job I instantiate a new @ferrum and after each @page.go_to I send #reset

    Then the accounting is expected, not a bug. #reset is not a teardown: it disposes browser contexts and nothing else. It does not close the connection and it does not stop the browser. A Ferrum::Browser you create and never #quit holds its socket and its threads for the life of the process — and your process is a Que worker with a thread pool, so "the life of the process" is thousands of jobs.

    Running exactly that flow against main — your browser_options, create_page, go_to, reset, drop the browser, GC.start, six times:

    boot       fds=7   threads=1   chrome=0
    job #1     fds=8   threads=3   chrome=0
    job #2     fds=9   threads=5   chrome=0
    job #3     fds=10  threads=7   chrome=0
    job #4     fds=11  threads=9   chrome=0
    job #5     fds=12  threads=11  chrome=0
    job #6     fds=13  threads=13  chrome=0
    settled    fds=13  threads=13  chrome=0
    

    One socket and two threads per job, and they never come back — not on the next GC, not ever. The leftovers are the client's two subscriber threads, still parked on Queue#pop; a live thread is a GC root, and through its registered callbacks it keeps Client -> WebSocket -> @sock reachable, so the socket is never finalized. Chromium itself is killed, by the process finalizer — that is the chrome=0 column. So your Chromium processes come and go while the socket count only ever climbs, which is exactly the shape of your 876 socket= and 490 socket=. Six jobs cost six fds here; a pooled worker chewing through a queue gets you to 876 without any leak on our side.

    Quit every browser you create:

    begin
      # ...
    ensure
      @ferrum&.quit
    end

    The same loop with #quit is flat.

    The thread pool breaks SpaceCleaner too. It walks the Chromium trees under $$ and kills them — but $$ is the whole worker process, shared by every job thread in the pool, so it kills the browsers of jobs that are still running in sibling threads, and rm -rfs their /tmp/ferrum_user_data_dir* underneath them. That is where killed: 385 and ferrum: 55 come from: your own cleaner, not a Ferrum leak. Those jobs then hit a dead browser, drop the Browser object without quitting, and leak another socket each.

    Also worth noting your last report is evidence the process leak is fixed: top: ["22574"] has no 1 in it, where your earlier one did — no Chromium tree outlived the worker and got reparented to init.

    So: add the #quit, turn SpaceCleaner off.

    #quit what kills the browser process meaning that it's no longer usable
    #restart is quit + start new pid

    You are in a thread pool you have to kill the browser, where instead you were doing either reset or restart none of which applicable in a pool. You can't pass browser instance like that between threads.

  13. dima4p commented on Sep 28, 2026

    @dima4p
    Author

    I think there's a confusion here on your side!

    #reset is used in capybara tests, when you need to re-use your browser for next test, this essentially wipes all the pages but browser stays #quit what kills the browser process meaning that it's no longer usable #restart is quit + start new pid

    You are in a thread pool you have to kill the browser, where instead you were doing either reset or restart none of which applicable in a thread pool. You can't pass browser instance like that between threads.

    Initially I did not call #restart nor #quit and got a lot of open files and running chromium instances. I was sure that when I release my instance of ferrum it cleans all but it turned out to be wrong. And this is from my point of view the bug in the gem. So I ask you to fix it. How to do it I explained. When an instance is released it should clean the space. Tomorrow I'll tell you if #quit works, but you should not require all the users to do that and take care to remove not more required staff the gem created.

  14. route commented on Sep 28, 2026

    @route
    Member

    Oh please! Kick your LLM to make it work hard and shed your light on us with an open PR.
    I don't have time to explain both simple reset/restart/quit and complex things like finalizers to someone if they can't grasp even the simple ones.

    I was sure that when I release my instance of ferrum it cleans all but it turned out to be wrong

    If you stated this from the beginning I would close it immediately w/o spending so much time on it.

  15. locked as too heated and limited conversation to collaborators on Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions