Installing earlyoom by default

[alosslessdev]

Can we add https://github.com/rfjakob/earlyoom as part of the default installation? And set it to terminate sigterm processes when only 5% of ram and swap are available? I had to do this because the default oom_killer wouldn't work on user space processes fast enough on Linux mint 22.1 cinnamon. This is also a well known limitation of linux desktop OSes so it would be nice if linux mint didn't suffer from this problem.

[Jeremy7701]

Don't want it. Don't even use swap.

free -m
               total        used        free      shared  buff/cache   available
Mem:           31969        2851       26318          20        3273       29118
Swap:              0           0           0

[AtomicRobotMan0101]

Curiously enough, a single users experience isn't entirely representative of the whole community.

Let's give OP some room and discuss the virtues of the proposal vs the current OOM process. It may lead to improvements especially in constrained systems.

[fosslover306]

Not every one has 32GB of ram, and some workloads and games still need or want more RAM to function optimally, and a single program can go crazy and use tons of RAM for nothing out of the blue, which is what earlyoom is for.

Program goes nuts and free ram is running rather short? just kill the largest process which is probably the problem, and if that don't work kill the largest again until free ram is recovered = System does not crash, and the user can keep doing whatever they were doing before, and if desired restart the crashed program to continue their work.

If that scene seems unlikely to you that's great, but if not and the system and user needs it in a emergency and it works why not include it?

[Heavygrass]

and some workloads and games still need or want more RAM to function optimally, and a single program can go crazy and use tons of RAM for nothing out of the blue

Some tasks like ffmpeg for 4K+ do need tons of RAM (especially with heavy settings), how does earlyoom gonna distinguish whether a process like ffmpeg "go crazy" or not?

[Heavygrass]

Not every one has 32GB of ram

I don't have anywhere close to 32GB of ram and still consider earlyoom harmful for a desktop.

[fosslover306]

Earlyoom's job is to kill the larget process to free up RAM and SWAP in a emergency, weather or not it is "crazy" is not it's concern, and the solusions that do are often to slow to react and still kill the wrong things.

[alosslessdev]

Even with 32 gb of RAM, I occasionally launch processes that easily eat it all up. I believe that this would address, even if not in a correct way, a very common complaint in Linux systems, making Linux mint more user friendly (apps closing instead of freezing the entire system) currently, the default OOM doesn't kick in if the process is launched in user space, or it does very slowly, taking several minutes which can be very frustrating and cause productivity loss when you have several programs open. I don't think programs have to lose data if they are sent SIGTERM first.

[fosslover306]

Here is the rest of the quote
"which are RAM hungry applications which if they crash or encounter major bugs or crashes will happily bring even my 32GB machine into a unresponsive state in less then 3 minutes, and If I don't notice and react immediately I have a very good chance of just watching the resource usage skyrocket waiting for the terminal to load, just so I can launch btop and hope my computer has enough RAM and CPU left to react to any of my inputs, when I had only 16GB of ram I had only 45 seconds or so to react."

I would also point out have both flightgear and luanti setup to work properly with no free RAM problems, unless something goes wrong and happily uses every bit of swap and RAM available, and swaping all that data tends to eat a lot of CPU to, so it is not just "RAM happy applications" that are the problem any process that goes full memory leak will happily use every scrap of RAM, swap, and maybe CPU(depending on the specs) and lockup the system which is the main point you are conveniently missing, instead you are jumping to "buy a better PC" which will only buy time for those with lots of RAM, and those with only 8 or 16GB, usally on older slower systems which are even more sensitive to runaway memory leaks are only screwed sooner, and on a machine with 32GB of RAM 930KB is not even .05% of available RAM making it harmless unless something goes horribly wrong.

In real life 911 exists for the extremely rare situation you need emergency assistance, following this logic it would be better to plan for perfect situations, and have no 911 to call because only an idiot would need 911.

I have seen no argument so far against this idea except for some people will wine about a 930KB RAM, 83KB of storage increase, so don't even offer it by default, or even mention in a blog post or something.

[cplittlfield]

This is very useful information. While we wait to see what the devs think about your proposal, why don't you put it in a tutorial on the Linux Mint Forums, so that users can benefit in the meantime? No harm if it gets superseded later by events.

[fosslover306]

How do I do that? besides it would only be "sudo apt-get earlyoom" ,or just open the software center, and search for earlyoom.

It's not hard at all if you know that earlyoom exists in the first place, and I don't have an account for the forums, it was painful enough to make this one on github.

OH you mean the tutorial section in the discussions tab, I just found it I didn't know there were any tutorials here.

[fosslover306]

What does ctrl+alt+num do exactly? And how do I select what i want to kill? Does it work when the computer is locked up, or is this just a fast way to open a task manager?

which gives me the perfect idea, just make a key combo to launch btop is seconds, thank you for the idea i always appricate new ways to save my computers from forced restarts.

[fosslover306]

what's "it"? I'm not running random process killing commands on my main pc, secondary pc, or server just for funsies, for all i know i'll be dropped into a terminal with a type this pid and hope for the best. Take a screen shot or something to both show that it works, and what i'm supposed to see after it is important to be technical and scientific right?

[AtomicRobotMan0101]

Ive watched the comments through the day, but there is one consideration that people are overlooking.

If a n00b or gamer were to experience this kind of runaway memory issue, the program would just crash violently after locking everything up for a time. How would they know where to look?

There is the DBUS GUI notifier and the log (plus earlyoom can execute a script for each process killed). Those would/could be useful to pass messages to users as to why their favourite game just vaporised.

I feel OPs initial proposal is worth consideration. It has merit in the form of making system management easier for new users - which can be terribly intimidating.

[fosslover306]

The kernel oom killer ignores user space for the most part, and yes it has been basicly abandoned, after all modern computers have so much RAM, and all linux distros have some sort of "this program is unresponsive (wait) or (end)" unfortantly the "end" part is correct becase it only sends an end command to avoid data loss or something, which means if a program is seeminly responsive, or is to screwed up to respond (assuming i'm even able to press the "end" button in the first place) then it won't do anything. I assume kernel maintaners belived those pop ups would work flawlessly so why would they update or modernize it if somone else can deal with the problem. And as such earlyoom was born to fill in the gap the kernel maintainers could not care less to fix, by actually killing the offending process outright.

Which means earlyoom would not count as bloat on account that it performs an important role, if you for some reason don't want to protect you then just remove it.
I have to remove a web of like 50 printer related packages just remove printer support both for security reasons, and the fact I can not afford a printer or even use one, and that "bloatware" takes at least 200MB of storage, and 10MB of RAM so if bloatware is the problem I have the solution.

[Jeremy7701]

Just noticed!
According to https://last9.io/blog/understanding-the-linux-oom-killer/
The System OOM killer is disabled if /proc/sys/vm/overcommit_memory is zero,
Which appears to be the case for my LMDE6 - and presumably also for Mint 22?

[fosslover306]

The kernel oom killer was not made to handle modern RAM amounts as of 2006- 2010 ish as far as I understand it, and was originally intended to protect 1998 PCs from kernel paniking, agian if my understanding is correct brodie robertson on youtube covers Linux and open source events, and one of the was on the kernel oom killer being almost useless from the view point of kernel maintaners, lost source code, hard to read code and documention, and considering just scraping it altogether, disabling it to avoid further problems, or rewritting it. If memory serves they went with just disabling to avoid kernel bugs. I'll see if I can find that video.

[Heavygrass]

and one of the was on the kernel oom killer being almost useless from the view point of kernel maintaners

Even if all you're saying here is correct, this would be a kernel level issue that needs to be handled at the kernel level (one can look at Android e.g. on how to handle it).
Introducing a userspace hack to handle a kernel level issue is just architecturally wrong.
One is free to install whatever on its own, but it should be clearly optional.

[fosslover306]

They have no plans to fix it as far as know, but it still kinda works so it must be called manually because other wise it apparently kills and restarts stuff when there is no problems with RAM or SWAP, causing more problems and crashes then it solves while still being to slow to react to actual emergencies.

Meaning using said "hack" is the only thing coming to your rescue automatically, while the "real one" sits doing absolutely nothing.

So earlyoom is as much as a hack as replacing a broken fan with a new one that actually works.

On that note oomd does something similar but either kills way to aggressively and/or ignores SWAP entirely meaning if your RAM is "low" but there is still plenty of SWAP boom down goes your graphical session, or a whole web browser, if the kernel oom is enabled then it just made their system hang. Only earlyoom kept the graphical session working, and only killed some heavy browser tabs. earlyoom, oomd, and the kernel oom were all at default settings.

The reddit post I read (the search term was "oomd vrs earlyoom" is here: https://www.reddit.com/r/Ubuntu/comments/vl2ak0/personal_observations_on_systemdoomd_vs_earlyoom/

[Heavygrass]

Do you guys ever run long/heavy tasks like video encoding etc?
You know how fun when you run some task for 24 hours and more, and then OOM kills the process?
Try e.g. doing some long h265 video encoding on a phone (with termux on android e.g.) and see how fun it is. And Android actually has many settings to configure this.
We don't want any kind of aggressive oom on a desktop/laptop.

[Heavygrass]

And considering at that point your computer would be on the verge of a complete lockup and need a forced shutdown

Somehow I never encountered a complete lockup yet (just in case I'm not claiming its impossible).
I had uptimes for several months.
Do you have a reproducible case?
Did you report the issue upstream?
Do you have swap enabled? What size?

Not saying you need to upgrade anything but it seems what you are rendering videos on is not up to the task at the resolution or frame rate your using, or you may need to optimize your rendering rig a bit more.

Sorry I don't even understand what this has to do with my post?
Please reread my response.

[Heavygrass]

after your neighbor called them in when they saw smoke pouring from your bedroom window.

I don't consider analogy a scientific method at all. Please try to stay technical if possible.

[fosslover306]

My point was it is not aggressive at all, if your computer was a both RAM and SWAP stressed enough for earlyoom at it's default setting to kick in your computer is already dangerously close to system lockup, and your video render was already probably doomed to begin with.

And if you want the technical exclamation, your computer and your video render were already screwed, but if the largest process (probably the video render) is killed at least you didn't have a froze up PC waiting for you at home forcing a forced shutdown which risks both data loss and file system corruption, just optimize your system some more of you hadn't already by closing other programs or something and try again and deal with the lost time.

I already mentioned Flightgear and Luanti so I have dealt with a locked up PC before, even with a video render on a low speck laptop and forcing it to shutdown which corrupted it's ssd so I almost lost everything on it including my local media collection, I had to run a fsck to fix it, which was not fun and I still had to restart the video render (it was 720P) and it needed another 8 hours to finish. It had no oom protection installed, or at least it did not kick in.

And as for "Not saying you need to upgrade anything but it seems what you are rendering videos on is not up to the task at the resolution or frame rate your using, or you may need to optimize your rendering rig a bit more." You were complaining it just might ruin a 24 hour video render, and as I said given the conditions needed for it to be killed you would probably have other worse problems anyway, and again removing it would be a one line or one click option depending on how you prefer to manage your packages.

And optimizing your system would entail closing background programs, removing autostart programs, and turning off automatic updates to ensure nothing spikes resource usage during your hypothetical video render, also from what I have both heard and experienced video renders just fail sometimes due to random bugs or glitches even with low resource usage and nothing to kill it. And yes I have SWAP enabled I use zram so I can have a 15.7GB SWAP file in 32GB of RAM.

And what's the point in reporting an issue no one really seems to care enough to do anything most of the time, and I did not have a github account or care enough to make one, never mind the writing, socialization, or communication skills to report anything, and didn't know why it crashed anyway that's why I called it a "random bug" how do you even report a hard to recreate and unknown bug anyway?

[Heavygrass]

You were complaining it just might ruin a 24 hour video render,

Don't you realize that was just one of many examples?
What do you do with PC?
Another example is 3D rendering.
Simply killing the most heavy process is just nonsense. Even Android do not do that (at least it not that simple).

your computer and your video render were already screwed,

There's ALWAYS a task that will take say 12+ hours no matter how big the machine is. Weird that this needs to even be talked about.
Take software AV1 encoding e.g (and there's h266 coming).

[fosslover306]

Gaming, flightsims, file management, media server, medicine managment, reading, repying on github, read, watch moves, lissen to music, document organizing, experimenting in powder toy, downloading files, take backups, money managment, ocasional video rendering when needed, and troubleshooting when stuff goes eneviably wrong. Over 3 different computers.

In short most things I do involve a computer in some way, execpt for most social media of course.

So a computer crash and/or lockups give me litteral nightmares, only beat by the nightmares of something bad happening to my loved ones.
So earlyoom is no brainer for me.

And unfortanly other oomkillers do no better with their fancy inteligence like killing the graphical session because chrome ate to much RAM, or hanging up the system while the conciel convenes, and the system completly locks up before any decions are made . You did read the reddit post at the link right?

also you can configure earlyoom to not kill certain processes and even change the value it starts killing things at, and it can also be scripted to tell the user what happened, or be removed with no strings atached.

You gave an example I responed with a example solution, you respond with example problems, and i respond with example solutions. And as the cycle goes on we learn more about eachothers view points and concerns.

[Jeremy7701]

Have you considered making any install of earlyoom flash a BIG message on the screen saying "You are about to install software that may terminate user programs without warning".

If you have "(only 10% RAM, and 5% swap left)" left, Linux may be happily maximising its RAM usage for buffers etc when this earlyoom launches and terminates a vital program.

Since the ideal killee is the largest process, it seems likely that it is a rather important one.

[fosslover306]

well the biggest memory using process is probably also the problem, and if not it will rapidly become it anyway.

A crashed program is anything from inconvenient to annoying, but a system lockup takes longer to fix, risks data corruption and data loss if you force shutdown, and then you have to reopen everything you were doing before, and if a system update is happening and your system locks up due to a browser crash then you really start to sweat, particularly if a new kernel was in the update list.

[AtomicRobotMan0101]

I think we need to calm down a bit. Lets stick to the problem.

From my experience, an earlyoom would have been an absolute god-send on many occasions.

- e.g 1: I was the maintainer of the old Planet Mirror system. It was a colossus back in the day. Our data centre (our own!) was RAMMED with 1U/2U servers (forward proxies), some 4U bulk management machines and endless storage arrays.

We had many situations where the core 4U's were overrun with traffic. It was hectic. There were a few nightmare situations where one of the two core management units were saturated. Our only choice was a hard restart and it ALWAYS ended in grief. It was a diabolical situation. Earlyoom would have saved us DAYS of fixups.

- e.g 2: I was CTO of a childrens maths competition. It was a worldwide event involving millions of children all competing on a single day. The traffic was INSANE. The databases were smashed, the front-end units were smashed and the data management servers were smashed. Yes, this was before the cloud became a thing.... but earlyoom would have been an absolute saviour on many occasions.

Sometimes things are bigger than a single users "video encoding" crashes. Mint is used in a myriad of situations, not just home PCs'.

Another example, I consult on Linux issues to business and the number of times an office PCs will seize up because some numbskull did something sublimely stupid is beyond count. Earlyoom would save businesses hundreds of hours in lost production, frustration and IT callouts.

My suggestions are these:

1. If a process is going to lock up a system and inevitably crash it, why not kill the offending process anyway? Whats the harm?
2. If its such a be-all requirement that "my video encoder" uses 99.98% of all resources, one can always create a regex in the earlyoom config to exclude it from being removed (I assume those who are the opponents have READ THE DOCO?)
3. If opponents are worried about n00bs experiencing random crashes, it would be trivial to report these crashes to the Mint Cinnamon Log, or even pop up a message in the [email protected] Alert tool. Wouldn't n00bs be happy that their system properly handled an out-of-control process rather than the disaster of a locked up computer????
4. Advise people that earlyoom is installed. One can remove it trivially, the same as removing any other tool that offends those who are sensitive to such impositions.

[Heavygrass]

1. If a process is going to lock up a system and inevitably crash it

There's a lot of assumptions in this statement. From the responses it appears there's not a single reproducible case yet of "complete locking up".
Furthermore it appears that by "complete locking up" some actually mean only GUI lock up which is not the same.

[fosslover306]

A "offending process" is the one using most of resources in a RAM and SWAP stressed system, android puts apps to sleep to save CPU power and RAM because it is a mobile phone OS, earlyoom does not react for no reason, the system is almost out of RAM and will lockup at any moment, something needs to go and the biggest target is the best one to fix things.

A "complete lockup" is when the computer becomes impossible to interact with, and even if you tried to do something nothing seems to happen.

As said above ether A) remove earlyoom. B) add the the super important process to the regex so it will be ingnored. C) set the trigger values to lower values so they kick in later. Are you gojng to explain your fancy key command you say will fix every thing with out any automatic help?

[Heavygrass]

Are you gojng to explain your fancy key command you say will fix every thing with out any automatic help?

1. I never said that.
2. You'll have to learn it yourself, there's nothing special about it and there's plenty of info online.
3. Please change the tone

[fosslover306]

What tone? Just take a screen shot to show me what this looks like if not then i'm moving on from this.

"I don't consider analogy a scientific method at all. Please try to stay technical if possible." so i'm asking for a "technical" screenshot, or does this only work if my system is RAM stressed or something?.

[Jeremy7701]

Yes, probably time to make a note of "earlyoom", check if it gets installed and take the appropriate action.
There are two diametrically opposed opinions of this software.

So I'm out of here!

[cplittlfield]

One assumption not being examined here is the matter of "locking up" the system. The system is not supposed to lock up when it runs out of RAM. At the very least, applications that try to allocate RAM should get an error from the system and report that to the user. In other words, things are supposed to fail gracefully.

What I am driving at here is that, if the system is actually put in a state where it cannot be used, then there is a serious problem with memory being overwritten, and that, I should think, is a kernel bug to be addressed. If this is a reproducible situation, then it should be reported.

I see two possibilities: first, the kernel is messing up RAM itself and putting itself in an inconsistent state. This is not good, and should be addressed at the kernel level. Second, an app is messing up so badly it is putting the kernel in an inconsistent state. This is not supposed to be able to happen, so all the more reason for it to be addressed at the kernel level.

So these are the considerations that have been coming to mind as I've been watching this thread. What do you all think? Am I missing something? Is there more to the picture?

[fosslover306]

A system that has lost interactivity via a gui is for most users, noobs, and the general population is considered "locked up" "frozen" or "stuck", and since most people are using a gui, and the ones that don't have probably 20 ways to "fix it" with some quick typing.

GUI's are a lot more sensitive to RAM and SWAP running out on account of all the stuff that has to work to start something, and that's assuming the program causing problems is not blocking or stealing the mouse focus or key commands. I usually try to open a terminal asap to launch btop to kill the problem process via control+alt+t but that is often blocked by something else going on, or moving the mouse triggers a animation or action that was pushed out of RAM and SWAP and now has to load to do anything else, only making the unresponsiveness worse. In short I do not care what the specifics are if the kernel is panicked, or the gui is frozen because if i can't kill the problem process manually a forced restart is necessary to use my computer again.

And as for why a complicated large program can crash and eat all the memory, well it is rather easy when nothing comes into play to stop it from happening. Earlyoom or other oom killers (I provided a link up above to a reddit post comparing earlyoom, oomd, and the kernels oom killer) and the user using a GUI is probably the first to be blocked out in a matter of speaking, so a very tiny already loaded daemon that can react quickly and automatically is basically the last line of defense unless you are a expert programmer or something else.

And as for why they crash in the first place it is usually because of a random code error in a piece of the very large code base, often is hard or impossible to reasonably recreate, or recreating it takes the whole machine down again so not way to dig deeper into the problem anyway, however if a plane model in Flightgear has caused it 3 times In a row I remove it and never aim to use it again, and I of course have btop open to kill it immediately before the system lockup occurs.

Modern computers are doing more then ever before, and different programs are interacting to do important things at a ever increasing rate, which is a grate recipe for something to go wrong but despite this Linux systems in general do not have any protection against it, or they are way to slow. Windows XP would automatically kill programs to save the operating system in 2008. And apparently fedora switched to earlyoom years ago to avoid these very problems in corporate systems, and android has done it since android 2.3.5 which I happen to use as a alarm clock at the moment, which released was 15 years ago.

It is time to get with the times and actually have some sort of useful oom service, the mint devs don't even need to do anything to make this improvement just add it as a recommended dependency, or give it a as an option in the getting started tool which also promts the user to install codecs, and set the system theme.

[cplittlfield]

What I am driving at here is that, if the system is actually put in a state where it cannot be used, then there is a serious problem with memory being overwritten, and that, I should think, is a kernel bug to be addressed. If this is a reproducible situation, then it should be reported to the kernel devs and to the application devs.

This includes submitting a bug report for the kernel oomkiller, if it is not working properly.

Unfortunately, I cannot seem to stress my system sufficiently, even though I only have 12 GB of RAM. In fact, yesterday was the first time in 2.5 years that I have noticed swap actually being used. Till now, swap use has been 0 every time I've checked.

[fosslover306]

How do I Report "if I run out of ram your oomkiller does not work, I have waited 45 minutes before to give the system time to fix it's self, and no dice and usually wait 10-15 minutes and by defualt and no dice I'm blocked out and nothing came back" and according to > Just noticed! According to https://last9.io/blog/understanding-the-linux-oom-killer/ The System OOM killer is disabled if /proc/sys/vm/overcommit_memory is zero, Which appears to be the case for my LMDE6 - and presumably also for Mint 22?

If it is true in Debian and given the lack of oom killer appearances when it would be needed indicates, That would make a bug report even more meaningless, I do not like screwing around with kernel parameters, so I won't go changing things that in the case it was disabled witch was probably for a reason.

"tail /dev/zero" is what test case was and it worked perfectly with earlyoom installed and enabled for @AtomicRobotMan0101, so personal experience is good enough for me. If you want to replicate my problem and do a apples vs apples test, one run without earlyoom and default kernel control values, and one run with earlyoom enabled also at default settings which actually seems to be 5% RAM, 5% SWAP. run the heavest applications you have while running the above command and just see what happens?

[cplittlfield]

You use the kernel.org bug reporting mechanism to report problems with RAM being overwritten and with the oomkiller not working properly. The devs of whatever application is causing the problem should also be notified of a bug in their software.

I'm sorry, but I can't be more helpful than this.

[AtomicRobotMan0101]

Another thought occurred to me while mulling this debate over lunch.

If this OOM thing weren't a problem, across all Linuxen in all forms, why did "the they" bother to put an OOM killer into the kernel?

i.e. its a real, persistent problem. Rare, thankfully, but it exists.

One counterargument goes along the lines of "report the bug" but it has been shown over the decades that bugs can be obscure, no one cares, it affects only a minority, transient, or there is no real fix. Kernel issues triply so.

Having an OOM manager in place mitigates against turning ones computer into a useless brick. The cost of this management option is non-existent.

The argument of where it belongs (kernel vs. Userspace) is also specious, for the only question is "does it work as intended".

The answer is a categoric YES.

[Fantu]

I thought I remembered that earlyoom didn't clear cache and buffer before doing kills but checking again it seems that it does.
Cache/buffer can be large even for things that are no longer needed, such as a large file move/copy. Having to deal with earlyoom kills when you have large amounts of easily freed up unnecessary RAM is very annoying. If earlyoom clear cache/buffer before a kill, the situation would be significantly different.

Setting which processes to avoid and which to prefer is also very useful on workstations and desktops. This is one of the reasons why in many cases, adding it by default leads to a worsening, while it would be better to set it manually based on the programs you use, the amount of RAM/swap, the type of swap, etc.

With large amounts of RAM (such as 32 GB of RAM) 5% RAM and swap is a fairly large value so it could have a good chance of not incurring risks (except for example processes that have memory leaks)

I did a fast look to fedora package and change the default to:
EARLYOOM_ARGS="-r 0 -m 4 -M 409600 --prefer '^(Web Content|Isolated Web Co)$' --avoid '^(dnf|packagekitd|gnome-shell|gnome-session-c|gnome-session-b|lightdm|sddm|sddm-helper|gdm|gdm-wayland-ses|gdm-session-wor|gdm-x-session|Xorg|Xwayland|systemd|systemd-logind|dbus-daemon|dbus-broker|cinnamon|cinnamon-sessio|kwin_x11|kwin_wayland|plasmashell|ksmserver|plasma_session|startplasma-way|sway|i3|xfce4-session|mate-session|marco|lxqt-session|openbox|cryptsetup)$'"
https://src.fedoraproject.org/rpms/earlyoom/blob/rawhide/f/earlyoom.conf

and this already seems better to me for some situations, so if it is installed by default I think it is very important to improve the default values, Debian package don't seems to improve the default value, I don't have time to search a possible good default value and propose MR, but if someone experienced enough and with time wants it I think it would be a good start
https://salsa.debian.org/yangfl-guest/earlyoom/-/tree/master/debian?ref_type=heads

[AtomicRobotMan0101]

There also a strange sort of irony here with this discussion, noobs and normal users who have the least memory management experience are expected to be sysadmins with terminal commands memorised to fix the problem, and the experienced sysadmins and users who can run sudo apt remove earlyoom, but can also manage memory themselves can't seemingly see the easy solution, or edit the configs themselves.

Am I the only one noticing this pattern? And I'm not upset just disappointed how little new users are valued in this discussion, for improvement ideas in a BEGINNER focused distro.

You have absolutely identified the dichotomy.

I read this this morning (I'm in Aus) and it has had me thinking during the day.

My thoughts:

• This forum is absolutely the right place to raise these issues
• n00bs should not be discouraged from the discussion, ever, for just like kids they have the strangest way (sometimes) of asking the most profound questions that stop us as adults. It would be ill-advised to diminish this valuable foil (though, reading this beforehand would be wise: https://linuxsilo.net/docs/smart-questions_en.html )
• This said, I believe it is very important that everyone looks past their own needs and considers the broader community. We all do many different things, one must remember this. We need to think of ALL people and obtain the highest benefit.
• The "MY specific use doesn't need it, therefore it should not be included" isn't a good place to start.

Unbelievably, I had this very discussions situation with a user today! Unbelievable coincidence. They started a fresh user profile on Linux Mint, with Firefox they'd started their sync and copied down all their bookmarks. To populate the favicons (the little thumbnails) they had gone to the bookmark manager and said "open all".

Thats the right thing to do... but not for several hundred!

It locked that high powered PC up tighter than a misers wallet! It locked the mouse, the machine was heaving like Fukushima and utterly unresponsive. It required a hard restart - the disk needed checking and the Firefox needed fixing.

So, had the machine had earlyoom, it would have prevented a potential disaster.

So, a very real and practical example of exactly what does occur in The Real World.

[Fantu]

What you've encountered is a bug in Firefox, perhaps similar to one fixed in the past ( https://bugzilla.mozilla.org/show_bug.cgi?id=978932 ), and should be reported.
earlyoom can be a "workaround" for cases like this (but I reiterate that it must be set up well before enable by default to reduce the cases where it would cause more problems than workaround), but a solution is better, at least in open source software with a higher chance of solving it, while unfortunately when it happens with proprietary software (especially when you don't have paid support and paying for the license often doesn't guarantee support) unfortunately you run into many more problems and you try to survive with workarounds.

[fosslover306]

Mozila no longer wants to be browser company, they now intend to be a group of ativists they siad in thier own blogpost, and the change lists have gone fron 3 pages, down to half a page and it's mainly security and bug fixes which while great and all but shows they are going to leave Firefox to rot in the near future.

Besides that problem, I have accientaly opened all bookmarks in a folder, it had 1,500 bookmarks in I and it would have opened all of them but I killed firefox just before it locked my laptop, and I know how to manage memory and watch RAM and CPU usage like a hawk, the first time it happed tho it did take my laptop down, it was only 15 tabs but they were very heavy tabs, and it had only 4GB of RAM to play with, and that was 3 years ago.

So that is defnitly not a bug, with the current state of most linux distros "opening all bookmarks" is the most dangerus option in Firefox.

And yes earlyoom won't protect your machine if it's not installed that's why I support it being installed by default, or at least an option during setup, and you are right proitary software, and program bugs in general can take your system down, so a "workaround" that is done in every single mainstream OS, like android since v2.3.5, windows xp, mac os, ios, and every version of windows after xp is done everywhere else is expected at this point.

Linux is sold to windows user with "it never crashes" as the main point, and that is 99% true unless you count running out of ram anyway then all bets are off once thier new system which they are still learning locks up on them with no warning or cause given. A popup letting the user know what happened and why is going to be far better expeince for everybody who does not want to make system management thier hobby.

Normal PC users treat their computers like a tool, not grampas old vase, and that is the reality of how the computing world has been for the last 10 years, a solution is needed that does not point fingers, sends people on a wild goose chase, or anything else, and leave it to the advanced nerds to remove said protection or configure it if they wish.

[fosslover306]

Agian if earlyoom was installed by default, a custom config could easyly be used, but considering this disscussion is still stuck at the question in the tittle of this disscussion, and yet to get to the "how do tell users what happened" or "how should it get configured" I would love to move on to how to fix the shortlist of cons.

[GitHubIsToxic]

I acknowledge the long history of the Linux OOM problem, and have experienced it many times.

IMO: an OS that cannot gracefully handle OOM is literally not an "Operating System".

So, I still use the hard reset button on my system occasionally for this very reason. Software should never crash the hardware.
(Semi-relevant: you can render a Mint box into a brick if the disk is full, just the same!)

[AtomicRobotMan0101]

Ive been using it now on three systems as a test. One is a media+gaming system, the other is a NAS (an Intel NUC) which runs multiple VMs and a Laptop (old and busted).

All are behaving beautifully.

In addition, I was a fan of ZRAM for a long time, but dropped it in favour of ZSWAP. Ive since flipped back to ZRAM and paired it with the EarlyOOM. It has been P E R F E C T.

Given ZRAM is used in Android and earlyoom saves the day on out of control processes, it seems like a great combo.

[Fantu]

Using zram or zswap can be useful for storing more data in RAM (thanks to compression), reducing disk usage (with swap, I mean), and in some cases even reducing OOM occurrences. However, using them isn't optimal with very limited CPU capacity or if high loads with low latency are required.
It might be helpful to create documentation about them to make users aware of them and understand if they need them and how to enable and configure them. It would also be a good idea to integrate this into the installation or post-installation process, and do the same for earlyroom.

[Fantu]

Has anyone tried nohang which seems to have more features and more customizability compared to earlyoom?

[Fantu]

Copied from my old msg:

I did a fast look to fedora package and change the default to:
EARLYOOM_ARGS="-r 0 -m 4 -M 409600 --prefer '^(Web Content|Isolated Web Co)$' --avoid '^(dnf|packagekitd|gnome-shell|gnome-session-c|gnome-session-b|lightdm|sddm|sddm-helper|gdm|gdm-wayland-ses|gdm-session-wor|gdm-x-session|Xorg|Xwayland|systemd|systemd-logind|dbus-daemon|dbus-broker|cinnamon|cinnamon-sessio|kwin_x11|kwin_wayland|plasmashell|ksmserver|plasma_session|startplasma-way|sway|i3|xfce4-session|mate-session|marco|lxqt-session|openbox|cryptsetup)$'"
https://src.fedoraproject.org/rpms/earlyoom/blob/rawhide/f/earlyoom.conf

and this already seems better to me for some situations, so if it is installed by default I think it is very important to improve the default values, Debian package don't seems to improve the default value, I don't have time to search a possible good default value and propose MR, but if someone experienced enough and with time wants it I think it would be a good start
https://salsa.debian.org/yangfl-guest/earlyoom/-/tree/master/debian?ref_type=heads

Start from Fedora default and replace Fedora specific critical process with the Debian ones I think can be a good start.

[fosslover306]

So far I have set both the memory and swap values to 5, and only protected systemd, package management, and login related processes in order to make a mostly universal config, and no prefer flags have been set in order to not kill someones web browser over a single unrelated game or something that gernaded, and I have also learned that earlyoom has it's process taged to never be pushed out of ram so it can work during a ram crisis, and set the memory report to system log to 5 hours instead of every 1 hour to reduce log entries.

[fosslover306]

Based on feed back so far i will add back most of the avoid tags, except the fedora ones and upload the new config soon.

[fosslover306]

I put all of your --avoid aruments back in, removed dnf, and added captain(the new tool that the mint team has built, and will use more in the future.

[Fantu]

you need to see which other essential processes do not have oom_score_adj = -1000 (which if I'm not mistaken earlyoom respects, barring a bug) and add them to those to avoid
if I'm not mistaken these are all relating Debian packages managements and they don't have oom_score_adj = -1000:
apt
apt-get
aptitude
dpkg
unattended-upgrade
synaptic

[AtomicRobotMan0101]

I have a story that might help relate the the issue at hand.

I've been asked to re-purpose some old iMacs that were used industrially in a busy industrial studio (dust, noise, horror, etc). They ran mostly Chrome and AutoCAD for the CNCs and other whizzbangery. The lads were grizzling about their speed, so I was asked to convert them to Linux.... well, Mint was the obvious choice (obviously!)

They are a hodge-podge of 4/8GB with spinning rust disks. Some fairly old gear. Mint, despite their scepticism it would work now makes them fly. Happy chaps now. They'll be happier when I plug the USB3 NVMes (older 128's bought real cheap) in the back to boot off rather than the rusty spinners.

I thought to be cheeky and put EarlyOOM on them as they essentially POS's. Apple makes nice hardware but they suck at an upgrade path, so extra RAM is a challenge. They wanted to keep running CAD and some industrial machine control programs, so I thought the EarlyOOM with ZRAM would be a good choice.

It has proven to be so!

It has captured a few events that these ham-fisted louts have done that would have locked the machines tight AF. They popped them, asked me to look at why (as I told them to tell me about crashes as part of the "tuning process") and it has allowed me to give them all workarounds, one problem at a time.

The Boss wrote to me today (a Sunday) and said how happy everyone was with the new "upgrades". It didn't cost the biz a cent!

Happy workers = better stuff out the door.

So, I now think that EarlyOOM SHOULD be a default install.

My only consideration would be a good method of ensuring end users :

1. get a pop up saying why their program fried itself (and not just disappear)
2. ensure the logs tool captures sufficient detail
3. a little GUI tool (an applet would suffice) to set the various knobs

[Fantu]

the notification must be functional in the program itself and not look for something external that may have more limitations or unexpected events

[fosslover306]

That is not how that works, you can have it notify the user via dbus-notify but unless I am missing something that package in not available for me, so a script would need to be used instead witch uses standard calls anyway.

[Fantu]

I hadn't looked carefully, is this what would be useful but it's not currently available on Debian? https://github.com/rfjakob/systembus-notify
In addition to a better default configuration, having notifications working would be a great thing to improve earlyoom and increase its use.

[fosslover306]

To answer your question if "systembus-notify" is available, the answer is no on lmde 6 anyway so a script will need to be written, and appearently the developer knows that because that is not enabled by default.

[fosslover306]

Here is the second version
earlyoom config v2.zip

Please feel free to read it, and comment on what you think would improve things, beside the script variable which is I have already planed to add as soon as I have a script to add.