macOS 13.5 no longer allows setting system wide ulimits

[kevincox]

The default limit is in place because select() uses a fixed size bitmap for file descriptors. If a program that uses select opens more than that many files it will make an out of bounds write. It is probably better to make the file descriptor allocation fail than have memory corruption.

All programs that don't use select() should raise the limit to the hard limit on startup. Then drop it back down before exec()ing another program. It is a silly dance but that is the cost of legacy. I'm surprised that these programs that are affected can't add this dance in.

[arghwhat]

I am fairly confident that this is not the reason for the file descriptor limit, especially since select was superseded over 3 decades ago by by poll (1987, picked up by Linux in 1997), which in turn is superseded locally by kqueue. Use of select in modern times is a bug.

ulimits are quotas that disallow excessive resource consumption by an application, not bug shields.

[crest]

Select has 1024 bit/file descriptor bitmap and anyone using it gets what they deserve. MacOS also provides poll() as a slightly less braindead, but still POSIX option. The proper solution is to use kqueue()/kevent() allowing userspace processes to efficiently get notified about changes to tens of thousands of file descriptors (tracking a million file descriptors using this API on FreeBSD works fine).

[saagarjha]

Are you sure it's that, and not people setting ulimit to something very low and causing critical programs to start failing in ways that open up security issues?

[hashhar]

The reasonable solution for that would be to fail calls to select() if more than FD_SETSIZE fds are being held instead of nerfing all applications to do the setrlimit dance - some of which may not be actively maintained or even if they are would take time for fixes to be available and distributed.

[StewardMcOy]

This seems like an unfortunate, but reasonable, given the circumstances, approach, but you'd have to make sure that none of your dependencies, including Apple's frameworks, have code that calls select.

[crest]

At least it's only the soft limit. The hard limit isn't reduced. The simplest workaround is to wrap the start of the of process with a shell script raising the soft limit before it execs into the application you want to run.

[c0l0]

Let's just hope Apple will not alter the deal any further, right? [-:

[Angostura]

The error message indicates that "Operation not permitted while System Integrity Protection is engaged".

.... and if SIP is turned off?

[saagarjha]

It works as expected.

  $ sudo launchctl limit maxfiles
   maxfiles    256            unlimited
  $ sudo launchctl limit maxfiles 65536 200000
  $ sudo launchctl limit maxfiles
   maxfiles    65536          200000

[sgjohnson]

> .... and if SIP is turned off?

You lose Touch ID & Apple Pay

[c-hendricks]

I've ... always used ulimit on macOS, going back almost a decade. Never heard of using `launchctl` to change it.

https://www.google.com/search?q=ulimit+mac+site%3Astackoverf...

vs

https://www.google.com/search?q=%22launchctl+limit+maxfiles%...

[henvic]

Globals are a nightmare. Despite this being something really common to do for developers and something that will cause a pain for a lot of early adopters of [whatever software that requires it], I see this as a great move!

[crest]

Do you really consider 256 file descriptors a sane resource limit on processes?

[semireg]

As an aside, Quinn “The Eskimo!” is a legend. They have helped me on a few occasions with code signing electron apps. In the American south there is a saying “doing the lords work.” Thank goodness we have people like Quinn. Their understanding of complex system and abilities to troubleshoot are invaluable.

[dinkblam]

also, his posts on the Apple Developer Forums are also much more professional and insightful than the crap which Apple calls their documentation.

[coldcode]

I worked with Quinn at Apple during my brief stay in the mid-90s. He's amazingly knowledgeable about almost everything in the Apple world, which is why he is so valuable at answering questions.

[jamesfmilne]

Quinn's been around forever, you can see his name in Apple Developer Connection CDs from the 90s.

http://preserve.mactech.com/articles/mactech/Vol.16/16.06/Ju...

[e61133e3]

Agree! Quinn “The Eskimo!” is indeed a legend. Hope to meet him one day during WWDC. Just to shake his hands.

[roblh]

I am, at this exact moment, getting bodied by a ulimit problem on mac. Apparently with pacman, directories attached with bind mounts have a fixed ulimit of 64 internally and running npm install inside a mounted project explodes completely because of it. Funny that this turns up right now, even if it’s not a fix for my particular problem.

[CodesInChaos]

I think a security option that does the following would make sense:

1. exec should reduce the hard-limit to FD_SETSIZE unless it's passed a flag

2. When a process calls `select` while having a hard-limit > FD_SETSIZE, it gets terminated. (Strictly speaking this doesn't prevent the memory corruption, since it happens in select's support macros, not select itself. But I'd expect it to be good enough in practice)

A modern application which doesn't use `select` should raise its own hard-limit during startup, option into the select-termination in exchange for the ability to open many files.

[amelius]

Only superroot (i.e., Apple) can change it.

[baq]

I've been saying for as long as I've got a macbook pro that the hardware is absolutely above anything available from other vendors but macOS is crap for development. Interesting to know it actively and intentionally becomes even worse :/

[solarkraft]

I still don't know a better plattform. Linux may be fine if you like tinkering and can compromise on UX and Windows is the same but worse (but at least with Linux built in now).

[pjmlp]

Depends what one understands by development.

For many of us it isn't classical UNIX CLI and daemons, stuck on 1970's programming model.

[gorjusborg]

Yep. Great hardware and software integration too (lower level software like device drivers and power management).

MacOS as a user interface is an abomination with pretty bows on it.

[behnamoh]

Apple is notorious for breaking things in their APIs, macOS, etc. It's just not a stable ecosystem to develop for (or in this case, to develop in).

I remember when they removed HiDPI support for <4K monitors. Such a ridiculous move by Apple...

[fouc]

use & support asahi linux

[anta40]

>> macOS is crap for development.

I assume you don't do mobile app development?

[gregoriol]

If you want a better macOs, just disable SIP

[hospitalJail]

> I've got a macbook pro that the hardware is absolutely above anything available from other vendors

How can you say this? They don't have CUDA. Right now M$ + Nvidia have the market completely cornered :'(

Linux + Nvidia suck btw. The only option is M$.

[dinkblam]

a much worse recent change is documented here:

https://www.mothersruin.com/software/SuspiciousPackage/faq.h...

basically no experienced Mac user would ever open a .PKG package with the Installer, but rather open it with a safe inspection app (Suspicious Package, Pacifist) first, either for security inspection of the pre- & post-install scripts, or to do a manual install outright to prevent the security nightmare that actually installing a package is.

while inspecting and/or manually installing packages is much safer than installing packages with the installer, it is now effectively outlawed by Apple. continuing to do things in the safe way now takes a lot more time...

[zarzavat]

Just turn off SIP. SIP is for regular users who don’t know what a ulimit is, the whole point of SIP is to lock down the OS as much as possible.

If you’re a developer, you live in the Terminal, you obviously need full control over your OS.

edit: I appreciate the irony of being downvoted for suggesting having control over your OS on Hacker News, so keep it coming please. Mo’ downvotes mo’ irony.

[sspiff]

I'm a software developer at a company where ~50% of the staff is a developer and our IT fleet management enforces SIP. This simply isn't an option for us because of security requirements from our customers.

[DannyBee]

I would suggest you likely aren't being downvote because you suggest having control, instead, you are being downvoted because you seem to believe that turning off SIP is like a normal thing that people should be doing regularly. I think you are finding strong disagreement with that.

People probably don't believe they should have to do that to perform regular development work.

[_sojh]

You aren’t being downvoted for telling people to have full control over the OS. You can do that with SIP enabled, or boot to recovery, disable, modify, enable, and have full control over your OS. How often are you needing to modify low level OS config that you’d rather make your entire machine vulnerable to root exploits than dance around SIP a couple times a year if that? That’s why you’re being downvoted, for advocating folks make their machine way less secure to save 3 minutes worth of reboot time a year, if that. Bump the hard limit once and you never need to touch it again.

[baq]

laughs in corporate IT

[CodesInChaos]

A user who understands fd limits will not raise it above FD_SETSIZE globally.

[saagarjha]

Turning off SIP allows for any user process to immediately gain root privileges. This is surprising to most people, so I generally would not recommend it without fully understanding the risks behind it.

[pierat]

So, who's computer is it REALLY when you can't make settings/decisions about the computer you bought? When some other entity, manufacturer included, is making decisions you cant change or reverse, sounds like they retain real ownership.

Its high time for the FTC to start busting fraudulent rentals misrepresented as a "sale".

(And anybody viewing the Apple ecosystem KNEW this anti-owner lockdown phone shit would eventually hit their laptop and desktop computers as well. But for some reason, the Apple fanatics are OK with this stuff. )

[donatj]

You own the hardware. You license the software. That relationship is pretty clear.