No more system dylibs on the filesystem; only loadable with dlopen (from shipped dyld cache). The title is a shocker that sounds like system libraries are no longer dynamically linked...
Edit: Since this comment is currently at the top, here's a better link to the source: https://developer.apple.com/documentation/macos-release-note... (using scroll-to-text fragment[1] since there's unfortunately no usable anchor to the specific entry; not supported in non-Chromium browsers at the moment, in that case search for "built-in dynamic linker cache").
>not supported in non-Chromium browsers at the moment
Not supported in any-browser-that-follows-the-spec, really, considering scroll-to-text is a google made, unilaterally implemented, nonstandard extension (https://wicg.github.io/scroll-to-text-fragment/)
Not a macOS dev but maybe one can clarify: is the title "No more system shared libraries in macOS 11" actually true? The documentation reads to me as a restriction on how you read dynamic libraries (is using only dlopen controversial here? also just trying to understand), not on which system libraries are actually there. It seems dlopen with custom system dylibs would still work if there is also a means to get your dylibs in the system cache? Based on that piece of docs alone I see no reason to grab my pitchfork just yet.
Anyway, I was expecting a few shitstorms to start happening over macOS 11. Apple not announcing any major deprecations (edit: or removals) for this transition - even OpenGL/CL is still there, depecrated - was suspicious.
You're correct. For 3rd-party provided dylibs, it also isn't necessary to get them into the system cache.
dlopen() will look for the dylib first (as today), and if the file doesn't exist, it'll look in the shared cache.
So the only people this actually impacts are those who stat() system-provided dylibs before calling dlopen(). They should just skip the stat(), and go straight to calling dlopen().
Certainly not worthy of the end-of-the-world vibes in the twitter thread...
Isn't this the model that has been encouraged on Windows forever with COM, WinRT and .NET etc? When you consume an API in one of those environments you don't have to worry about the location of the physical assets (library files).
I don't see the big deal as long as app local copies of e.g. curl can still be used
FWIW it is also how you can use Win32 APIs introduced in recent versions of Windows when you want to also support older versions - call LoadLibrary with a system DLL name (no path) and use GetProcAddress to check if the call you want to perform is supported. AFAIK it is very rare for Windows applications to hardcode any DLL path.
Maybe for new libraries, but I know that enough older Windows apps rely on expecting existing system libraries to exist as actual files that Wine has to create fake DLL files in order to keep them all happy.
This is mildly interesting but not much more than that. Who cares if the set of system libraries are available in a series of separate dylib files, instead of being packed together in one mega archive?
Most comments here on HN seem to misunderstand what this is.
Yeah, they’ve been doing the pre-linked mega archive thing for years. It’s in /var/db/dyld. The only difference here that I see is that they’re not shipping the originals and not rebuilding it all the time. And I’m guessing the resulting archive is now sitting on a protected partition.
This thing can be taken apart, but it’s a little tricky, since it’s pre-linked. I wrote something years ago to do it, not perfect, but enough to get symbols out. (I was trying to annotate iOS stack traces.) I don’t recall if I got it working well enough to open the libraries up in hopper and poke around.
The move seems to enforce memory-always, not file-based dyn linking. Whatever is sitting in the memory that responds to the established API.
On the first look it's not that different from how this works now. However, I'm thinking of the next step where the 'system cache' is not even local to the PC, like in a case of a netboot the whole or core set would get downloaded from the remote server be it Apple's or your friendly custom provider/employer.
If I understand correctly, this is just the same behavior as iOS's "dyld_shared_cache" which is basically all the system shared libraries stuffed into a single file.
Correct, all the system libraries can be now found in the dyld shared cache can be found in /System/Library/dyld/dyld_shared_cache_x86_64(h) (it's moved in Big Sur!).
Good. Apple is basically the only entity seriously looking at dynamic linking performance these days. We can do a lot better in the ELF world. We should not only copy dyld's closure cache, but also teach the kernel to do relocations on demand, like NT does. A relocated data-segment page should be a clean page, not a dirty page!
To think, the Amiga was doing something like this in the 80's. The "OpenLibrary" call would transparently access shared libraries either in ROM, already loaded into memory, or on disk.
I wonder how many years we've got where you're still able to read the big mega archive...
Someday it may sit in some kind of "protected enclave", mapped into your process with "execute-only" permission for code, and maybe graded protections for its data as well, so you can't debug, trace, instrument, reverse engineer or otherwise get to really know the system libraries to see what they are doing and learn from them.
The trend has been that way for a while. For the moment, that kind of isolation is done at process boundaries, on devices that have enclaves.
There is no technical reason why that kind of isolation cannot extend inside processes, for a company that designs its own CPUs. For the same kind of "black boxes" as we already have inside mobile devices, but with faster API calls and data exchange than doing it across processes boundaries.
Likely an unpopular opinion: not everything has to be a file. UNIX removed a lot of obstacles and helped usher in the era of computing accessible to individuals -- but we can't keep relying on old paradigms when it's now clear (with the benefit of hindsight) that it's not a be-all-end-all solution.
Filesystems are awful databases and I believe everyone who was a programmer for some time realizes this. The lack of ACID being at the top.
The sooner we move to specialized APIs for, well, everything, the better off we're going to be.
---
As one recent example, the PS 5 will just directly load 3D artefacts from the SSD storage to the GPU when the corresponding API is called. This gives Sony peace of mind that people won't be coming up with 100+ ways of doing it by themselves which Sony will then have to support for the entirety of the PS 5 lifecycle.
We the programmers have demonstrated to ourselves, time and again, that too much freedom can actually be a bad thing.
It's okay to move away from the "everything is a file" philosophy. If anything, I'd love to see a Linux-like OS that utilizes sqlite3 for, say, the entirety of the current `/etc` directory and uses sqlite3's backup/changeset APIs to do incremental and point-in-time backups. Sure it's little less transparent but you can't ever capture an inconsistent snapshot. That has to be an improvement, right?
---
EDIT: I suppose people can't find a link between my comment and the topic but my point was that moving to dedicated APIs -- and not relying on the filesystem -- for various services (like loading a dynamic library) is a good thing because the "everything is a file" paradigm isn't as bulletproof as it's made out to be (also because "stat" + "dlopen" one after another are race-condition prone as others pointed out).
I'd love to hear the downvoters elaborate their reasoning.
I think this is a good change. It's eliminates a potential avenue of accidentally doing the wrong thing but not noticing because it "works on my machine."
Can anyone come up with a reason this would be preferable to having just the index cached for the linker and handing out mapped memory / open files instead? What's the benefit here over essentially ld.so.cache with bells and whistles?
Historically the dyld_shared_cache has been a pre-bound copy of most system libraries that is mapped into memory once and then shared across all processes. It was updated automatically by the system at various times, and if a given system library was determined to be newer then the dynamic linker would simply load it from disk. The result of this was that for most system libraries you ended up with two copies on disk: one at the normal location, and one within the shared cache. Now only the latter copy exists, and Apple ships it directly rather than creating it on each machine from the copy of the system libraries that they ship.
Sorry, but the /usr model for compiling is out of date / flawed. It isn't about "me" it is about "you" the target for running the executable.
For many years, macOS has supported being a cross compilation environment for multiple platforms and multiple OS releases.
This means that include files and libraries for the target OS and platform are not stored at the "host filesystem" paths that were used in the 1960's, 1970's and 1980's... (an embarrassingly long list). Include files and references to libraries are available in the SDKs. One should use the SDK path as a prefix for the include and library paths - something that most GNU tools, and other open source projects have supported for a very long time, but sadly a little inconsistently. There are runtime tools and recipes that can be used to get these paths, including for "myself".
All compilation on macOS should be considered as cross compilation -- because macOS simultaneously supports multiple architectures and OS versions.
This really is a good abstracted model for all compilation.
This has benefits! Install time and disk space usage are reduced for most customers and developers. Developers on x86_64 do not need the aarch64 content in /System, /usr to compile. Install the SDKs for the targets you wish to compile for.
Frees up cheap disk space by moving the library contents to expensive RAM? That doesn't seem like a good thing. And it also doesn't seem possible — you want system libraries to survive reboot. Presumably these files exist somewhere on disk, just hidden.
I assume this is for privacy (an app can’t fingerprint you by looking at which libraries you have) but macOS is starting to become a terrible dev environment. Why are they locking down my device from me?
If nothing else, it's a size win. These libraries serve no purpose with a prebuilt dyld cache, so you'd be paying to download / would be paying to upload them to every customer for no good reason.
When you say "Starting To" I presume you mean from about 2015 on. It's clear Windows is trying to pick up slack and also be cloud developer friendly with WSL integration.
Apple sticking with Intel and now reverting to a terrible proprietary cpu architecture, locking software down more and more, releasing pro versions with inadequate options and keyboard all acts to ensure they lose their status as top developer environment
Before they dropped powerpc for intel, no one would have considered using a mac for dev work.
It's clear they don't understand the developer market at all or are only interested in Mobile because the revenue is insane.
They had a good ten years as the best dev environment but as they say "The King is dead, long live the King".
How common is this technique for loading libraries? How about in the kind of POSIX tools that are available, but not exclusive to, on mac os (sucha as bash, termux, vim, git, apache etc)?
The old code checks for the filepath to be present before attempting to dlopen it. The new way requires you to dlopen the path. This is very meh because dlopen already executes code from the library, so if you didn't want that, good luck.
(glibc and some other core libs like openssl like to detect plugins by probing the paths for libs and only loading them when necessary)
I would expect this to impact mainly build systems, and I would expect to impact them really hard. But to be honest this is just a gut feeling. On the other hand, I am sure that Apple has thought this through, as we all know developer experience is of paramount importance for them.
Is there a good term which distinguishes a general-purpose OS (Unix, Windows, Plan 9, etc) from operating systems like Android, iOS, Windows RT, etc? The term "operating system" is being overloaded to ship systems which are really more of a graphical shell than a classical operating system. The moves to lock down operating systems, ostensibly made for "security" reasons but conveniently centralizing power and authority with the vendor at the same time, are being marketed to consumers as the same kind of product as a conventional operating system while in fact being profoundly different.
[+] [-] oefrha|5 years ago|reply
Edit: Since this comment is currently at the top, here's a better link to the source: https://developer.apple.com/documentation/macos-release-note... (using scroll-to-text fragment[1] since there's unfortunately no usable anchor to the specific entry; not supported in non-Chromium browsers at the moment, in that case search for "built-in dynamic linker cache").
[1] https://caniuse.com/#feat=url-scroll-to-text-fragment
[+] [-] ohgodplsno|5 years ago|reply
Not supported in any-browser-that-follows-the-spec, really, considering scroll-to-text is a google made, unilaterally implemented, nonstandard extension (https://wicg.github.io/scroll-to-text-fragment/)
[+] [-] enedil|5 years ago|reply
[+] [-] dang|5 years ago|reply
[+] [-] DCKing|5 years ago|reply
Anyway, I was expecting a few shitstorms to start happening over macOS 11. Apple not announcing any major deprecations (edit: or removals) for this transition - even OpenGL/CL is still there, depecrated - was suspicious.
[+] [-] frogblast|5 years ago|reply
dlopen() will look for the dylib first (as today), and if the file doesn't exist, it'll look in the shared cache.
So the only people this actually impacts are those who stat() system-provided dylibs before calling dlopen(). They should just skip the stat(), and go straight to calling dlopen().
Certainly not worthy of the end-of-the-world vibes in the twitter thread...
[+] [-] Someone|5 years ago|reply
It wouldn’t surprise me if there’s no way to get your dynamic library there anymore, as this explicitly says it’s of the system provided ones.
[+] [-] akmarinov|5 years ago|reply
[+] [-] nly|5 years ago|reply
I don't see the big deal as long as app local copies of e.g. curl can still be used
[+] [-] badsectoracula|5 years ago|reply
[+] [-] makomk|5 years ago|reply
[+] [-] Hello71|5 years ago|reply
[+] [-] gruez|5 years ago|reply
[+] [-] 0x0|5 years ago|reply
Most comments here on HN seem to misunderstand what this is.
[+] [-] dunham|5 years ago|reply
This thing can be taken apart, but it’s a little tricky, since it’s pre-linked. I wrote something years ago to do it, not perfect, but enough to get symbols out. (I was trying to annotate iOS stack traces.) I don’t recall if I got it working well enough to open the libraries up in hopper and poke around.
[+] [-] asveikau|5 years ago|reply
I often dig at shared libraries using tools like nm to see what functions are where.
[+] [-] zoomablemind|5 years ago|reply
On the first look it's not that different from how this works now. However, I'm thinking of the next step where the 'system cache' is not even local to the PC, like in a case of a netboot the whole or core set would get downloaded from the remote server be it Apple's or your friendly custom provider/employer.
MacBook Pro VT. It's a stretch, of course...
[+] [-] eugenekolo|5 years ago|reply
https://iphonedevwiki.net/index.php/Dyld_shared_cache
[+] [-] saagarjha|5 years ago|reply
[+] [-] coldtea|5 years ago|reply
[+] [-] john_alan|5 years ago|reply
Does this affect homebrew?
[+] [-] quotemstr|5 years ago|reply
[+] [-] icedchai|5 years ago|reply
[+] [-] sys_64738|5 years ago|reply
[+] [-] jlokier|5 years ago|reply
Someday it may sit in some kind of "protected enclave", mapped into your process with "execute-only" permission for code, and maybe graded protections for its data as well, so you can't debug, trace, instrument, reverse engineer or otherwise get to really know the system libraries to see what they are doing and learn from them.
The trend has been that way for a while. For the moment, that kind of isolation is done at process boundaries, on devices that have enclaves.
There is no technical reason why that kind of isolation cannot extend inside processes, for a company that designs its own CPUs. For the same kind of "black boxes" as we already have inside mobile devices, but with faster API calls and data exchange than doing it across processes boundaries.
[+] [-] pdimitar|5 years ago|reply
Filesystems are awful databases and I believe everyone who was a programmer for some time realizes this. The lack of ACID being at the top.
The sooner we move to specialized APIs for, well, everything, the better off we're going to be.
---
As one recent example, the PS 5 will just directly load 3D artefacts from the SSD storage to the GPU when the corresponding API is called. This gives Sony peace of mind that people won't be coming up with 100+ ways of doing it by themselves which Sony will then have to support for the entirety of the PS 5 lifecycle.
We the programmers have demonstrated to ourselves, time and again, that too much freedom can actually be a bad thing.
It's okay to move away from the "everything is a file" philosophy. If anything, I'd love to see a Linux-like OS that utilizes sqlite3 for, say, the entirety of the current `/etc` directory and uses sqlite3's backup/changeset APIs to do incremental and point-in-time backups. Sure it's little less transparent but you can't ever capture an inconsistent snapshot. That has to be an improvement, right?
---
EDIT: I suppose people can't find a link between my comment and the topic but my point was that moving to dedicated APIs -- and not relying on the filesystem -- for various services (like loading a dynamic library) is a good thing because the "everything is a file" paradigm isn't as bulletproof as it's made out to be (also because "stat" + "dlopen" one after another are race-condition prone as others pointed out).
I'd love to hear the downvoters elaborate their reasoning.
[+] [-] Spivak|5 years ago|reply
[+] [-] loeg|5 years ago|reply
[+] [-] viraptor|5 years ago|reply
[+] [-] bdash|5 years ago|reply
Historically the dyld_shared_cache has been a pre-bound copy of most system libraries that is mapped into memory once and then shared across all processes. It was updated automatically by the system at various times, and if a given system library was determined to be newer then the dynamic linker would simply load it from disk. The result of this was that for most system libraries you ended up with two copies on disk: one at the normal location, and one within the shared cache. Now only the latter copy exists, and Apple ships it directly rather than creating it on each machine from the copy of the system libraries that they ship.
[+] [-] szc|5 years ago|reply
For many years, macOS has supported being a cross compilation environment for multiple platforms and multiple OS releases.
This means that include files and libraries for the target OS and platform are not stored at the "host filesystem" paths that were used in the 1960's, 1970's and 1980's... (an embarrassingly long list). Include files and references to libraries are available in the SDKs. One should use the SDK path as a prefix for the include and library paths - something that most GNU tools, and other open source projects have supported for a very long time, but sadly a little inconsistently. There are runtime tools and recipes that can be used to get these paths, including for "myself".
All compilation on macOS should be considered as cross compilation -- because macOS simultaneously supports multiple architectures and OS versions.
This really is a good abstracted model for all compilation.
This has benefits! Install time and disk space usage are reduced for most customers and developers. Developers on x86_64 do not need the aarch64 content in /System, /usr to compile. Install the SDKs for the targets you wish to compile for.
[+] [-] sjnu|5 years ago|reply
[+] [-] loeg|5 years ago|reply
[+] [-] qppo|5 years ago|reply
Does this break pkg-config?
What about fishing for dylibs in .bundle directories that may be installed in multiple locations?
[+] [-] tobyhinloopen|5 years ago|reply
[+] [-] celeritascelery|5 years ago|reply
[+] [-] gspr|5 years ago|reply
Well I mean it only applied to system libraries. And for those you can now just `dlopen` and check for failure, so it can't be about privacy.
> Why are they locking down my device from me?
I mean isn't this what Apple has been doing at least since the turn of the millenium?
[+] [-] 95014_refugee|5 years ago|reply
[+] [-] antishatter|5 years ago|reply
Apple sticking with Intel and now reverting to a terrible proprietary cpu architecture, locking software down more and more, releasing pro versions with inadequate options and keyboard all acts to ensure they lose their status as top developer environment
Before they dropped powerpc for intel, no one would have considered using a mac for dev work.
It's clear they don't understand the developer market at all or are only interested in Mobile because the revenue is insane.
They had a good ten years as the best dev environment but as they say "The King is dead, long live the King".
[+] [-] coldtea|5 years ago|reply
Because it belongs to the end users, and it should be as task focused, opaque, and easy to use as an appliance, not something you tinker with.
[+] [-] olliej|5 years ago|reply
[+] [-] afandian|5 years ago|reply
[+] [-] zaarn|5 years ago|reply
(glibc and some other core libs like openssl like to detect plugins by probing the paths for libs and only loading them when necessary)
[+] [-] jdub|5 years ago|reply
System libraries (depending how Apple defines that)? Unusual.
[+] [-] lultimouomo|5 years ago|reply
[+] [-] ddevault|5 years ago|reply
[+] [-] saagarjha|5 years ago|reply
[+] [-] messe|5 years ago|reply
[+] [-] saagarjha|5 years ago|reply