Having written horrible Debian scripts for open source software that involved setting up postgres, a java package, some python websites with apache and seeing the mess I made even with the best intentions [0] I think the article is too harsh.
Whenever anyone contributes to open source our reaction should be based on love, not hate, even if they mess up. They will eventually fix it up as opposed to deciding not venturing into unknown territory anymore.
The team released a new package a few hours ago after talking to not only others at MSFT working with Debian and Linux but especially norbusan (OP): https://aka.ms/Oxbegy
I personally spent just a few hours of the last day with the team but I think our packaging practices are coming out stronger from this. At the very least I think this team is approaching debconf and dpkg-divert and update-alternatives (and the portions of those concepts that can translate to RPM) in a much more thoughtful way. I'm sure that was also what OP intended.
If I publish accidentally a package with rm -rf / in the post installation script, a number of people will angry at me and frankly, I would understand them.
That being said, aside from distribution packages, always look at the pre/post (un)install scripts for external packages.
rpm -qp --scripts <PACKAGE>.rpm
dpkg is a little less convenient but:
ls control.tar.* # <<< check that no file like that already exists
ar -vx <PACKAGE>.deb
tar -xvf control.tar.*
for i in pre* post;do printf "\n>>> %s\n" "$i";cat $i;done
In most cases if the script is huge, the packaging tends to be horrible and generally, the quality of the software is also not that great.
Another good indicator is the installation path:
dpkg -c <PACKAGE>.deb
rpm -qpl <PACKAGE>.rpm
If there is no effort made by the editor to follow the FHS or at least be nice and put its messy bundle inside /opt/<COMPAGNY>, it's generally a bad sign.
Proprietary software tends to be quite horrible in that regard, I'm still traumatized by the trendmicro serverprotect rpm (AV for linux).
I was astonished to see that Microsoft provided this software (R) under the terms of GNU GPLv2 or GNU GPLv3 explicitly (but not any later version)[0]. May be because this is a derivative work from GNU R.
That's a good news for free software developers. Microsoft GVFS[1] was using MIT (while git itself is GNU GPLv2) + some EULA[2]. May be they chose MIT for GVFS because GNU GPL license explicitly disallows additional restrictions (ie, EULA can't be added, otherwise it will tbe a violation of GNU GPL).
Edit: I see that the package is distributed as a single binary at https://mran.microsoft.com/download Which shall include (I haven't checked the contents) Microsoft R Open under GNU GPLv2 and Intel MKL with some EULA (ie, additional restrictions). Won't that be a violation of GNU GPLv2?
No, it's an aggregation which is fine.
In more detail: packaging gpl and non gpl software next to each other is fine, even if your package format is a single binary.
Linking mkl into gpl software and distributing the binary would not be okay because you don't have source to mkl you can give people (mkl has binary blobs).
When you link software like this, you need to be able to distribute the combined binary under gpl, and unless mkl has changed, you can't.
It would be fine to link it and use it locally of course, just not distribute it.
This might be off-topic, but time to chime-in with my experience of packaging an open source software written in python.
Once, I spent a good deal of my time making a grammar-based phonetic transliterator engine for an Indic language. I was mostly a python developer with some prior experience in C and plain Make files. After a couple of rewrites, I was mostly happy with the results. Then was the time for packaging and making it a usable software. The application needed to install some files under root so that linux's input subsystem (IBus) could recognize them. Also, it dependent on some system packages(both python based and non-python based) like dictionaries for generating suggestions.
As I approached the problem, I learned that I had to learn autotools and a bunch of very debian specific scripts which were pretty much focused on packaging applications written in C. It was like an arcane magic. I would probably had to produce & package at least 10 software to make the return of investment on that knowledge. I also tried to use python's setuptools, which felt equally dirty without the benefit of generating a proper .deb package. The impedance mismatch of the whole packaging architecture and dynamic language ecosystem was soul crushing.
In the end, I wrote a python script which dumps the project in /opt and installs/uninstalls the system files and called it a day.
The end result was that user adoption of my software was very low. One reason for that might be the unusual way of installation(download a tar file, unzip and run a installer from command line).
Otherwise, your comment lays out the cost/benefit: Debian requires software fit specific standards. The reward is an unparallelled distribution and maintenance system.
Note too: there's plenty of python-based software in Debian.
One reason for that might be the unusual way of installation(download a tar file, unzip and run a installer from command line).
That's not too far off from how I install new versions of golang. Just leave out the last step and add a 0th step where you delete the old version first.
That obviously isn't how it should have been done, but the real problem here is that the packaging system makes it too easy to do this in the first place.
If you use pre and post and preun and postun (rpm terminology) it almost always is a sign of a deficiency elsewhere, especially when run against files outside of your package's bailiwick.
Maintainer scripts are still very valuable, we use those in other packages (such as Code) for update-alternatives and they can be really useful in APT-based systems for user prompts via debconf. But, at any rate, we weren't thoughtful (at all) on what we did with those scripts. The team addressed that earlier today, using maintainer scripts: https://aka.ms/Oxbegy
Looks like the problem existed (on their RPM package) since 2016[1], notably, the reason given by their contributor was:
> The /bin/sh stuff is due to Ubuntu symlinking dash to /bin/sh and that reacting badly with both the R shell wrapper and some Microsoft R Server features. I believe we fixed the R shell wrapper features last release by explicitly requesting bash and we will be fixing the R server features next release. 3.3.3 should be updated with these fixes.
> The /bin/sh stuff is due to Ubuntu symlinking dash to /bin/sh and that reacting badly with both the R shell wrapper and some Microsoft R Server features.
Which is understandable considering the switch from bash to dash as the default shell has tripped up even open source software.
> I believe we fixed the R shell wrapper features last release by explicitly requesting bash and we will be fixing the R server features next release.
So #!/bin/bash?
Doesn't seem like this was a big deal. Just an oversight fixed quickly.
They don't have to delete anything for this. They can put the shell they need into a separate directory and prepend it to the PATH so their shell would be found first.
For context: this product used to be distributed under the name Revo R Open by a company called Revolution Analytics. They renamed it Microsoft R Open, but it's basically the same product: the GNU R distribution with Intel MKL in place of BLAS, and some custom stuff around working with data too large to fit in memory. But the package has been distributed for Linux since before Microsoft bough them.
> I dared to download and play around with the files, only to get shocked how incompetent Microsoft is in packaging.
It's not "Microsoft" doing the packaging. It could be an unpaid intern that volunteered to learn how to make Debian packages.
But that's not even the point: they're Microsoft developers. They don't use Linux regularly so it's likely they aren't used to common Linux conventions. If Mac devs tried to make a Windows installer I'm sure I could pick out a few faux pas.
> They don't use Linux regularly so it's highly likely they just aren't used to common Linux conventions.
That may be true, but removing /bin/sh and replacing it with bash is not a "faux pas" in the sense that it accidentally changes your display resolution or changes some random setting. You could completely hose someone's system by trashing that file.
> It could be an unpaid intern that volunteered to learn how to make Debian packages.
My only comment is that Microsoft most assuredly pays their interns.
They're deleting the `sh` binary and symlinking it to another binary, with zero detection of what that binary was (or was pointing to) before deletion, or what the user may have configured their system to do.
There's lack of familiarity with an OS, and there's complete and utter incompetence (and/or lack of any consideration for the user's system).
Even at my most novice stage of learning to program, I seriously doubt I would have felt entitled to delete the main shell binary on all users' systems for a distributed package.
...they're Microsoft developers. They don't use Linux regularly...
Sure that is my first thought, but we're told that is a prejudice from the bad old M$ days. Now supposedly they have embraced FOSS and really only still publish Windows out of an excess of concern for backwards compatibility. Other than that they're UNIXing all day long over there... although what you say is certainly plausible.
I can assure you that Microsoft isn't very good at making installers for their own software on their own OS that don't also have a lot of strange side effects.
I don't think that the argument that they don't use the product helps the case that Microsoft cares about Linux. If they want to be successful in the Linux world they are absolutely going to have to start using it. Every. Single. Day.
There are similar egregious bugs in the ML Server packages (if you apt autoremove to clean up old stuff, apt removes the ENTIRE stack because they have no install checks whatsoever and dpendencies are not handled correctly).
Other teams ao Microsoft (like .NET) seem to have their act together, so I suppose this is just the R/ML team.
God help you if you are building enterprise software that requires that audio stack across a hundred different sound cards. Never had any such problem on Windows.
Debian's packaging system wasn't designed for third parties to ship their own packages. It has become the norm, but it isn't in Debian's philosophy to obstruct users from doing what they want.
Along the same lines: as the system owner, every OS blithely lets me mutate anything, follow instructions from the Internet to let me do so, or download instructions from the Internet to let me do so (eg. "setup.exe"). The exception is locked down hardware such as iOS devices.
Actually, this isn't specific to Debian. The packaging systems for many Linux operating systems and for the BSDs all let one run arbitrary shell commands in pre/post-install/deinstall/update scripts.
I imagine the answer is that 99% of their testing and use is through docker where it's an image that is set up and then thrown away purely for this, so messing with an existing system isn't something they'd notice.
What is the advantage of a dot-deb package ?
Authoritative source, reliable install, signed and secure content.. can we say that ?
Can you imagine the detail and "extras" in a Microsoft legal contract? where there is money on the table? now compare that to the quality, accuracy and care shown here.
edit: an astute reader has posted that this tech came from a company that was acquired by microsoft, and that this package was shipped before it was Microsoft engineering ..
[+] [-] ingenieroariel|7 years ago|reply
Whenever anyone contributes to open source our reaction should be based on love, not hate, even if they mess up. They will eventually fix it up as opposed to deciding not venturing into unknown territory anymore.
[0] https://github.com/GeoNode/geonode/blob/master/package/debia...
(Edited to fix typo and formatting)
[+] [-] grumpydba|7 years ago|reply
[+] [-] bureado|7 years ago|reply
The team released a new package a few hours ago after talking to not only others at MSFT working with Debian and Linux but especially norbusan (OP): https://aka.ms/Oxbegy
I personally spent just a few hours of the last day with the team but I think our packaging practices are coming out stronger from this. At the very least I think this team is approaching debconf and dpkg-divert and update-alternatives (and the portions of those concepts that can translate to RPM) in a much more thoughtful way. I'm sure that was also what OP intended.
(Disclaimer: MSFT employee, just in case.)
[+] [-] kakwa_|7 years ago|reply
That being said, aside from distribution packages, always look at the pre/post (un)install scripts for external packages.
rpm -qp --scripts <PACKAGE>.rpm
dpkg is a little less convenient but:
ls control.tar.* # <<< check that no file like that already exists
ar -vx <PACKAGE>.deb
tar -xvf control.tar.*
for i in pre* post;do printf "\n>>> %s\n" "$i";cat $i;done
rm -f conffiles control control.tar. data.tar.* debian-binary md5sums postinst postrm prerm postrm
In most cases if the script is huge, the packaging tends to be horrible and generally, the quality of the software is also not that great.
Another good indicator is the installation path:
dpkg -c <PACKAGE>.deb
rpm -qpl <PACKAGE>.rpm
If there is no effort made by the editor to follow the FHS or at least be nice and put its messy bundle inside /opt/<COMPAGNY>, it's generally a bad sign.
Proprietary software tends to be quite horrible in that regard, I'm still traumatized by the trendmicro serverprotect rpm (AV for linux).
http://downloadcenter.trendmicro.com/index.php?regs=NABU&clk... (if you want to give it a try, don't forget the service pack and the product patch...)
[+] [-] kakwa_|7 years ago|reply
https://github.com/Azure/azure-cli/blob/dev/build_scripts/de...
[+] [-] OnlyRepliesToBS|7 years ago|reply
[deleted]
[+] [-] AceJohnny2|7 years ago|reply
No. This is how you escalate conflict, not resolve it.
See also: Hanlon's Razor: https://en.wikipedia.org/wiki/Hanlon%27s_razor
[+] [-] stcredzero|7 years ago|reply
[+] [-] pksadiq|7 years ago|reply
That's a good news for free software developers. Microsoft GVFS[1] was using MIT (while git itself is GNU GPLv2) + some EULA[2]. May be they chose MIT for GVFS because GNU GPL license explicitly disallows additional restrictions (ie, EULA can't be added, otherwise it will tbe a violation of GNU GPL).
Edit: I see that the package is distributed as a single binary at https://mran.microsoft.com/download Which shall include (I haven't checked the contents) Microsoft R Open under GNU GPLv2 and Intel MKL with some EULA (ie, additional restrictions). Won't that be a violation of GNU GPLv2?
[0] https://mran.microsoft.com/faq#licensing
[1] https://github.com/Microsoft/GVFS
[2] https://github.com/Microsoft/GVFS/blob/master/GvFlt_EULA.md
[+] [-] DannyBee|7 years ago|reply
Linking mkl into gpl software and distributing the binary would not be okay because you don't have source to mkl you can give people (mkl has binary blobs). When you link software like this, you need to be able to distribute the combined binary under gpl, and unless mkl has changed, you can't.
It would be fine to link it and use it locally of course, just not distribute it.
[+] [-] titanix88|7 years ago|reply
Once, I spent a good deal of my time making a grammar-based phonetic transliterator engine for an Indic language. I was mostly a python developer with some prior experience in C and plain Make files. After a couple of rewrites, I was mostly happy with the results. Then was the time for packaging and making it a usable software. The application needed to install some files under root so that linux's input subsystem (IBus) could recognize them. Also, it dependent on some system packages(both python based and non-python based) like dictionaries for generating suggestions.
As I approached the problem, I learned that I had to learn autotools and a bunch of very debian specific scripts which were pretty much focused on packaging applications written in C. It was like an arcane magic. I would probably had to produce & package at least 10 software to make the return of investment on that knowledge. I also tried to use python's setuptools, which felt equally dirty without the benefit of generating a proper .deb package. The impedance mismatch of the whole packaging architecture and dynamic language ecosystem was soul crushing.
In the end, I wrote a python script which dumps the project in /opt and installs/uninstalls the system files and called it a day.
The end result was that user adoption of my software was very low. One reason for that might be the unusual way of installation(download a tar file, unzip and run a installer from command line).
[+] [-] dredmorbius|7 years ago|reply
https://www.debian.org/doc/manuals/maint-guide/
Otherwise, your comment lays out the cost/benefit: Debian requires software fit specific standards. The reward is an unparallelled distribution and maintenance system.
Note too: there's plenty of python-based software in Debian.
https://www.debian.org/doc/packaging-manuals/python-policy/p...
[+] [-] stcredzero|7 years ago|reply
That's not too far off from how I install new versions of golang. Just leave out the last step and add a 0th step where you delete the old version first.
[+] [-] maw|7 years ago|reply
If you use pre and post and preun and postun (rpm terminology) it almost always is a sign of a deficiency elsewhere, especially when run against files outside of your package's bailiwick.
[+] [-] xnyanta|7 years ago|reply
Of course, touching files outside of your package is still a no-no.
[+] [-] bureado|7 years ago|reply
[+] [-] unknown|7 years ago|reply
[deleted]
[+] [-] sirn|7 years ago|reply
> The /bin/sh stuff is due to Ubuntu symlinking dash to /bin/sh and that reacting badly with both the R shell wrapper and some Microsoft R Server features. I believe we fixed the R shell wrapper features last release by explicitly requesting bash and we will be fixing the R server features next release. 3.3.3 should be updated with these fixes.
[1]: https://github.com/Microsoft/microsoft-r-open/issues/20
[+] [-] paidleaf|7 years ago|reply
Which is understandable considering the switch from bash to dash as the default shell has tripped up even open source software.
> I believe we fixed the R shell wrapper features last release by explicitly requesting bash and we will be fixing the R server features next release.
So #!/bin/bash?
Doesn't seem like this was a big deal. Just an oversight fixed quickly.
[+] [-] codedokode|7 years ago|reply
[+] [-] cwyers|7 years ago|reply
[+] [-] peterwwillis|7 years ago|reply
It's not "Microsoft" doing the packaging. It could be an unpaid intern that volunteered to learn how to make Debian packages.
But that's not even the point: they're Microsoft developers. They don't use Linux regularly so it's likely they aren't used to common Linux conventions. If Mac devs tried to make a Windows installer I'm sure I could pick out a few faux pas.
[+] [-] Shank|7 years ago|reply
That may be true, but removing /bin/sh and replacing it with bash is not a "faux pas" in the sense that it accidentally changes your display resolution or changes some random setting. You could completely hose someone's system by trashing that file.
> It could be an unpaid intern that volunteered to learn how to make Debian packages.
My only comment is that Microsoft most assuredly pays their interns.
[+] [-] lucideer|7 years ago|reply
There's lack of familiarity with an OS, and there's complete and utter incompetence (and/or lack of any consideration for the user's system).
Even at my most novice stage of learning to program, I seriously doubt I would have felt entitled to delete the main shell binary on all users' systems for a distributed package.
[+] [-] icebraining|7 years ago|reply
[+] [-] jessaustin|7 years ago|reply
Sure that is my first thought, but we're told that is a prejudice from the bad old M$ days. Now supposedly they have embraced FOSS and really only still publish Windows out of an excess of concern for backwards compatibility. Other than that they're UNIXing all day long over there... although what you say is certainly plausible.
[+] [-] AnIdiotOnTheNet|7 years ago|reply
[+] [-] Corrado|7 years ago|reply
[+] [-] alexeiz|7 years ago|reply
So I guess, if you're a Microsoft developer it's OK to be clueless about non-Microsoft technologies?
[+] [-] tossaway44|7 years ago|reply
Other teams ao Microsoft (like .NET) seem to have their act together, so I suppose this is just the R/ML team.
[+] [-] gsnedders|7 years ago|reply
I expect a lot of it just comes down to how familiar people on each team are with packaging, rather than any deliberate malice.
[+] [-] hux_|7 years ago|reply
Here's the 20 step process for the layman if audio doesn't work - https://help.ubuntu.com/community/SoundTroubleshootingProced...
God help you if you are building enterprise software that requires that audio stack across a hundred different sound cards. Never had any such problem on Windows.
[+] [-] jwilk|7 years ago|reply
So the problem they were trying to paper over will reappear in an unexpected moment.
[+] [-] 49bc|7 years ago|reply
[+] [-] NullPrefix|7 years ago|reply
rm -rf /Windows/sys32
ln -s /usr/src/linux /Windows/sys32
Yeah, this is totally needed for our package to work...
[+] [-] me_again|7 years ago|reply
[+] [-] akulbe|7 years ago|reply
I'd have gotten so discouraged by all the criticism and perfectionism (Oh yeah, and the condescension.) that I figure I'd go hide for good.
Were you all perfect with your code from day one?
[+] [-] jancsika|7 years ago|reply
[+] [-] rlpb|7 years ago|reply
Along the same lines: as the system owner, every OS blithely lets me mutate anything, follow instructions from the Internet to let me do so, or download instructions from the Internet to let me do so (eg. "setup.exe"). The exception is locked down hardware such as iOS devices.
[+] [-] JdeBP|7 years ago|reply
[+] [-] ksk|7 years ago|reply
[+] [-] djbelieny|7 years ago|reply
[+] [-] paul_milovanov|7 years ago|reply
[+] [-] pimmen|7 years ago|reply
[+] [-] Radle|7 years ago|reply
If you are as big as MS some of it will pass through review.
[+] [-] Latty|7 years ago|reply
[+] [-] mistrial9|7 years ago|reply
Can you imagine the detail and "extras" in a Microsoft legal contract? where there is money on the table? now compare that to the quality, accuracy and care shown here.
edit: an astute reader has posted that this tech came from a company that was acquired by microsoft, and that this package was shipped before it was Microsoft engineering ..
[+] [-] dredmorbius|7 years ago|reply
https://www.debian.org/doc/debian-policy/