I wouldn't recommend following the Haskell approach. It hasn't worked well for us. (I took part in creating the Haskell Platform and the process used to add packages to it. I also used to maintain a few of our core libraries, like our containers packages and networking).
Small vs large standard library:
A small standard library with most functionality in independent, community-maintained packages has given us API friction as types, traits (type classes), etc are hard to coordinate across maintainers and separate package release cycles. We ended up with lots of uncomfortable conversions at API boundaries.
Here's a number of examples of problems we currently have:
- Conversions between our 5(!) string types are very common.
- Standard library I/O modules cannot use new, de-facto standard string types (i.e. `Text` and `ByteString`) defined outside it because of dependency cycle.
- Standard library cannot use containers, other than lists, for the same reason.
- No standard traits for containers, like maps and sets, as those are defined outside the standard library. Result is that code is written against one concrete implementation.
- Newtype wrapping to avoid orphan instances. Having traits defined in packages other than the standard library makes it harder to write non-orphan instances.
- It's too difficult to make larger changes as we cannot atomically update all the packages at once. Thus such changes don't happen.
Empirically, languages that have large standard libraries (e.g. Java, Python, Go) seem to do better than their competitors.
I don't think most of these are applicable to Rust.
> - Conversions between our 5(!) string types are very common.
> - Standard library I/O modules cannot use new, de-facto standard string types (i.e. `Text` and `ByteString`) defined outside it because of dependency cycle.
We have one string type defined in std, and nobody is defining new ones (modulo special cases for legacy encodings which would not be worth polluting the default string type with).
> - Standard library cannot use containers, other than lists, for the same reason.
> - No standard traits for containers, like maps and sets, as those are defined outside the standard library. Result is that code is written against one concrete implementation.
Hash maps and trees are in the standard library already. Everyone uses them.
> - Newtype wrapping to avoid orphan instances. Having traits defined in packages other than the standard library makes it harder to write non-orphan instances.
This is true, but this hasn't been much of a problem in Rust thus far.
> - It's too difficult to make larger changes as we cannot atomically update all the packages at once. Thus such changes don't happen.
That only matters if you're breaking public APIs, right? That seems orthogonal to the small-versus-large-standard-library debate. Even if you have a large standard library, if you promised it's stable you still can't break APIs.
This is exactly the main problem with Haskell. A stunning language with a lousy standard library. In my opinion, Haskell should offer arrays and maps as built-ins (like Go) and ship with crypto, networking, and serialization in the standard library (I know serialization is already there, but everyone seems to prefer Cereal, so...)
Haskell's actual problem isn't the lack of a comprehensive standard library, but rather the presence of core language features that actively hinder large-scale modular programming. Type classes, type families, orphan instances and flexible instances all conspire to make as difficult as possible to determine whether two modules can be safely linked. Making things worse, whenever two alternatives are available for achieving roughly the same thing (say, type families and functional dependencies), the Haskell community consistently picks the worse one (in this case, type families, because, you know, why not punch a big hole on parametricity and free theorems?).
Thanks to GHC's extensions, Haskell has become a ridiculously powerful language in exactly the same way C++ has: by sacrificing elegance. The principled approach would've been to admit that, while type classes are good for a few use cases, (say, overloading numeric literals, string literals and sequences), they have unacceptable limitations as a large-scale program structuring construct. And instead use an ML-style module system for that purpose. But it's already too late to do that.
One of the common mantras I've heard among Rust core devs is "std is where code goes to die". Where do you feel the line should be drawn between standard lib and external libraries?
>Conversions between our 5(!) string types are very common.
All five of those string types do different things. This isn't a problem; we just have increased expressivity. We couldn't fix this by having a more coordinated standard library. 5 is also a very manageable number IMO.
>It's too difficult to make larger changes as we cannot atomically update all the packages at once.
If a language is to play the long game, they must be conservative on what they
add. Even a minimal runtime like Node is still wounded by the addition of a few
broken interfaces into the core platform (even emitters, streams, domains to
name a few). These cannot be stripped out of the platform because they're
"blessed" and now everyone will forever have a bad time. I suggest we don't
do that for Rust.
For a language to remain relevant in the long term, a system must be capable of
evolving. Going out there and blessing potentially conceptually incorrect
packages such as Mio is therefore not a good idea. The notion of "platforms"
best resides in userland, where collections can compete and evolve.
By keeping platforms / package collections in userland we can get organic
collections of stuff that make sense to bundle together. Imagine a "server
platform", an "event loop platform", a "calculus platform", a "programming
language platform". It would be detrimental to creativity to have all these
collections live in the shadow of the "rust committee blessed platform".
But so yeah, I'm not opposed to the idea of platforms - I just don't think
blessing userland stuff from the top down is the right play in the long term.
Tl;Dr: package collections sound like a cool idea, but if you care about the
long term, imposing them from the top down is bad
> These cannot be stripped out of the platform because they're "blessed" and now everyone will forever have a bad time. I suggest we don't do that for Rust.
It sounds like the proposal in the OP avoids this problem by having versioned platforms that are independent of the Rust version. So if something turns out to be a bad idea, it can be stripped out of later platform versions and replaced with something better without disrupting users of the older platforms.
> By keeping platforms / package collections in userland we can get organic collections of stuff that make sense to bundle together. Imagine a "server platform", an "event loop platform", a "calculus platform", a "programming language platform". It would be detrimental to creativity to have all these collections live in the shadow of the "rust committee blessed platform".
That's exactly what we want to do. We don't have the domain expertise in many of these areas, but we want to enable those ecosystems to help develop their own platforms. It's one of our goals to make sure any infrastructure we develop for the "Rust Platform" is usable throughout our community.
I'm a bit worried when a language develops a "platform" and an "ecosystem". This usually means you need to bring in a large amount of vaguely relevant stuff to do anything. It adds another layer of cruft, and more dependencies.
Write standalone tools, but don't create a "platform". Don't make the use of the language dependent on your tools.
C++ does not have a "platform". Nor does it need one.
C++ has innumerable de facto "platforms" and "ecosystems". You have to choose one to get anything done, whether that be various Boost libraries, POSIX, Win32, Cocoa, Qt, GTK(mm), even stuff like XPCOM…
Helpfully, many of these platforms reinvent basic things like strings [1] and reference counted smart pointers [2] in incompatible ways.
Wouldn't it be better if there were just one platform?
I get the feeling that you didn't really read the post. Nothing in the platform is required to use Rust, and you can trivially write Rust packages that don't use the platform. The point of the platform is for convenience. In most cases it will make sense to use it because it provides a convenient set of libraries that are known to work well together, but you could also choose to just ignore the Rust platform entirely and continue to use Rust the same way we've been using it up to now.
Go watch recent talks by Herb Sutter and Bjarne Stroustrup--they both lament the fact that C++ never developed as strong a standard library as Python, etc. With C++14 and beyond the C++ working committee is actively trying to make the language and libraries more complete and comparable to larger libraries out there.
Qt / Boost / .net are C++ platforms. The difference is you can choose one or none, and the OP actually explicitly talks about how important it is not to have one absolute blessed platform like Java has.
Haskell Platform is the last thing you should take inspiration from. Many of us have been doing our best to kill it off. Maybe the downsides involved wouldn't affect Rust in the same ways.
My suggestion, look at how Stack (the tool) and Stackage (the platform) work:
Most of your arguments linked in a comment below are unrelated to the application of metapackages to cargo. Cargo already includes a good portion of the behaviors found in Stack and Stackage. The idealogical battle of Stack vs Haskell Platform is irrelevant to this proposal.
Haskell Platform is a perfectly adequate example to take high-level inspiration for the core of this idea: use Cargo (which is like Stack/Stackage for rust) to help bootstrap Rust libraries when using rustup to install and upgrade (mostly equivalent to `stack setup`).
The one thing stack does do nicely that this proposal doesn't is allow curated comparable versions without forcing course dependencies. In plainer English, stack can pick the versions while you still opt in pack by package. I beleive that this is crucial to not slow down the evolution of the ecosystem.
In Cargo jargon, a solutuon would be for metapackages could double as sources: `foo = { metapackage = bar }` to use foo from bar.
I actually cross-posted this to /r/haskell to get explicit feedback. Someone else mentioned stack/stackage. In my understanding, Cargo already does this specific behavior. Can anyone who's more familiar with both confirm this?
Having used Java and having experienced how you learn to replace the JDK URL parser with something else, the JDK HTTP client with something else, the JDK encoding libraries with something else, etc., I'm worried about entrenching first-published libraries beyond their natural first-mover advantage and making it even harder for better later-comers to be adopted.
OTOH, compared to e.g. C++, where even strings aren't standard across multiple real large code bases (e.g. Gecko and Qt) having standard types especially for foundational stuff can be very useful if the foundation is right or right enough. Java having a single (though not optimal) foundational notion for Unicode (UTF-16 even though UTF-8 would be optimal) and a common efficient and spec-wise correct (though not design-wise optimal) XML API (SAX) was a key reason to develop Validator.nu in Java as opposed to e.g. Python (whose notion of Unicode varied between UTF-16 and UTF-32 based on how the interpreter was compiled!).
Still, at this early stage, I'd point to crates.io in general as the "batteries" instead of creating a layer of official approval on top. But I'm biased, because lately, I've been writing a second-to-market crate for a topic for which a crate already exists.
Good work! The idea of dropping extern crate is worrying, however. Most of the ways that that would be done would add more irregularity, complexity, and implied environment to the language (Rust Platform is always there, whether or not you want it), all of which are opportunities for bugs to creep into code.
I mean really, why is this necessary?
The reasons for adding this much bloat are tiny and irrelevant. You get a big download full of packages you dont all need, you dont know if you need and all for what?
We all have Google, if I want an HTTP library for Rust I'll google whatever the best one is and make a judgement call myself.
This is all optional. You can just ... not add the one line to the cargo.toml and manually specify your favorite http library. Folks using many of these libraries or people new to the language can just specify rust-platform deps. This has the added benefit of bringing in versions that are known to work well together -- because of semver this isn't usually a problem, but people aren't perfect so sometimes things break. An added guarantee against that is nice to have.
You can still have a tiny install if you want to. But most people not knowing what to do will be able to have batteries included setup.
Basically: people knowing will still have freedom, people not knowing will have an easier life.
And in the end, in 2016, you generally don't care if you download a few extra 100Mo. If you do, you will just find spend the time to find to have a smaller setup.
If the aim of the Rust Platform is to provide a 'blessed set' of 3rd-party libraries, why not have something akin to an official, curated 'Awesome Rust', where depending on what you want to achieve you can get a streamlined list of well-maintained libraries, perhaps with user-submitted usage-examples, comments, alternatives and the like - that way, you're not constraining Rust itself to be in sync with all the 3rd-party libs, (and thus by necessity stagnating to a certain degree), nor you're making authors of libraries that are not in the Platform feel essentially invisible.
How is that different than the proposal? The only part that's maybe different is "feel essentially invisible", which is something that might happen, I'll grant you that. But I don't think that people feel invisible when some package is in a standard library, which is roughly equivalent here.
Combine this with my stdlib deps RFC[1], and I think we could shrink the nunber standard library crates versioned with the language! The standard library metapackage would pull crates from crates.io as needed.
Can someone summarize what's going on here for lay people? I know this "Rust" thing is pretty popular on HN nowadays but not sure what the difference is between a language and a platform. I was trying to read the article, but I don't have enough knowledge about rust itself to understand what it's talking about...
Rust is the language itself. The platform in this context is the standard libraries you use for things like string manipulation, network connections, etc.
The relationship between Boost and the C++ standard library might provide a good example (especially in recent years, as there's been sustained focus on expanding stdlib). Boost has a stringent peer review and comment period. Sometimes I think they let in designs that are too clever, but they rarely let in the clunkers that are seen in Python's standard library.
People then gain experience with it and sometimes subtle issues emerge. Those issues can be fixed, either in Boost itself or when libraries move from Boost to stdlib (and sometimes the issues inform language features).
While C++'s standard library is not "batteries included" like Python's, it's been very gratifying to see it expand slowly and surely over the last few years.
Couldn't much of the function of such a "platform" be automated?
One of the main constraints that would guide selection of crates for this platform metapackage is that they must have compatible dependencies. So package A depending on Bv1 and package C depending on Bv2 wouldn't work because they Bv1 and Bv2 would both have to be included, leading to a conflict.
But this information is (in theory) encoded in semantic versioning. Assuming proper semantic versions for crates, a target set of crates to be included could be specified, and then automatically the various sets of crate versions that do not have conflicting dependencies could be calculated.
These compatible crate/version sets could be automatically generated and published as metacrates.
Consider the following crate/version dependencies:
By imposing an ordering on these compatible sets they could be automatically identified. compatible_A_C_0 is {'Av1','Cv1'}, compatible_A_C_1 is {'Av1','Cv2'}, and so on.
Obviously the semver could be wrong and unexpected incompatibilities could crop up. But couldn't these just be autogenerated and then voted on? Then the top best compatible sets will filter to the top and, de facto, the Rust Platform has been autogenerated?
> So package A depending on Bv1 and package C depending on Bv2 wouldn't
> work because they Bv1 and Bv2 would both have to be included,
> leading to a conflict.
As mentioned in this thread, this already works just fine. Rust can handle both versions.
Furthermore, it's more than just a constraint problem; there's also integration issues, testing the whole thing together, etc.
I love seeing Aaron's work within the Rust community. I had the pleasure of studying under his father and his family's gifts are clear in both their work.
Are metapackages going to be available for others to utilize? If so, how will conflicts be resolved if packages require two different versions of the same package?
Yes, metapackages are intended to be a general cargo feature, available to all. I suspect the design has not progressed far enough to definitively answer your question about conflict resolution, but I'd imagine you have to override that dep explicitly to fix it.
The TeXLive of the Rust world? Could be helpful. But the small std probably implies a constantly revolving cast of "best practice" libraries that's hard to keep up with. I know in TeXLive there are no stability guarantees regarding the collection as a whole. It's more of a collection than a platform.
Historically, there have been a number of proposals by the Rust team that were received badly. We've always benefited from outside eyeballs on things; usually the final proposals end up much stronger. We've only sought to _increase_ this kind of thing over time.
For contrast, with .NET Core, Microsoft decided to cut up .NET's quite extensive stdlib (BCL + FCL) into packages. On paper, it looks pretty good, but it remains to see how well the versioning will work in the long run.
[+] [-] tibbe|9 years ago|reply
Small vs large standard library:
A small standard library with most functionality in independent, community-maintained packages has given us API friction as types, traits (type classes), etc are hard to coordinate across maintainers and separate package release cycles. We ended up with lots of uncomfortable conversions at API boundaries.
Here's a number of examples of problems we currently have:
- Conversions between our 5(!) string types are very common.
- Standard library I/O modules cannot use new, de-facto standard string types (i.e. `Text` and `ByteString`) defined outside it because of dependency cycle.
- Standard library cannot use containers, other than lists, for the same reason.
- No standard traits for containers, like maps and sets, as those are defined outside the standard library. Result is that code is written against one concrete implementation.
- Newtype wrapping to avoid orphan instances. Having traits defined in packages other than the standard library makes it harder to write non-orphan instances.
- It's too difficult to make larger changes as we cannot atomically update all the packages at once. Thus such changes don't happen.
Empirically, languages that have large standard libraries (e.g. Java, Python, Go) seem to do better than their competitors.
[+] [-] pcwalton|9 years ago|reply
> - Conversions between our 5(!) string types are very common.
> - Standard library I/O modules cannot use new, de-facto standard string types (i.e. `Text` and `ByteString`) defined outside it because of dependency cycle.
We have one string type defined in std, and nobody is defining new ones (modulo special cases for legacy encodings which would not be worth polluting the default string type with).
> - Standard library cannot use containers, other than lists, for the same reason.
> - No standard traits for containers, like maps and sets, as those are defined outside the standard library. Result is that code is written against one concrete implementation.
Hash maps and trees are in the standard library already. Everyone uses them.
> - Newtype wrapping to avoid orphan instances. Having traits defined in packages other than the standard library makes it harder to write non-orphan instances.
This is true, but this hasn't been much of a problem in Rust thus far.
> - It's too difficult to make larger changes as we cannot atomically update all the packages at once. Thus such changes don't happen.
That only matters if you're breaking public APIs, right? That seems orthogonal to the small-versus-large-standard-library debate. Even if you have a large standard library, if you promised it's stable you still can't break APIs.
[+] [-] kibwen|9 years ago|reply
[+] [-] bitmadness|9 years ago|reply
[+] [-] catnaroek|9 years ago|reply
Thanks to GHC's extensions, Haskell has become a ridiculously powerful language in exactly the same way C++ has: by sacrificing elegance. The principled approach would've been to admit that, while type classes are good for a few use cases, (say, overloading numeric literals, string literals and sequences), they have unacceptable limitations as a large-scale program structuring construct. And instead use an ML-style module system for that purpose. But it's already too late to do that.
[+] [-] saysjonathan|9 years ago|reply
[+] [-] rcfox|9 years ago|reply
[+] [-] wyager|9 years ago|reply
All five of those string types do different things. This isn't a problem; we just have increased expressivity. We couldn't fix this by having a more coordinated standard library. 5 is also a very manageable number IMO.
>It's too difficult to make larger changes as we cannot atomically update all the packages at once.
That's what Stack is for, no?
[+] [-] yoshuaw|9 years ago|reply
For a language to remain relevant in the long term, a system must be capable of evolving. Going out there and blessing potentially conceptually incorrect packages such as Mio is therefore not a good idea. The notion of "platforms" best resides in userland, where collections can compete and evolve.
By keeping platforms / package collections in userland we can get organic collections of stuff that make sense to bundle together. Imagine a "server platform", an "event loop platform", a "calculus platform", a "programming language platform". It would be detrimental to creativity to have all these collections live in the shadow of the "rust committee blessed platform".
But so yeah, I'm not opposed to the idea of platforms - I just don't think blessing userland stuff from the top down is the right play in the long term.
Tl;Dr: package collections sound like a cool idea, but if you care about the long term, imposing them from the top down is bad
[+] [-] rcthompson|9 years ago|reply
It sounds like the proposal in the OP avoids this problem by having versioned platforms that are independent of the Rust version. So if something turns out to be a bad idea, it can be stripped out of later platform versions and replaced with something better without disrupting users of the older platforms.
[+] [-] erickt|9 years ago|reply
That's exactly what we want to do. We don't have the domain expertise in many of these areas, but we want to enable those ecosystems to help develop their own platforms. It's one of our goals to make sure any infrastructure we develop for the "Rust Platform" is usable throughout our community.
[+] [-] Animats|9 years ago|reply
Write standalone tools, but don't create a "platform". Don't make the use of the language dependent on your tools.
C++ does not have a "platform". Nor does it need one.
[+] [-] pcwalton|9 years ago|reply
Helpfully, many of these platforms reinvent basic things like strings [1] and reference counted smart pointers [2] in incompatible ways.
Wouldn't it be better if there were just one platform?
[1]: http://doc.qt.io/qt-4.8/qstring.html
[2]: http://doc.qt.io/qt-4.8/qshareddatapointer.html
[+] [-] eridius|9 years ago|reply
[+] [-] tdicola|9 years ago|reply
[+] [-] zanny|9 years ago|reply
[+] [-] pjmlp|9 years ago|reply
Sure it does, it is called C++ Standard Library and ISO C++ catching up to what other languages already offer.
https://isocpp.org/std/status
[+] [-] coolsunglasses|9 years ago|reply
My suggestion, look at how Stack (the tool) and Stackage (the platform) work:
https://docs.haskellstack.org/en/stable/README/
https://github.com/commercialhaskell/stack
https://www.stackage.org/
[+] [-] saysjonathan|9 years ago|reply
Haskell Platform is a perfectly adequate example to take high-level inspiration for the core of this idea: use Cargo (which is like Stack/Stackage for rust) to help bootstrap Rust libraries when using rustup to install and upgrade (mostly equivalent to `stack setup`).
[+] [-] Ericson2314|9 years ago|reply
In Cargo jargon, a solutuon would be for metapackages could double as sources: `foo = { metapackage = bar }` to use foo from bar.
[+] [-] steveklabnik|9 years ago|reply
[+] [-] dikaiosune|9 years ago|reply
[+] [-] hsivonen|9 years ago|reply
OTOH, compared to e.g. C++, where even strings aren't standard across multiple real large code bases (e.g. Gecko and Qt) having standard types especially for foundational stuff can be very useful if the foundation is right or right enough. Java having a single (though not optimal) foundational notion for Unicode (UTF-16 even though UTF-8 would be optimal) and a common efficient and spec-wise correct (though not design-wise optimal) XML API (SAX) was a key reason to develop Validator.nu in Java as opposed to e.g. Python (whose notion of Unicode varied between UTF-16 and UTF-32 based on how the interpreter was compiled!).
Still, at this early stage, I'd point to crates.io in general as the "batteries" instead of creating a layer of official approval on top. But I'm biased, because lately, I've been writing a second-to-market crate for a topic for which a crate already exists.
[+] [-] qwertyuiop924|9 years ago|reply
[+] [-] steveklabnik|9 years ago|reply
[+] [-] hubert123|9 years ago|reply
[+] [-] Manishearth|9 years ago|reply
[+] [-] sametmax|9 years ago|reply
Basically: people knowing will still have freedom, people not knowing will have an easier life.
And in the end, in 2016, you generally don't care if you download a few extra 100Mo. If you do, you will just find spend the time to find to have a smaller setup.
[+] [-] AsyncAwait|9 years ago|reply
[+] [-] steveklabnik|9 years ago|reply
[+] [-] Ericson2314|9 years ago|reply
[1]: https://github.com/rust-lang/rfcs/pull/1133
[+] [-] cocktailpeanuts|9 years ago|reply
[+] [-] hueving|9 years ago|reply
[+] [-] jonstewart|9 years ago|reply
People then gain experience with it and sometimes subtle issues emerge. Those issues can be fixed, either in Boost itself or when libraries move from Boost to stdlib (and sometimes the issues inform language features).
While C++'s standard library is not "batteries included" like Python's, it's been very gratifying to see it expand slowly and surely over the last few years.
[+] [-] pc2g4d|9 years ago|reply
One of the main constraints that would guide selection of crates for this platform metapackage is that they must have compatible dependencies. So package A depending on Bv1 and package C depending on Bv2 wouldn't work because they Bv1 and Bv2 would both have to be included, leading to a conflict.
But this information is (in theory) encoded in semantic versioning. Assuming proper semantic versions for crates, a target set of crates to be included could be specified, and then automatically the various sets of crate versions that do not have conflicting dependencies could be calculated.
These compatible crate/version sets could be automatically generated and published as metacrates.
Consider the following crate/version dependencies:
compatible_sets({'A', 'C'}) returns {{'Av1','Cv1'},{'Av1','Cv2'},{'Av2','Cv3'},{'Av3','Cv3'}}By imposing an ordering on these compatible sets they could be automatically identified. compatible_A_C_0 is {'Av1','Cv1'}, compatible_A_C_1 is {'Av1','Cv2'}, and so on.
Obviously the semver could be wrong and unexpected incompatibilities could crop up. But couldn't these just be autogenerated and then voted on? Then the top best compatible sets will filter to the top and, de facto, the Rust Platform has been autogenerated?
[+] [-] steveklabnik|9 years ago|reply
Furthermore, it's more than just a constraint problem; there's also integration issues, testing the whole thing together, etc.
[+] [-] saysjonathan|9 years ago|reply
Are metapackages going to be available for others to utilize? If so, how will conflicts be resolved if packages require two different versions of the same package?
[+] [-] brson|9 years ago|reply
[+] [-] pc2g4d|9 years ago|reply
[+] [-] xfactor973|9 years ago|reply
[+] [-] EugeneOZ|9 years ago|reply
[+] [-] steveklabnik|9 years ago|reply
[+] [-] moogly|9 years ago|reply
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] lindstorm|9 years ago|reply
[+] [-] steveklabnik|9 years ago|reply