heathjohns's comments

SEEKING WORK | Canada, West Coast | Remote | Senior Dev

My specialty is startups and early-stage - gap-filling and jumping in as a swiss-army knife. I have a business degree and decades of deep technical skills throughout the whole stack, from hardware design to UX.

More info here: https://heathjohns.com/

While you're right in terms of technical prowess, there absolutely is a use case for BSD: it has a distinct culture with different values.

There's a large group of people in technology who don't really care how something works, as long as it works for their purposes. That's set against another group who will put up with incovenience in the name of what they think is "correctness". I genuinely don't want to suggest that either group is better than the other - but they are distinct.

Around 2010 I felt that the first group had started to dominate the decisions around the direction of the Linux userland. It was wonderful to be able to switch to BSD, which I sense is used and maintained mostly by the second group. Having an OS that is aligned with your culture is a sort of a "happiness by a thousand cuts": the defaults are already what you would have set them to, you don't have to declutter after the first install and what you do want is in the package repo, your niche use cases get attention when it comes to bug fixes because they're not considered niche in that crowd, etc. When the maintainers think the way you do it feels like they've anticipated your needs - it's presumably the same feeling that Apple-culture people feel when they use an Apple product; like it was made for them.

In a lot of ways, technology has mostly become a matrix for competing cultures. From that perspective it's not a matter of what performance edge the BSDs have over Linux, it's about whether there's a distinct enough culture to justify the costs of the duplication of effort. In my mind that distinction has only grown in the last 10 years.

I switched to desktop Linux in 1999, and then to FreeBSD around 2014. For me it's about culture. When you use software you're interacting with hundreds of decisions that the devs made, and the more they think like you the more they will have made the same calls that you would have.

Linux's success ended up bringing in a distinctly different crowd than it had when I first started using it, and when the software decisions started to reflect that evolving culture I just found myself going against the grain for a lot of common tasks.

Switching to FreeBSD was like coming back to your home country after being abroad - there were definitely nice things about the place you were travelling in, but it's still a relief to be back in your own culture. When you travel there's lots of random surprises because everything's built on a different set of values and assumptions - that's part of the charm of being abroad - but eventually you want to get things done and it's great to be back home where you can devote your whole brain to the task at hand.

Good point - sorry if my comment was a derail!

It's a different mechanism with the same outcome. If the scale of a solution is so immense that only one or two of the largest corporations in the world can realisitically implement it, then you've got a duopoly and all the bad power imbalances that come with it.

Which is part of why I think freedom is the wrong goal. You're free to choose between Kodos and Kang, but the ability to choose conveys no power.

I think we need to zoom out more, even. The focus on freedom is too narrow - freedom at the cost of becoming disconnected is a raw deal.

What we want is power. In the desktop era, having source code gave us power, so we associated the two and demanded the former - but this obviously hasn't translated to the cloud.

You can't download a Facebook profile and then re-upload it somewhere else and still be connected to all your friends etc. That lock-in gives Facebook power over its users: proportional to how much they need what Facebook gatekeeps (e.g. "all the news about my grandkids is on there").

When I want to stop using something, but I can't, that's power someone holds over me. In my case, that's Google search, GMail, and Android. I actually hate the latter two, and I've tried leaving them all, but here I am. What they've got in common is that they're all protective layers over what I need to stay connected to the outside world despite being highly dysfunctional (i.e. discoverability on the web, email spam, and phone hardware).

Because of this, Google's got me in a way that - because of open source - Microsoft never could. Whatever comes after open source needs to address that power delta directly. We need choice, but also connection.

Similar in that you get to keep your own data.

Different in that it's lower level - it's not a social networking protocol - it's something you can build a social networking protocol on.

It's stored on the server.

Names are big chunky things - you want to use them as sparingly as possible, and in general they should be "well known" - meaning a list function (which does exist as pub.srv.edsu.listings.names, though it's currently not something you can request permissions to) goes pretty hard against the grain of the protocol.

Essentially, you'd only want more than one name if there's different visibility requirements for different parts of your data, or you specifically want to disconnect the state of some of data from others. All of your structure (e.g. lists, trees, lookup tables, etc.) you want to keep in the blocks.

So in your case of having multiple notes, you'd likely want the name block to keep track of the block hash of each one (so, yes, exactly as you say). And correct, if you wanted all of them you'd need to do N block-get calls. However, due to pipelining and chaining, that's not as costly as it might sound: in terms of latency it should be costless, and in terms of bandwidth the overhead is a fraction of a percent if you're using full blocks (i.e. what you'd be doing if you're concerned about bandwidth).

A lot of this stuff is quite low level (think almost SCSI kind of low level) - it's meant to be abstracted away by libraries like basic-storage.js. In that case, having multiple notes is trivial - you put each note under a different storage key, and the library sorts out how to retrieve them efficiently and update them independently, even though they're all be under the same name.

In case you haven't run into it yet, I talk a little bit more about why names are meant to be used sparingly at https://edsu.org/use-cases/storage/

Very nice of you to say. Thanks for your interest, and best of luck :)

Thanks for your interest!

It is indeed stored on an Edsu server. The data format is trivial and completely specified, so switching providers should be straightforward (an Edsu app to do the data transfer would be easy to write, and I plan on writing one). If you're using your own domain (which is encouraged), from there you'd just update an A record to complete the transition, if not, you'd set up a transparent redirect on your old host to keep all your old links/permissions alive.

It's pretty old school in that if your provider loses a server and they're not doing replication/backups then there can be data loss. If they are, then a failover should work fine, with minimal loss and no corruption (the important stuff is atomic).

With localStorage, how it's used is up to the app (generally it'd only be used for keeping a token and maybe a block cache), so I wouldn't expect a loss of that to matter in most cases.

Sorry about that, and thanks for the bug report! I haven't been able to test with an iOS device yet - I'll get on that.

From a quick look through remoteStorage, it looks to me like it's is focused on storage (and so appropriately named!), while Edsu is more explicitly a foundation for applications.

One place this shows up is that Edsu has a permissions model where other people can read and interact with what you've stored (within very tightly defined parameters), which is the basis for writing multi-user apps like, say, a distributed Reddit or Slack.

They both have the same model regarding the ownership of data though. The biggest difference there is that Edsu uses a Merkle tree as the storage, like IPFS and git, which has a lot of consequences in terms of how it gets used.

Edsu's data store is content-addressed - i.e. everything is hashed and immutable from a storage perspective.

A great thing about this is it makes a lot of caching problems simply go away. So while there's no explicit support for local caches disconnecting and then re-connecting later, the underlying protocol gives any library wanting to implement this feature a lot of support for it.

Funnily enough, Edsu and git's data stores are very similar in that they're both Merkle trees. It means that they both have similar benefits of trivial checkpointing, automatically deduplicating revision histories, etc.

So if you squint enough, Edsu is kind of an "APIized" version of your setup :) So in addition to text files, it can also be structured application data.

Thanks for reading!

If an app wanted to be able to switch something from public to private or back, it'd get a token with write permissions for two names, one with the prv.* prefix, and one with the pub.* prefix. And that's a good thing: it makes it clear to the user that the app is requesting the ability to make the things that they write public.

It's still just one grant request, it's just that there'd be two line items instead of one. Also, names are simple pointers to blocks of data, so, for instance, if both names happen to be pointing to the same written piece, there's no duplication of data.

(Oh man, sorry, this ended up being an essay!)

Take, for instance, statically-hosted HTTP: if I'm, say, hosting at AWS and I want to switch to Netlify, it's trivial - I just change the place I upload my files to and switch the DNS. To me that's a complete success in decentralization - there's no hassle, no compatibility problems, no one other than me even knows that I've made the change.

So I don't see it as federated being a lost cause, I see it as a protocol needing to be an HTTP not an email.

For Edsu, it's something that I've considered at every design choice. For instance, the data storage format fully specified and trivial, and hopefully therefore trivially transferable. And there's features like transparent proxying/forwarding of usernames. But I'll absolutely admit I've not been able to completely mitigate it, aside from urging people to BYOD (Bring Your Own Domain).

However, a lot of it comes down factors that aren't the protocol itself. I'm hoping that Edsu's userbase is mostly just the open source community, with its consolidation-hostile "herding cats" nature. When Facebook first showed up, it struck me as a home page for people who didn't want to deal with HTML. So in a way, it was a bifurcation of HTML users, with the technical and non-technical people each going their own way. And in the technical people's world, HTML stayed a commodity.

SSH is another successful federated protocol - but only in the technical community. My family members don't use SSH, and that's just fine with me. That's one big advantage relative to alternatives that focus on social networking - Edsu is useful even if only a minority of people use it. And even email had 22 years before Gmail happened - if we've got 2 decades before we need to come up with a better thing than Edsu because the eternal September boat finally docked, I think that's fine.

So for sure, the gestalt at the moment is that full decentralization is the way to go. Edsu is a hedge - a bet against that. Federated is the devil we know, and there's successful examples of it avoiding its biggest problem. On the other hand, I think full decentralization's challenges are still unknown, and so A) it's unclear if it's any more resistant to centralization (a la Coinbase), and B) it might have other emergent bugbears that are an even bigger problem.

I think only time will tell.

It's only because FreeBSD is my desktop OS, and so I used kqueue during dev. I understand that there's a compatibility shim for Linux, so theoretically a port should be quite easy. The bulk of the code is in Rust, and that shouldn't need porting at all.

And thanks for the kind words :)

Thanks! One of the goals of Edsu is that it's simple enough that you can "telnet" in and just talk directly to the server like you can with HTTP or SMTP. It can seem like a trivial feature, but I've found that being able to speak a subset of a protocol from any language after just an hour or two of duct-tape hacking ends up being a big unsung feature.

And itgoon has it right - this was the most useful thing I could think to write that only took a single day (I had an self-imposed deadline to hit earlier this week).

Here's a discussion of more interested projects and how Edsu could be used in each (and its advantages and disadvantages):

https://edsu.org/use-cases/

I absolutely sympathize with the concern - but I don't believe it's an a priori solvable problem.

Bitcoin is theoretically fully decentralized, but when I sold mine off it took 2 days to sync the chain - I get why people use Coinbase. I'd still be running my own email server right now if it weren't for spam.

My point is that I think there's inflection points - when weaknesses in design or implementation become apparent - where centralization can get a foothold, and I don't think it's inevitable which course things take at those points. HTTP hosting, for example, has some big players, but it's still very much a commodity.

I realize that this is a crowded field, with lots of contenders, like yours. I think it's an important enough problem that it warrants parallel attempts, so that at least one of them sticks. Edsu picks a very specific strategy, which is that it's a compromise - it's not like, say, IPFS in its level of decentralization. For an app platform, I think there's challenges enough at any level of decentralization, and Edsu tries to b-line it straight there by being very orthodox and old-skool in nearly every other way. I thought it was a good bet, but only time will tell.

Props for the password resets, BTW. The importance of that feature is underappreciated :)