0data appears to be a curated collection of five best-in-class libraries/frameworks that enable building SPA apps with user controlled data, rather than Big Tech controlled platforms and services.
We are developing both server and apps for [3] https://solidproject.org/ The effort is backed by Tim Berners Lee, with great auth libraries including dPop and PKCE token exchange
As someone dabbling with such an app right now, there are several hurdles if you want to exclusively use self sovereign identity.
1. Local data gets lost, so you need to sync for backup. But when your device gets lost or when you need to access it from elsewhere, how would you login to the said backup service? The most intuitive option is still a username and a password.
2. It's hard to transfer data from one device to another. And to asynchronously share data with friends and family. Solutions like ipfs exist, but in practice they rely on many centralized services (eg: for pinning). And besides, securely sharing private data is not an option on any of them.
3. Lay people being able to safely handle cryptographic keys is an unrealistic assumption.
So IMHO, perfect 0-data is not going to happen. It used to work earlier (pre-internet) because we had only one device, didn't have a lot of media, and could carry floppy disks for the rare occasion.
What I think would make a difference is: browser apps being able to do TCP/UDP (with safe-guards) and sandboxed local disk-access. This allows web apps to do things like SSH, and login to the vast storage infrastructure that already exists - such as GitHub. Data remains fully under your control and becomes portable. Git IMHO is a better path to where Solid wants to go. Especially once SHA256 commit hashes land - to become your personal, consensus-less blockchain.
The internet also fundamentally changed how people perceive of their data being accessible. When I was trying to give advice to someone after their Chromebook bit the dust (from being dropped too many times), doing things like using a thumbdrive to maintain portable copies of the local data on the device... It was met with a can of worms of further questions starting with:
a) what's a thumbdrive?
and
b) what's local data?
Each answer resulted in even more questions. Why isn't it in the cloud? What do you mean I'd have to copy it again? I can run out of room?
I could understand needing to get across to some people that not all data ends up synced (eg, saved memes are not necessarily automatically backed up to Drive)... but reaching a point to where people could be completely unaware of USB storage media actually shook me to my core.
People now expect it all to just "be there" after they log in.
If you're looking for secure sharing of private data on IPFS you should check out Peergos [0][1][2]. Disclaimer - I'm a co-founder. The model there is each user needs at least one Peergos instance storing their data, normally their home-server. We've designed it so that people should never have to see an encryption key, or cryptographic hash, nor know what pinning is. As far as apps go, we plan to build a sandbox and api in the browser for running untrusted apps served directly from Peergos which you can grant fine grained permissions to your data in Peergos. The sandbox here would be designed so an app couldn't even exfiltrate data via loading an external img tag.
A few years ago (~2017) I built an app that uses a hybrid backend. As a user you can choose if you want to use a WebDAV server of your own or simply the builtin AWS backend. That way, people who care about their data can use their own server (e.g. a Nextcloud) as a backend and everyone else doesn't have to bother.
Sadly the code quality is prototypical at best and while I still use the app multiple times per week, I never released it and have quite a few leanings which I want to implement (like rewriting the sync logic in a separate library, changing the server side data model to improve performance, completely reorganize the user interface and add more features to the app), but aren't coming to due to other priorities.
However, what I also learnt is that it is possible to build something like it, even if your use-case includes advanced topics like offline collaboration. My hope is that one day we will have a library that make these things for developers as easy as firebase and every app will have it out of the box.
The data (in solid for example) isn't local - it's just controlled by the user. The key point is that the user's data is decoupled from your service - it's not necessarily decentralised.
Backups are handled (by the user) - the user can purchase cloud storage through a provider (like https://inrupt.com/products/enterprise-solid-server/, though there are other free services - or you can host your own wherever you want). Users don't have to safely handle cryptographic keys or deal with device syncing.
For (1), why not have all your devices sync up when they are on your local wifi network? If your phone automatically backs up to your NAS, and syncs with your home desktop, your data should always be up to date and backed up in multiple places. Then if you really have to sync while not being at home, you could do something over ssh/using a vpn.
3. The crypo space is actually doing this fairly well: end users remember a 12-word seed phrase, and that can be used to recover the multiple private keys that make up a “wallet” (bad term since wallet already means something, but that’s what the hardware vendors went with).
Otherwise I’d say it’s clear you’re in the space and I agree with your assessment.
For the vast majority of users, isn't having solid export options enough, or even better than this? Your average non-techie isn't going to want to configure each app to get syncing with NextCloud or Dropbox working just right, they just want to perform a task with the app and move on with their lives, regardless of where the data lives. And with an export option, if the app dies, you've still got your blog posts or pictures or drawings or whatever it is you were using the app for.
I think being in this bubble of working in and thinking about tech 24/7 can often make us miss the point - these apps are just meant to be tools to do a task.
No, because exporting is not automatic. Most users won't do it.
It's true that most users don't want to configure remote storage for each app. That's solvable though. There should be a way to store your remote storage preferences and just give each app permission to store there in one click.
Pre cloud, this was just apps and local files. Just because an app stored a file on your hard drive didn't mean you could do anything useful with it outside that app, if it was a proprietary format. I'm not sure the typical user misses dealing with files.
While some people care about privacy, huge majority doesn’t (even if they say they do), as you can see be various services that have billions of users.
The only way to make privacy happen is if it’s dead simple, works out of box, and not only doesn’t regresses your experience, but meaningfully enhances it. I’ve failed to see projects that would do that so far, and same here - it adds extra complexity for end user, with no tangible benefits to the experience.
Im working on something that also have the goal to give the power back to the user in terms of control, but i've also understand it the same way you do.
That in the end the platform have to provide a better experience to the end user and also give more power/flexibility to the developer, otherwise no one will use it.
That's the reason i think it requires a lot more work to make this feasible. Only "iterational innovation" wont do it.
I hope that what i'm almost about to launch here, might reach those goals.
AND it goes against economic incentives for the typical dev (i.e. make migration harder to engender lock-in)! But, for devs who have portability and openness as priorities (volunteers and altruists, in other words) it seems very helpful to have a precompiled model of how to build apps that respect these priorities.
people don't care about privacy, but at the same time if facebook (or whatever) didn't mine your information when you signed up first and you had to go deep into the settings to turn on tracking, nobody would care about that either. they just care about being able to do xyz for free.
do you know about the safe network? that seems relatively easy. the way its being designed is that you will have to log-in to access the network (which does add complexity compared to now) but the idea will be that you can sign up for websites and services using your log in ID. so you dont have to share your name or email address and there will be no need for a service to store your password
The message is good but the name makes no sense, and is almost the opposite of what’s described. In a traditional web app, the client has zero data, or just some cache. But in a "zero data app", in the way they mean it, the app has all the data, so certainly not "zero".
Go explain the concept to someone unfamiliar with it with a nonsensical name like this.
when looking at different data management approaches i see the following:
- third party manages encryption keys and data custody, users manage none (dropbox, G drive et al)
- third party manages data custody, users manage encryption key (e2e encryption, icedrive, pcloud, 1password etc)
- third party manages none, user manages data custody (and eventually encryption) (0data and more generally "storing files in your computer")
the 0data is just like going back to what we used to do a decade or two ago and we all know it has its drawbacks (data can be lost, stolen, corrupted, difficult to move)
the most popular data custody model where a third party has total control over our data but we don't also has its drawbacks (data misuse, data breach, data mining, data transfer etc)
the second approach which i am surprised not a lot of providers adopt is where we delegate data custody to a third party but we still have e2e encryption over the data contents also has its drawbacks (data can be lost if keys are lost) but it's what i think it's more compelling compared to this 0data philosophy.
One large reason that option 2 isn't great is because people will forget or lose their key. And at that point you'll have a very angry customer when you tell them that you can't let them in or give them their data back (or at least not any useful data). People don't truly understand just how impossible it is to regain access to their data.
I can't speak for all projects listed here, but usually with option 3 you can also choose to e.g. pay someone to do the hosting for you and effectively get option 2, provided that data is not stored locally and can be encrypted.
I'd like to see this applied to online advertising, making cross-site tracking against the law and forcing advertisers to bid per real-time request for user data and the money is paid to the user themselves. Currently advertising platforms make more money by allowing advertisers to better target their ads. I have nothing against this as long as I'm paid for access to my data at the price I think is right (might be MAX_INT, to each their own).
1. Each user would have a "data pod" configured in their browser, storing has as much structured information about the user as the user wants (can be empty, or it can have all the structured data fields you want to insert).
2. The user can update any fields at any point and how access to the data pod is done.
3. The user can setup a BID MINIMUM or MARKET value for access to its data pod, perhaps even having different bid values for each set of data. For example, an advertiser wants to know my name? $0.000001 per request. You want to know my address and what TV shows I like? $0.001 per request. Want my bank data? $1000 per request.
Further this data could be authenticated cryptographically by certain authoritative entities. My government could authenticate that I am indeed from country A, and my pod's data would be signed by them (netflix and spotify could authenticate my media consuming history, etc). From that point onwards advertisers know that this field has been validated and can be incentivised to pay more. This should get rid of the incentives where everyone will self-report as being a US citizen just so their requests have larger bids.
>My government could authenticate that I am indeed from country A, and my pod's data would be signed by them
This is gonna happen sooner or later, but I really don't like it. Once most users have a cryptographically signed national ID on their PC, a lot more websites will require you to provide it. Sites like Netflix that region lock media will force you to show your ID just to sign up
Won't be long before companies get away with the invasive crap they tried and failed to do in the past: "Users Revolt Over Blizzard's Requirement Of 'Real Names' In Forum Comments": https://www.techdirt.com/articles/20100708/03054610123.shtml
The problem is someone making good money will likely not bother with that system - they will just say “my info is not for sale” instead of trying to figure out settings which would earn them them 0.1% of their regular income.
So this system will mostly be used by the people with little disposable income, enthusiasts who enjoy gaming the system, and scammers with tricks to maximize earnings. Those people are not going to spend much money on buying expensive products, so the advertisers will find out that this bidding thing is just not bringing any results.
Storing information in the URL itself is a grossly underutilized way to allow users to control their own data. If data is stored in the URL, there is often no need to send it to a server. And since URLs are so widely-supported, sharing the data is inherently easy.
For example, it takes about 3 lines of JavaScript to store and retrieve a base64-encoded, stringified JSON object in the URL fragment (the part after the # – https://en.wikipedia.org/wiki/URI_fragment).
This is great. For wide scale adoption of this (or any other tech for that matter) there has to be a way for people to make money. What are some business models that this enables?
Such sites are a distraction from the real problem that the non-tech users have:
They want the app to work forever, even if the underlying cloud storage service gets killed.
Hence, being able to transparently transfer your data to another underlying storage service without the app changing at all is what most normal people want.
They want a box saying "This app's storage expires on 20th of April. Please choose where you want to transfer your data without disruption of service (and where it will be hosted from now on): Dropbox, Google Cloud, OneDrive, pDrive, Mega, ..."
I've spoken with a lot of regular folks. They don't care about privacy or self-hosting or decentralization at all. They never will. To them technology is a tool with which they make their lives easier. We should take a page from them. :)
(Although to make it 100% clear, I do care about privacy and anonymity very much; but it shouldn't be idolized and put on a pedestal, otherwise people like me will be made irrelevant with time -- one could argue this has already happened).
---
TL;DR: Sites strictly advocating for some tech principles lost most of my vote a while ago. We the techies get too distracted by our own shinies and must make a come-back to pragmatism and serving the regular people. It's OK to code stuff as hobbies but promoting them as universal values gets a "nope" from me. And that comes from a guy who wants to retire at his own house with 3 internet connections, and work on Tor-on-steroids and automatic replication of encrypted data until he dies.
I believe Obsidian (https://obsidian.md) follows this principle. That app, and ones like it (Foam the open source VSCode version) have introduced me to the field of Personal Knowledge Management. I believe these two tools help one lead a wiser life. Regardless of outcome, they’ve made learning and thinking fun again, at least for me.
That was one of the principles that made me enjoy https://www.hashapass.com, a password manager with zero storage. Not even local storage, because then you'd have to deal with backups.
I'm glad to see so many applications follow such a principle, though whenever I use something like it I remain scared of not doing backups right.
Used to use http://www.nullpass.org which is the same thing but open source. Ran in to issues with websites having varying password requirements that sometimes the generated one did or did not meet, and then I wouldn't remember which alterations were needed for it the next time I needed the passwords for the sites.
I've been thinking of starting a community (forum + wiki, etc) based around similar principles, but I haven't settled on what to call it yet. The core ideas are self-hosting, privacy, control of data, open source clients, etc.
The best term I've come up with so far is "data ownership". I like this, but the principles are broader than just data storage. Having control over your computation can also important.
From the end user’s perspective, what would the advantages of an app without centralized storage, like these, have vs. an app with E2E encryption and traditional centralized storage? If the end user’s data is encrypted on the service provider’s side and cannot be read without the user providing a key, doesn’t that achieve the same level of privacy from the user’s perspective?
The saved "hub" (file/database) is just a JSON formatted text file renamed to `.vha2` and all the images are stored as JPG in a folder near the file. Yay!
[+] [-] sradman|5 years ago|reply
1. Autonomous Data
2. remoteStorage
3. Solid Project
4. Unhosted
5. Fission
[1] https://noeldemartin.github.io/autonomous-data/introduction....
[2] https://remotestorage.io/
[3] https://solidproject.org/
[4] https://unhosted.org/
[5] https://fission.codes/
[+] [-] gibsonf1|5 years ago|reply
[+] [-] jeswin|5 years ago|reply
1. Local data gets lost, so you need to sync for backup. But when your device gets lost or when you need to access it from elsewhere, how would you login to the said backup service? The most intuitive option is still a username and a password.
2. It's hard to transfer data from one device to another. And to asynchronously share data with friends and family. Solutions like ipfs exist, but in practice they rely on many centralized services (eg: for pinning). And besides, securely sharing private data is not an option on any of them.
3. Lay people being able to safely handle cryptographic keys is an unrealistic assumption.
So IMHO, perfect 0-data is not going to happen. It used to work earlier (pre-internet) because we had only one device, didn't have a lot of media, and could carry floppy disks for the rare occasion.
What I think would make a difference is: browser apps being able to do TCP/UDP (with safe-guards) and sandboxed local disk-access. This allows web apps to do things like SSH, and login to the vast storage infrastructure that already exists - such as GitHub. Data remains fully under your control and becomes portable. Git IMHO is a better path to where Solid wants to go. Especially once SHA256 commit hashes land - to become your personal, consensus-less blockchain.
[+] [-] techrat|5 years ago|reply
a) what's a thumbdrive?
and
b) what's local data?
Each answer resulted in even more questions. Why isn't it in the cloud? What do you mean I'd have to copy it again? I can run out of room?
I could understand needing to get across to some people that not all data ends up synced (eg, saved memes are not necessarily automatically backed up to Drive)... but reaching a point to where people could be completely unaware of USB storage media actually shook me to my core.
People now expect it all to just "be there" after they log in.
[+] [-] ianopolous|5 years ago|reply
[0] https://peergos.org
[1] https://book.peergos.org
[2] https://github.com/peergos/peergos
[+] [-] arendtio|5 years ago|reply
Sadly the code quality is prototypical at best and while I still use the app multiple times per week, I never released it and have quite a few leanings which I want to implement (like rewriting the sync logic in a separate library, changing the server side data model to improve performance, completely reorganize the user interface and add more features to the app), but aren't coming to due to other priorities.
However, what I also learnt is that it is possible to build something like it, even if your use-case includes advanced topics like offline collaboration. My hope is that one day we will have a library that make these things for developers as easy as firebase and every app will have it out of the box.
[+] [-] inglor|5 years ago|reply
Backups are handled (by the user) - the user can purchase cloud storage through a provider (like https://inrupt.com/products/enterprise-solid-server/, though there are other free services - or you can host your own wherever you want). Users don't have to safely handle cryptographic keys or deal with device syncing.
[+] [-] GhostVII|5 years ago|reply
[+] [-] joshspankit|5 years ago|reply
Otherwise I’d say it’s clear you’re in the space and I agree with your assessment.
[+] [-] 600frogs|5 years ago|reply
I think being in this bubble of working in and thinking about tech 24/7 can often make us miss the point - these apps are just meant to be tools to do a task.
[+] [-] ajb|5 years ago|reply
It's true that most users don't want to configure remote storage for each app. That's solvable though. There should be a way to store your remote storage preferences and just give each app permission to store there in one click.
[+] [-] satyrnein|5 years ago|reply
[+] [-] addicted|5 years ago|reply
People could do a lot with their photos, videos, music and movie files that were stored in standardized formats on their hard disks.
Witness the plethora of music and movie players, the tools formed around images and video browsing and manipulation, etc.
Even non standardized stuff like word and excel documents could be shared, backed up, organized using these files.
[+] [-] altcognito|5 years ago|reply
A simple "backup/restore to/from <x>" service/computer often will suffice.
[+] [-] justapassenger|5 years ago|reply
The only way to make privacy happen is if it’s dead simple, works out of box, and not only doesn’t regresses your experience, but meaningfully enhances it. I’ve failed to see projects that would do that so far, and same here - it adds extra complexity for end user, with no tangible benefits to the experience.
[+] [-] oscargrouch|5 years ago|reply
That in the end the platform have to provide a better experience to the end user and also give more power/flexibility to the developer, otherwise no one will use it.
That's the reason i think it requires a lot more work to make this feasible. Only "iterational innovation" wont do it.
I hope that what i'm almost about to launch here, might reach those goals.
[+] [-] asdfasgasdgasdg|5 years ago|reply
[+] [-] mackrevinack|5 years ago|reply
do you know about the safe network? that seems relatively easy. the way its being designed is that you will have to log-in to access the network (which does add complexity compared to now) but the idea will be that you can sign up for websites and services using your log in ID. so you dont have to share your name or email address and there will be no need for a service to store your password
[+] [-] nicbou|5 years ago|reply
However, I completely agree with the second paragraph.
[+] [-] laurent123456|5 years ago|reply
Go explain the concept to someone unfamiliar with it with a nonsensical name like this.
[+] [-] ethbr0|5 years ago|reply
All terms aren't coined exclusively to speak to developers. ;)
[+] [-] jwolfe|5 years ago|reply
[+] [-] skeeks|5 years ago|reply
[+] [-] AbuAssar|5 years ago|reply
[+] [-] haggy|5 years ago|reply
[+] [-] xyst|5 years ago|reply
[+] [-] hankchinaski|5 years ago|reply
- third party manages encryption keys and data custody, users manage none (dropbox, G drive et al)
- third party manages data custody, users manage encryption key (e2e encryption, icedrive, pcloud, 1password etc)
- third party manages none, user manages data custody (and eventually encryption) (0data and more generally "storing files in your computer")
the 0data is just like going back to what we used to do a decade or two ago and we all know it has its drawbacks (data can be lost, stolen, corrupted, difficult to move)
the most popular data custody model where a third party has total control over our data but we don't also has its drawbacks (data misuse, data breach, data mining, data transfer etc)
the second approach which i am surprised not a lot of providers adopt is where we delegate data custody to a third party but we still have e2e encryption over the data contents also has its drawbacks (data can be lost if keys are lost) but it's what i think it's more compelling compared to this 0data philosophy.
[+] [-] ziml77|5 years ago|reply
[+] [-] Vinnl|5 years ago|reply
[+] [-] dannyw|5 years ago|reply
If you have terabytes of data to store, it can work out significantly cheaper.
[+] [-] bajsejohannes|5 years ago|reply
[+] [-] vasco|5 years ago|reply
1. Each user would have a "data pod" configured in their browser, storing has as much structured information about the user as the user wants (can be empty, or it can have all the structured data fields you want to insert).
2. The user can update any fields at any point and how access to the data pod is done.
3. The user can setup a BID MINIMUM or MARKET value for access to its data pod, perhaps even having different bid values for each set of data. For example, an advertiser wants to know my name? $0.000001 per request. You want to know my address and what TV shows I like? $0.001 per request. Want my bank data? $1000 per request.
Further this data could be authenticated cryptographically by certain authoritative entities. My government could authenticate that I am indeed from country A, and my pod's data would be signed by them (netflix and spotify could authenticate my media consuming history, etc). From that point onwards advertisers know that this field has been validated and can be incentivised to pay more. This should get rid of the incentives where everyone will self-report as being a US citizen just so their requests have larger bids.
What have I missed?
[+] [-] NavinF|5 years ago|reply
This is gonna happen sooner or later, but I really don't like it. Once most users have a cryptographically signed national ID on their PC, a lot more websites will require you to provide it. Sites like Netflix that region lock media will force you to show your ID just to sign up
Won't be long before companies get away with the invasive crap they tried and failed to do in the past: "Users Revolt Over Blizzard's Requirement Of 'Real Names' In Forum Comments": https://www.techdirt.com/articles/20100708/03054610123.shtml
[+] [-] theamk|5 years ago|reply
So this system will mostly be used by the people with little disposable income, enthusiasts who enjoy gaming the system, and scammers with tricks to maximize earnings. Those people are not going to spend much money on buying expensive products, so the advertisers will find out that this bidding thing is just not bringing any results.
[+] [-] bluesign|5 years ago|reply
[+] [-] dt3ft|5 years ago|reply
I tried one of the projects listed there, and it required me to log in. ¯\_(ツ)_/¯
[+] [-] blain|5 years ago|reply
[+] [-] jstrieb|5 years ago|reply
For example, it takes about 3 lines of JavaScript to store and retrieve a base64-encoded, stringified JSON object in the URL fragment (the part after the # – https://en.wikipedia.org/wiki/URI_fragment).
[+] [-] victor106|5 years ago|reply
[+] [-] ssss11|5 years ago|reply
[+] [-] pdimitar|5 years ago|reply
They want the app to work forever, even if the underlying cloud storage service gets killed.
Hence, being able to transparently transfer your data to another underlying storage service without the app changing at all is what most normal people want.
They want a box saying "This app's storage expires on 20th of April. Please choose where you want to transfer your data without disruption of service (and where it will be hosted from now on): Dropbox, Google Cloud, OneDrive, pDrive, Mega, ..."
I've spoken with a lot of regular folks. They don't care about privacy or self-hosting or decentralization at all. They never will. To them technology is a tool with which they make their lives easier. We should take a page from them. :)
(Although to make it 100% clear, I do care about privacy and anonymity very much; but it shouldn't be idolized and put on a pedestal, otherwise people like me will be made irrelevant with time -- one could argue this has already happened).
---
TL;DR: Sites strictly advocating for some tech principles lost most of my vote a while ago. We the techies get too distracted by our own shinies and must make a come-back to pragmatism and serving the regular people. It's OK to code stuff as hobbies but promoting them as universal values gets a "nope" from me. And that comes from a guy who wants to retire at his own house with 3 internet connections, and work on Tor-on-steroids and automatic replication of encrypted data until he dies.
[+] [-] for_i_in_range|5 years ago|reply
[+] [-] ridaj|5 years ago|reply
I'm glad to see so many applications follow such a principle, though whenever I use something like it I remain scared of not doing backups right.
[+] [-] austhrow743|5 years ago|reply
[+] [-] pietromenna|5 years ago|reply
[+] [-] ssddanbrown|5 years ago|reply
Login is not required by the way, You can choose "Device" or "Decide Later" when presented the save options.
[+] [-] anderspitman|5 years ago|reply
The best term I've come up with so far is "data ownership". I like this, but the principles are broader than just data storage. Having control over your computation can also important.
Any suggestions for a better term?
[+] [-] peterthehacker|5 years ago|reply
[+] [-] ForHackernews|5 years ago|reply
[+] [-] yboris|5 years ago|reply
https://github.com/whyboris/Video-Hub-App <-- MIT open source
The saved "hub" (file/database) is just a JSON formatted text file renamed to `.vha2` and all the images are stored as JPG in a folder near the file. Yay!