CondensationDB: A general-purpose distributed database with end-to-end security

[LukeEF]

Wow - great to see more work in this area. And interesting to see that the Swiss Government is supporting.

We’ve been working on a graph/document database with these kind of collaborative revision-control features over at TerminusDB (https://github.com/terminusdb/terminusdb). I think it is the wave of the future. And interestingly we also came out of a European Commission backed project.

Our approach to distribution is to use delta encoding and succinct data structures. We borrowed a fair few ideas from Git. Might be interested to read our storage layer white paper: https://github.com/terminusdb/terminusdb/blob/master/docs/wh...

Conflict free merges sound fantastic - not an easy road! Good luck.

[Malexik]

Thanks a lot, I heard about TerminusDB in the past, the capacity to do queries is interesting, that's something Condensation doesn't provide as it is on the client side and because at least now there are no tools to do it on the store.

I would be very happy to keep in touch and you have to give me the secret of these European funds.

[jarym]

This looks really neat, cannot wait to read the paper to see the details.

The other research db I saw recently I liked was sirix (https://github.com/sirixdb) - I can certainly see some room for cross-over between these two.

[Malexik]

That's cool for querying, I will have a deeper look thanks

[dfeiertab]

I'm not sure if I have fully gotten it correctly. It's a distributed database that syncs without conflict. So, cool for collaboration tools. What other things would you do with it?

[Malexik]

Yes, it excels at synchronization, you could just put a server synchronized and you have your backup, or let each user of an application have his own server which is synchronized with others (e.g., for a smart lock system). That's something pretty useful for privacy or if you have connectivity problems like in a mesh network with interruptions.

For the app itself, its really about getting end-to-end encryption be able to use the app while offline without loosing data.

[lifty]

People that are interested in a similar feature set should check out https://github.com/attic-labs/noms and the SQL fork of Noms, https://github.com/dolthub/dolt

[Malexik]

Interesting one, I think the difference is that you dont really control where your data is, it's like IPFS? I am surprised they managed to have queries, I will dig deeper into that thanks.

[expliced]

How does data synhchronization and conflict resolution work in CondensationDB? I expected to see something about OT or CRDT since it says it can be used for building collaborative applications such as Google docs (something like Docs is not possible without OT or CRDT).

[Malexik]

Exactly, look my reply to Fiahil a bit below.

[Fiahil]

Can you tell us a bit more about the "conflict free merge" ? Is it based on CRDTs ? How does it work ?

[Malexik]

Yes exactly, it's based on CRDTs and there is a strategy out there to mark the entries with a timestamp to figure out which one are the latest. An object may contain many entries and when they are read by the client they are just compared one by one to find the union, or the latest version.

The beauty of it is that the algorithm decide on how much entries to put in objects to ensure that only the data that is changed is sent on the network and compared on the other client. That's why we call it Condensation.

[Malexik]

Author and contributors are here for your questions

[cdbex]

Hi, I tried running the example from https://github.com/CondensationDB/Condensation-java/tree/mas... with the HTTP store from https://condensation.io/servers/php-files/. However, it seems that the later is outdated. Is there a functioning store available?

[ben509]

What's the story for deletion? Is that done through something like merge(record, delete_token) -> delete_token?

[bsenftner]

I find this quite interesting, but do all my development in the C/C++ ecosystem. Is there anything similar there?

[gritzko]

May you introduce yourself, please? Cause the accounts look like throwaways. That's weird.

[indymike]

Interesting. Would love to try this out, but I generally try to avoid Java for personal side projects. Is the plan to make a Javascript client or a full Javascript port?

[abdul169]

Nice jobs