A number of years back, I wrote an OT implementation in Clojure/ClojureScript as a hobby project [1] and learned a lot along the way. I even extracting out the core client/server implementations to their own package [2].
It's an incredibly fun topic, and one that's also really challenging. It also gave me first-hand experience with fuzz-testing where I could simulate all kinds of concurrent-edit conflicts and ensure both clients came out with the same end-result. While the end client/server implementations are working, it was a lot more effort than I anticipated building a full-fledged editor on top. I was too stubborn at the time to attempt to integrate with existing RTEs though, so the most notable part of the project was the core lib, not the end product.
For those interested in implementing it for themselves, I can't recommend this piece [3] by @djspiewak enough.
I found this recent paper highly enlightening: “Real Differences between OT and CRDT for Co-Editors”–
Chengzheng Sun, David Sun, Agustina, Weiwei Cai, October 2018.
It’s well written imo. The conclusions chapter is very good.
“In this work, we have critically reviewed and evaluated representative OT and CRDT solutions, with respect to correctness, time and space complexity, simplicity, applicability in real world co- editors, and suitability in peer-to-peer co-editing. The evidences and evaluation results from this work disprove superiority claims made by CRDT over OT on all accounts.”
And it is also argued that OT is a more direct approach to the fundamental issues of realtime multiuser editing, and that that directness brings effectiveness.
Surprised the author didn't mention Quill.js, which works really well with ShareDB, and is fully open source. We've been doing a lot of fun stuff with it - here's a talk I gave recently as a job talk: https://www.youtube.com/watch?v=gN37rJRmISQ. (It worked, I got hired :)). As a side project, I'm working on a ShareDB backed wiki, where we can use rich text for editing pages, but also other components, like spreadsheets or mindmaps, and everything supports live editing - I think it would be amazing for classrooms, hackathons, any kind of meeting etc. Would love to talk with anyone interested in collaborating. I have been waiting for a few features to complete to make a proper demo video, but this early one shows off some of the ideas: https://www.youtube.com/watch?v=9-lU-in3ydc (https://github.com/chili-epfl/FROG)
Thanks for the shoutout! We are actively developing on ShareDB (https://github.com/share/sharedb) and if anyone is really interested in this, please reach out to me (https://github.com/nateps). Also, Lever is looking to hire someone to work full time on our internal + open source frameworks including ShareDB.
I'm just curious as it doesn't mention anything about it being a wiki in the readme (it's also a bit unclear how/what to use it for from the description).
What many people not fully grasp, at first sight, is that offline support for a collaborative editor is at least an order of magnitude more complex than just building a collaborative editor. In online collaborative environments, solving conflicts is relatively simple, as the changes are supposed to be small. But when two (or more) people are working on the same document without synching it for hours or days, the merge strategy has to be extremely good to come up with anything remotely intended (assuming the authors had a common intention).
So just because your favorite editor can be used offline and it has an (online) collaboration feature, does not mean it is capable of being an offline-capable collaborative editor.
This is where differential synchronization shines (basically git). While OT/CRDT are really good for syncing small edits, they suck at being eventually consistent (with intentions preserved) on very large edits.
We make a powerful collaborative word processor for the browser (Zoho Writer https://writer.zoho.com) with complete offline support. This is one problem we'd like to solve in the long term. One way to solve this is to fallback to differential sync when syncing huge offline edits - and present the changes as tracked-changes (underlined and striked-out) to the user. This way the user has the power to resolve the conflicts, instead of the app messing it up by itself.
I'd like to hear if there are better ways to sync large offline edits, with the intentions preserved.
Of course, it depends on the collaborative editing solution. When it comes to OT -- if the implementation is correct, it doesn't really matter how many operations are queued for synchronising. So, it doesn't really matter how long users were de-synced when creating their content. But this is only theory and it applies more to technical side as there might be some semantical problems (intention preservation).
I believe the same is true for CRDT but I am not sure.
So, on one hand - if the editor is working fine for small batch of changes, it should also work fine for big batch of changes. The reality is often more harsh, though, and full of edge case scenarios.
Indeed, the space is growing quickly. I run a company that provides a backend for real-time collaboration, so we necessarily stay on top of what's out there. There's a reason the server-side tech tends be paid: it is extraordinarily difficult to provide guaranteed eventual consistency of data at low latency. The CKEditor guys (and us, for that matter) have put YEARS of development effort into their solutions.
We're working on both offline mode and a generic rich-text data model with support for the major editors.
If anyone's interested, I'd happily do some Q and A here.
I appreciate that you mentioned a competitor without shamelessly plugging your own product, but out of curiosity, and if you don't mind sharing, what editor do you work on?
Are you working only with editors running inside web browsers or also native ones? What I would like to have is something that bridges two native editors, let's say emacs and VS Code, and let's a team share the editor (whatever each developer's preferred editor is) and not only the screen on Slack or similar. That would be great for remote pair programming. I guess there would be a protocol for selecting the buffer/tab and opening files (with local approval to prevent remote file reads/writes).
Do you guys rely on a central authority approach like Prosemirror does? I'm desperately looking for something that's somewhat stateless or at least replicable (even better if shardable) on a document basis.
Thanks for the component you provided for shared cursors in text areas (https://github.com/convergencelabs/html-text-collab-ext). I've been using it with ShareDB, and have been working on adapting it to work with single text inputs, which has been surprisingly difficult (mainly dealing with scroll/overflow behaviour).
I love your description of "the dive". I am happy to see I am not the only one experiencing these recurring devouring interests that bring me much much deeper than I would like to go until I am starting to fork a github repo at 2am wondering "what the hell am I doing?"
And that article is tangentially related to a recurring interest of mine (shared editable 3d universe in which you can code collaboratively). If you happened to have an article comparing the merit of several server architectures/algorithms for collaborative text edition, I think many things can be related. I am now reading about CRDT (What the hell am I doing?) and that's something I may use to edit a scene graph collaboratively!
Hey thanks for that. :) Diving is super fun but also exceptionally exhausting. But often it's impossible to 'load' a complex problem in your head if you don't dive in for hours. Would be nice if there was a less stressful way.
You've probably heard of it already, but if not you should check out the game Cube/Sauerbraten. It's an open source quake-style fps game, with its level editor integrated right into the game itself, and you can edit levels collaboratively with other players at the same time.
I would be very appreciative if Google made a stripped down version of Google Docs for plaintext editing.
There are a bunch of basic integration features that would also be fantastic, but if the product was literally just Google Docs, minus the WYSIWYG bits, and all plaintext w/ maybe some syntax highlighting thrown in, I'd be super happy.
That's basically what I'm trying to build - we're starting with Quill.js and ShareDB, adding extensions, configuring, and then adding a plugin system based on React, which can take arbitrary components all based on ShareDB sync (think spreadsheets, forms, but I'd also love to integrate the Pyiodide stuff to have runnable Python code inside a rich text document, etc). We built this as part of a synchronous collaborative learning platform, but I'm currently trying to extract the key components so that it can be a stand-alone open source library that others can help build on. Very early prototype: https://www.npmjs.com/package/@chilifrog/reactive-rich-text
> Google Docs, minus the WYSIWYG bits, and all plaintext w/ maybe some syntax highlighting thrown in
That is the subset of features you want. Everyone else wants a slightly different subset of features. Add all of those subsets together and you end up with the total feature set of Google Docs.
Another product that should be mentioned here is OX Documents [1] by Open-Xchange. They have a web-based Office suite with MS file types (docx, xlsx, etc.) that has a collaboration feature. However, the product is not 100% Open-Source (CCSA:NC) [2]. The source code is accessible, but you are not allowed to use it in a commercial product.
IANAL, but if you want to use the product yourself (even as a company) that should be just fine, but you are neither allowed to host the product for someone else (commercially) nor to created a product based on their frontend code.
If you're looking for a plaintext collaborative editor, then Quill and a bit of glue-code is all you need. I believe there's even example code that shows how to maintain the document on a server.
We went with CKEditor5 with a commonmark backend and I cannot recommend it more highly. commonmark data is converted upon initially opening and when closing the editor, allowing all native functionality to happen efficiently.
However I do note that we do not yet have a production ready backend for collaborative editing. I would suggest to use their service if possible, since we're developing an open source product it would not be an option for us.
There is a quite leaning curve when implementing new plugins due to the strict separation of model and data (commonmark in our case) and user-facing views.
CKEditor 5 looks amazing (from tech to docs), but that lack of an available backend is a bummer. Do you know if there's anything out there that would work as an initial/hacky/example backend? How do you plan to tackle this?
I was able to get real-time collaborative text editing to work on one of my side projects by combining quill.js[1], ot.js[2], and firebase[3]. This was a few years ago when there weren't any fully open-source options. I'm in the process of switching out firebase for an elixir/phoenix backend[4]. I'm really glad for this article because it captures the exact feelings that I have.
This was one of the project idea I made myself work on during this April. The motivation being, "how hard is it to make a real-time code editor". Long story short, it is far more complicated than I had imagine and to complete in less than a week. Implementing the CRDT itself, the logic itself is challenging and than putting it down to code is another story.
From my understanding, an implementation of logoot[1] is built to create the skeleton of a collaborative environment. From there on, you have the building block to make other things work with it, such as collaborative canvas.
I built one using Socket.io on Angular and used Ace Editor own implementation of CRDT
I wonder why would one make a headline "Open source collaborative text editors", and then write about online editors which are neither fully open-sourced, nor fully collaborative. And then stop with them, without even mentioning solutions that are both truly real-time collaborative and truly open source.
You should try https://www.fiduswriter.org. It does have realtime collaboration, export to ODT/DOCX/Epub/LaTeX, tracked changes, WYSIWYG interface, etc. and in the latest version even a document template editor. I think relatively speaking it's quite developed in comparison to some of the other word processors presented here. It does include both frontend and backend parts to make collaboration work. (I did a lot of the programming over the past 7 years.)
So, lots of open-source UIs that use closed-source servers for collaborative editing?
This figures. This is one of the ways 'open source' companies can make money.
So, the author sets about making an open-source collaboration server that inter-operates with the existing open-source editor?
Kudos!
But now the company can retaliate by adding features or inconsistencies to the closed source server and its integration into the UI, making the open-source server forever out-of-date and playing catch-up.
SubEthaEdit is a collaborative real-time editor designed for Mac OS X. The name comes from the Sub-Etha communication network in The Hitchhiker's Guide to the Galaxy series.
Not sure why you got downvoted. SubEthaEdit was magical back when - here is the link to the github repo for the recently released source: https://github.com/subethaedit/SubEthaEdit
[+] [-] jahfer|7 years ago|reply
It's an incredibly fun topic, and one that's also really challenging. It also gave me first-hand experience with fuzz-testing where I could simulate all kinds of concurrent-edit conflicts and ensure both clients came out with the same end-result. While the end client/server implementations are working, it was a lot more effort than I anticipated building a full-fledged editor on top. I was too stubborn at the time to attempt to integrate with existing RTEs though, so the most notable part of the project was the core lib, not the end product.
For those interested in implementing it for themselves, I can't recommend this piece [3] by @djspiewak enough.
[1] https://github.com/jahfer/ot [2] https://github.com/jahfer/othello [3] http://www.codecommit.com/blog/java/understanding-and-applyi...
[+] [-] hashhar|7 years ago|reply
http://archagon.net/blog/2018/03/24/data-laced-with-history/
[+] [-] AnthonBerg|7 years ago|reply
Arxiv meta: https://arxiv.org/abs/1810.02137
PDF: https://arxiv.org/pdf/1810.02137.pdf
It’s well written imo. The conclusions chapter is very good.
“In this work, we have critically reviewed and evaluated representative OT and CRDT solutions, with respect to correctness, time and space complexity, simplicity, applicability in real world co- editors, and suitability in peer-to-peer co-editing. The evidences and evaluation results from this work disprove superiority claims made by CRDT over OT on all accounts.”
And it is also argued that OT is a more direct approach to the fundamental issues of realtime multiuser editing, and that that directness brings effectiveness.
HN discussion: https://news.ycombinator.com/item?id=18191867
[+] [-] houshuang|7 years ago|reply
[+] [-] nateps|7 years ago|reply
[+] [-] nused|6 years ago|reply
Here is a Quill.js coediting demo, without using a ShareDB backend: https://codepen.io/dnus/pen/OojaeN
[+] [-] VvR-Ox|7 years ago|reply
Is the FROG repo you linked the repo of the wiki?
I'm just curious as it doesn't mention anything about it being a wiki in the readme (it's also a bit unclear how/what to use it for from the description).
[+] [-] arendtio|7 years ago|reply
So just because your favorite editor can be used offline and it has an (online) collaboration feature, does not mean it is capable of being an offline-capable collaborative editor.
[+] [-] lewisjoe|7 years ago|reply
We make a powerful collaborative word processor for the browser (Zoho Writer https://writer.zoho.com) with complete offline support. This is one problem we'd like to solve in the long term. One way to solve this is to fallback to differential sync when syncing huge offline edits - and present the changes as tracked-changes (underlined and striked-out) to the user. This way the user has the power to resolve the conflicts, instead of the app messing it up by itself.
I'd like to hear if there are better ways to sync large offline edits, with the intentions preserved.
[+] [-] scofalik|7 years ago|reply
Of course, it depends on the collaborative editing solution. When it comes to OT -- if the implementation is correct, it doesn't really matter how many operations are queued for synchronising. So, it doesn't really matter how long users were de-synced when creating their content. But this is only theory and it applies more to technical side as there might be some semantical problems (intention preservation).
I believe the same is true for CRDT but I am not sure.
So, on one hand - if the editor is working fine for small batch of changes, it should also work fine for big batch of changes. The reality is often more harsh, though, and full of edge case scenarios.
[+] [-] anthonys|7 years ago|reply
[+] [-] alalonde|7 years ago|reply
We're working on both offline mode and a generic rich-text data model with support for the major editors.
If anyone's interested, I'd happily do some Q and A here.
[+] [-] blotter_paper|7 years ago|reply
[+] [-] pmontra|7 years ago|reply
[+] [-] tjungblut|7 years ago|reply
[+] [-] houshuang|7 years ago|reply
[+] [-] gtm1260|7 years ago|reply
[+] [-] Iv|7 years ago|reply
And that article is tangentially related to a recurring interest of mine (shared editable 3d universe in which you can code collaboratively). If you happened to have an article comparing the merit of several server architectures/algorithms for collaborative text edition, I think many things can be related. I am now reading about CRDT (What the hell am I doing?) and that's something I may use to edit a scene graph collaboratively!
[+] [-] juretriglav|7 years ago|reply
In terms of editable 3d universes (whoah!), have you considered chucking together Three.js's JSON scene description (https://github.com/mrdoob/three.js/wiki/JSON-Object-Scene-fo...) with https://github.com/automerge/automerge, a CRDT for JSON? What could possibly go wrong?!
[+] [-] improv32|7 years ago|reply
[+] [-] im_down_w_otp|7 years ago|reply
There are a bunch of basic integration features that would also be fantastic, but if the product was literally just Google Docs, minus the WYSIWYG bits, and all plaintext w/ maybe some syntax highlighting thrown in, I'd be super happy.
[+] [-] jraph|7 years ago|reply
Not plain text only but probably can be configured for that.
No syntax highlighting.
[+] [-] houshuang|7 years ago|reply
[+] [-] twblalock|7 years ago|reply
That is the subset of features you want. Everyone else wants a slightly different subset of features. Add all of those subsets together and you end up with the total feature set of Google Docs.
[+] [-] choward|7 years ago|reply
And the feature creep begins. Eventually you'll be asking for a full blown IDE.
[+] [-] arendtio|7 years ago|reply
IANAL, but if you want to use the product yourself (even as a company) that should be just fine, but you are neither allowed to host the product for someone else (commercially) nor to created a product based on their frontend code.
They also have a demo which you can try here:
https://gold.ox.io/admin-api/tryme/user/gold/assignment
[1] https://www.open-xchange.com/portfolio/ox-documents/
[2] https://en.wikipedia.org/wiki/Open-Xchange#Licensing
[+] [-] lewisjoe|7 years ago|reply
The online word-processor provides contextual options without as much clutter as google docs.
[+] [-] amelius|7 years ago|reply
[+] [-] dalore|7 years ago|reply
[+] [-] jimpick|7 years ago|reply
* https://blog.datproject.org/2019/03/05/caracara-react-dat-au...
* https://conclave-team.github.io/conclave-site/
* https://github.com/noffle/hyperpad
* https://xi-editor.io/xi-editor/docs/crdt.html
* https://github.com/atom/teletype-crdt
* https://github.com/atom/xray
* https://github.com/MicrosoftDocs/live-share
[+] [-] alceta|7 years ago|reply
However I do note that we do not yet have a production ready backend for collaborative editing. I would suggest to use their service if possible, since we're developing an open source product it would not be an option for us.
There is a quite leaning curve when implementing new plugins due to the strict separation of model and data (commonmark in our case) and user-facing views.
[+] [-] juretriglav|7 years ago|reply
[+] [-] hyperion2010|7 years ago|reply
[+] [-] mochidusk|7 years ago|reply
[1] https://quilljs.com/ [2] https://github.com/Operational-Transformation/ot.js/ [3] https://firebase.google.com/ [4] https://github.com/phoenixframework/phoenix
[+] [-] jsilence|7 years ago|reply
[+] [-] ilaksh|7 years ago|reply
But for other uses, it looks absolutely incredible.
https://github.com/xwiki-labs/cryptpad
[+] [-] oefrha|7 years ago|reply
[+] [-] threespice|7 years ago|reply
From my understanding, an implementation of logoot[1] is built to create the skeleton of a collaborative environment. From there on, you have the building block to make other things work with it, such as collaborative canvas.
I built one using Socket.io on Angular and used Ace Editor own implementation of CRDT
> https://github.com/abdusamed91/realtime-codeeditor
I used the following references to build it
1) https://alligator.io/angular/socket-io/
2) https://medium.com/@ofir3322/create-an-online-ide-with-angul...
To get started with the theory CRDT low level algorithm, these two post go into great details
1) https://conclave-team.github.io/conclave-site/
2) http://digitalfreepen.com/2017/10/06/simple-real-time- collaborative-text-editor.html
[1]https://hal.inria.fr/inria-00432368/document
I'd love to take it to another level if anyone here has mutual interest in this field of work.
[+] [-] qpiox|7 years ago|reply
[+] [-] johanneswilm|7 years ago|reply
[+] [-] amplecooz|7 years ago|reply
They built their custom app with the open source ProseMirror (http://prosemirror.net/) toolkit.
[+] [-] nullc|7 years ago|reply
[+] [-] ginko|7 years ago|reply
https://www.gnu.org/software/emacs/manual/html_node/emacs/TC...
[+] [-] jsilence|7 years ago|reply
[+] [-] jcagalawan|7 years ago|reply
[+] [-] willvarfar|7 years ago|reply
This figures. This is one of the ways 'open source' companies can make money.
So, the author sets about making an open-source collaboration server that inter-operates with the existing open-source editor?
Kudos!
But now the company can retaliate by adding features or inconsistencies to the closed source server and its integration into the UI, making the open-source server forever out-of-date and playing catch-up.
Ouch.
[+] [-] mpweiher|7 years ago|reply
SubEthaEdit is a collaborative real-time editor designed for Mac OS X. The name comes from the Sub-Etha communication network in The Hitchhiker's Guide to the Galaxy series.
From 2003.
[+] [-] musha68k|7 years ago|reply
[+] [-] ilaksh|7 years ago|reply
You can try a lot of them out in the demo online https://etherpad.org/