[leif]

No, none of us know how to use those. :(

If you're interested in building one and you have experience with them, get in touch and we can work through it together. I think the biggest challenge would be modeling the semantics of write concern, but I'm not that familiar with proof assistants, maybe that isn't too hard.

[lumpypua]

It looks like you've read the Call Me Maybe series of posts over at aphyr.com. He tests a number of distributed systems (Mongo, Riak, Cassandra, etc) and their behavior under network partitions and almost all of them fuck up and lose data. A summary of results can be found at [1].

Amazon has used a TLA+ model for their distributed systems and found a bunch of bugs [2].

Seriously, everybody fucks this up. Please please learn a model checker and check your algorithm.

[1] In the "Summary of Jepsen Test Results" section: http://blog.foundationdb.com/call-me-maybe-foundationdb-vs-j...

[2] https://research.microsoft.com/en-us/um/people/lamport/tla/a...

[leif]

We are working on several ways to prove to ourselves and the community that what we have is correct. The most important is obviously testing. We have tests that demonstrate the problems we found, and Ark passes those tests. Publishing this explanation and soliciting feedback is another.

We have run Jepsen and have not been able to get it to show data loss in TokuMX. The problems it found in MongoDB were already fixed in other ways in earlier versions of TokuMX, but we're trying to get Jepsen to demonstrate the other problems we've found.

Model checking may be another way we can prove correctness, but since Ark is so similar to Raft, I think the Raft model in TLA+[1] is probably sufficient. Anyway, we'd also need a proof that the model is equivalent to the implementation, and I don't know of a way to do that, so I think functional tests are more important.

In any case, we'll look in to using a model checker, and any help would be greatly appreciated. If you're interested, feel free to email me.

[1]: https://ramcloud.stanford.edu/~ongaro/raft.tla

[sseveran]

If you have not used a model checker you don't have a proof. Please don't say that you have one. You are just hoping for the best.

See this to get yourself started: http://research.microsoft.com/en-us/um/people/lamport/tla/by...

[aurelius]

If you have used a model checker, you probably don't have a proof either. You have a model that might not be accurate, and testing all its inputs may be combinatorially prohibitive.