Git is no good for binaries. And SparkleShare is dead. The proper alternative is SyncThing https://syncthing.net/ I have it on my home and work laptops, a NAS and a VPS and it just works brilliantly.
Awww, dammit. Why did I never hear of this before? I just spent like 400 hours building something equivalent but far less finished.
Syncthing has almost 10,000 stars, so it's not like others hadn't discovered it yet. Is there a way for me to browse all the popular repos on Github so I can see what other amazing things I've been missing? There's [0], but that just shows me an utterly useless view of all the repos I've starred.
Your story is very familiar to me, including trying and failing to use Dropbox due to terrible symlink-handling!
I built something similar recently [0] (also a bit like Sparkleshare, which I just learned about from another comment in this thread...), but for me I needed to sync all of my source code between multiple machines. You can't do that with stock git, but I was able to s/git/gut/gi to make "gut" (which uses .gut folders to store its own state and will happy gut-add files inside .git subfolders) and now I have a "~/work" folder on my laptop, on my desktop, and on a dev box on AWS, with all of my source code synced in real-time.
I also wrote much the same thing in Node and also in Python before rewriting (again) to Go. You can peruse my Python code at [1] if you're curious. I used plumbum (awesome library) for executing local/remote commands (makes it a bit simpler than using subprocess and calling ssh with shell commands). Also, you can use inotifywait with the "--monitor" option which makes it keep running when there are changes; you have to then parse out what changed, and make sure you handle state correctly, but that way you don't miss changes made while inotifywait starts back up again.
Very cool! I wish I'd found your repo before writing my implementation. In fact, I was going to rewrite mine in Golang to reduce memory consumption on my back-up server. I will take yours for a spin.
Very robust and well-written (Haskell), and extremely fast. An extension (daemon, client, and webui) to git. Gitlab uses this to provide large file support (which is similar to Github's LFS)[2]. Author is a Debian maintainer (and obviously a Haskell dev).
Sync Thing[3]
Fast and professional golang synchronizer with its own protocol[4]. Usually compared to BTsync.
Linking to alternatives is not exactly constructive criticism for someone looking for feedback from a Show HN. It feels dismissive and out of spirit of what's trying to be accomplished here. Without any commentary or questions, it's almost like you're saying, "Why did you bother?"
Now, I very much doubt that was your intention, but please have some empathy for the person taking a chance when sharing something on Show HN. It's not easy and I'd hate for someone to give up on something just because someone linked to other things that already existed in the world. We wouldn't have Dropbox or Google under those conditions.
At the very least ask the creator how his project is different from these other projects or ask if he's seen these alternatives or even ask why he might have decided to build his own. A little bit of effort on your end shows you recognize his effort and isn't tone deaf to what's going on here.
Despite all the competitors everyone is linking to here, I bet you learned a ton while building this. What was most surprising to you? What was the trickiest part?
It was a ton of fun on a saturday afternoon when I wrote this code. I was trying to avoid rolling my own syncing solution but I guess my Google-fu failed me :)
My goal was to write a very thin wrapper around the workflow I would follow if I had to sync the changes myself manually. The tricky part was in figuring out how to inform multiple client machines when the backup server noticed a change in the file system. I wanted to avoid writing a server-side component that I had to install on the server and maintain.
When I found that I could use a combination of "ssh" and "inotifywait" (run inotifywait on the server using ssh from the client and listen for changes), I was pleasantly surprised that this even worked! I see my implementation in this aspect as the equivalent of AJAX long-polling that used to be applied for chat like communication in the browser in some implementations. i.e. When some modification happens on the server filesystem, the "inotifywait" command quits thereby unblocking the "ssh" command upon which I do a "git pull".
Because of the above, I was able to keep my implementation really simple - The whole functionality was achieved in under 300 lines of code.
Really great code - clean and extensible! I also really like inotify for this sort of code and actually use it myself to kick off my build scripts automatically when changes are detected. BTW, have you run into too many files/dirs issues with inotifywait?
Thank you! I tried hard to keep the code very simple sacrificing some 20%-scenario requirements (such as automatic conflict resolution).
I am using this to synchronize my Markdown based notes files across machines. There are under a hundred files now so I haven't hit any issues in that department yet.
However, there is a gradual memory leak which persists even when I kill and restart my process. I observed this only on a KVM based Linux guest - not sure if this is because of inotify based listening but I'm going to have to dig deeper to find out.
[+] [-] bantunes|10 years ago|reply
[+] [-] dantillberg|10 years ago|reply
Syncthing has almost 10,000 stars, so it's not like others hadn't discovered it yet. Is there a way for me to browse all the popular repos on Github so I can see what other amazing things I've been missing? There's [0], but that just shows me an utterly useless view of all the repos I've starred.
[0]: https://github.com/stars?direction=desc&sort=stars
[+] [-] prashanthellina|10 years ago|reply
[+] [-] dantillberg|10 years ago|reply
I built something similar recently [0] (also a bit like Sparkleshare, which I just learned about from another comment in this thread...), but for me I needed to sync all of my source code between multiple machines. You can't do that with stock git, but I was able to s/git/gut/gi to make "gut" (which uses .gut folders to store its own state and will happy gut-add files inside .git subfolders) and now I have a "~/work" folder on my laptop, on my desktop, and on a dev box on AWS, with all of my source code synced in real-time.
I also wrote much the same thing in Node and also in Python before rewriting (again) to Go. You can peruse my Python code at [1] if you're curious. I used plumbum (awesome library) for executing local/remote commands (makes it a bit simpler than using subprocess and calling ssh with shell commands). Also, you can use inotifywait with the "--monitor" option which makes it keep running when there are changes; you have to then parse out what changed, and make sure you handle state correctly, but that way you don't miss changes made while inotifywait starts back up again.
[0]: https://github.com/tillberg/gut
[1]: https://github.com/tillberg/gut/tree/0ce233d44f7f55448c15411...
[+] [-] rakoo|10 years ago|reply
You'd get unison (https://www.cis.upenn.edu/~bcpierce/unison/) :)
[+] [-] prashanthellina|10 years ago|reply
[+] [-] jamiesonbecker|10 years ago|reply
Git Annex[1]
Very robust and well-written (Haskell), and extremely fast. An extension (daemon, client, and webui) to git. Gitlab uses this to provide large file support (which is similar to Github's LFS)[2]. Author is a Debian maintainer (and obviously a Haskell dev).
Sync Thing[3]
Fast and professional golang synchronizer with its own protocol[4]. Usually compared to BTsync.
1. https://git-annex.branchable.com/videos/
2. https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-...
3. https://syncthing.net/
4. https://github.com/syncthing/specs/blob/master/BEPv1.md
[+] [-] kevin|10 years ago|reply
Now, I very much doubt that was your intention, but please have some empathy for the person taking a chance when sharing something on Show HN. It's not easy and I'd hate for someone to give up on something just because someone linked to other things that already existed in the world. We wouldn't have Dropbox or Google under those conditions.
At the very least ask the creator how his project is different from these other projects or ask if he's seen these alternatives or even ask why he might have decided to build his own. A little bit of effort on your end shows you recognize his effort and isn't tone deaf to what's going on here.
[+] [-] kevin|10 years ago|reply
[+] [-] prashanthellina|10 years ago|reply
My goal was to write a very thin wrapper around the workflow I would follow if I had to sync the changes myself manually. The tricky part was in figuring out how to inform multiple client machines when the backup server noticed a change in the file system. I wanted to avoid writing a server-side component that I had to install on the server and maintain.
When I found that I could use a combination of "ssh" and "inotifywait" (run inotifywait on the server using ssh from the client and listen for changes), I was pleasantly surprised that this even worked! I see my implementation in this aspect as the equivalent of AJAX long-polling that used to be applied for chat like communication in the browser in some implementations. i.e. When some modification happens on the server filesystem, the "inotifywait" command quits thereby unblocking the "ssh" command upon which I do a "git pull".
Because of the above, I was able to keep my implementation really simple - The whole functionality was achieved in under 300 lines of code.
[+] [-] jamiesonbecker|10 years ago|reply
[+] [-] prashanthellina|10 years ago|reply
I am using this to synchronize my Markdown based notes files across machines. There are under a hundred files now so I haven't hit any issues in that department yet.
However, there is a gradual memory leak which persists even when I kill and restart my process. I observed this only on a KVM based Linux guest - not sure if this is because of inotify based listening but I'm going to have to dig deeper to find out.
[+] [-] tckr|10 years ago|reply
[+] [-] prashanthellina|10 years ago|reply
[+] [-] orahlu|10 years ago|reply
[+] [-] LukeB_UK|10 years ago|reply
[+] [-] prashanthellina|10 years ago|reply
[+] [-] xcopy|10 years ago|reply
[deleted]