Thanks, this is really cool. I don't have anything to add; I just want to let the author know this is appreciated. I recently decided to learn Go, and articles like this are a great way to get a feel for how Go programmers think about solving problems with the tools available to them. The kind of knowledge you don't get from just skimming The Go Programming Language and such.
I've recently started learning Go and chose a project-based approach. I've been reading "Writing an Interpreter in Go" and "Powerful Command-Line Applications in Go" and it's been loads of fun.
This didn’t dive into it, but I appreciated the suggested search terms for information on decentralized peer discovery. I always wondered how that would work, especially the bootstrapping process. Ultimately I found this StackOverflow answer to be really helpful: https://stackoverflow.com/a/22240583
One of the things that made the bootstrapping click for me was realizing there are in fact servers that DHT clients talk to once to find an entrance to the network. They all have a list (such as https://github.com/qbittorrent/qBittorrent/blob/c80238d66ff3... )
the first two characters look an awful lot like the number "14", when they're actually "L4". I'm not sure if bencoding is case sensitive or if anohter font may better differentiate the two, but that may be something the author may want to poke at if they happen to read this comment :)
I had the same confusion! In my browser, with the default monospace font, the two characters are so similarly as to be indistinguishable.
Reading the next example, showing an encoded dictionary, was what made the penny drop. Bencode doesn't go out of its way to be visually accessible, so different font choices would help.
You can normally control the number of peers you share with, so I guess if 50 peers are connected but you only want to share with 10, then when one completes you'd send an unchoke message to another peer to tell it you'll share with them. Meanwhile the other 39 will stay choked. I assume that's what's going on here.
Hey Jesse, we went to school together but I'm sure that's a ton of people you know by now ;). If you can figure it out hit me up ill show you some Zebra industry stuff I've reversed.
Really nice share! I think I'll have a follow-along with this next weekend to improve my go skills and try to learn rust by attempting to port this to it.
Hi I'm the original author. Sorry to see that you got downvoted because it's a good question and a real limitation of my project. I might get around to building a follow-up some day with upload/sharing since that's where the actual interesting stuff happens (and not just shuffling bits around, with a shitty download algorithm tacked on like in my post). No real plans though. If I do that then I might have to rewrite some of the code and edit parts of the blog post, since I've grown as an engineer and writer since I made this two years ago.
As for your question of "Is that tolerated?" the question is a social one. As the sibling commenter pointed out, if you use a private tracker, you're part of a community that often has a social contract (implicit or explicit) that participation means you need to contribute back and have a good seeding ratio. In that case, only downloading -- like my client does -- is not tolerated and might get you banned. That's why I chose to demo with torrenting a Linux ISO, since those tend to be public, have a lot of seeders anyway, and don't really care how much you upload or download.
You may be thinking of private trackers, where "ratio" is the currency of participation in that community. Debian, Libre Office, and even Humble Bundle care more about the community of seeders sharing the bandwidth load than whether anyone seeds their ratio
Was that worth a downvote? At least, if my question was that stupid, could you point me to the part of the blog post where it shows how pieces are sent to other peers?
this is good info. i was thinking about making my own, before i discovered picotorrent. but the lack of ui libraries for go prevented me from moving forward. nowadays, with nuts and others, it might be worth a shot.
Even if the peer isn't behind a NAT (or on IPv6), you still need to pass the firewall, which won't let incoming packets through. This can be solved by simply connecting from both ends simultaneously, (first attempt fails but opens the fw, second attempt in the other direction succeeds)
You only need hole punching if both peers are behind a NAT (TCP hole punching is substantially more complex and fragile than UDP). You can reduce the need for hole punching if you complicate the signaling protocol (eg "please dial me at <addr>").
That said, bittorrent is quite resilient to NATs in practice, because since it's many-to-many, chances are even NATed clients will find many non-NATed clients.
[+] [-] adastra22|3 years ago|reply
[+] [-] quirino|3 years ago|reply
[+] [-] throwaway894345|3 years ago|reply
[+] [-] banana_giraffe|3 years ago|reply
[+] [-] unboxingelf|3 years ago|reply
[+] [-] DoctorOW|3 years ago|reply
[+] [-] mdaniel|3 years ago|reply
[+] [-] andirk|3 years ago|reply
[+] [-] chrsig|3 years ago|reply
One nit, with the font:
the first two characters look an awful lot like the number "14", when they're actually "L4". I'm not sure if bencoding is case sensitive or if anohter font may better differentiate the two, but that may be something the author may want to poke at if they happen to read this comment :)[+] [-] pianoben|3 years ago|reply
Reading the next example, showing an encoded dictionary, was what made the penny drop. Bencode doesn't go out of its way to be visually accessible, so different font choices would help.
[+] [-] fazfq|3 years ago|reply
[+] [-] riley_dog|3 years ago|reply
[+] [-] latchkey|3 years ago|reply
[+] [-] merpkz|3 years ago|reply
What is the purpose of that choke part when connecting to remote peers? Is that some kind of flood protection?
[+] [-] nprateem|3 years ago|reply
[+] [-] 3ln00b|3 years ago|reply
[+] [-] elcapitan|3 years ago|reply
[+] [-] LinkLink|3 years ago|reply
[+] [-] tommica|3 years ago|reply
[+] [-] palata|3 years ago|reply
Anyway it's a cool post! Would be nice to see a follow-up for the upload/sharing part!
[+] [-] eat_veggies|3 years ago|reply
As for your question of "Is that tolerated?" the question is a social one. As the sibling commenter pointed out, if you use a private tracker, you're part of a community that often has a social contract (implicit or explicit) that participation means you need to contribute back and have a good seeding ratio. In that case, only downloading -- like my client does -- is not tolerated and might get you banned. That's why I chose to demo with torrenting a Linux ISO, since those tend to be public, have a lot of seeders anyway, and don't really care how much you upload or download.
There's nothing in the protocol that says you have to do it, and indeed, there are even "selfish" algorithms that try to maximize download speed while uploading nothing in return: https://www.wired.com/2007/01/bittorrent-bullies-bittyrant-a...
[+] [-] mdaniel|3 years ago|reply
[+] [-] palata|3 years ago|reply
[+] [-] jbirer|3 years ago|reply
[+] [-] hknmtt|3 years ago|reply
[+] [-] amits1995|3 years ago|reply
[+] [-] klabb3|3 years ago|reply
You only need hole punching if both peers are behind a NAT (TCP hole punching is substantially more complex and fragile than UDP). You can reduce the need for hole punching if you complicate the signaling protocol (eg "please dial me at <addr>").
That said, bittorrent is quite resilient to NATs in practice, because since it's many-to-many, chances are even NATed clients will find many non-NATed clients.
[+] [-] UltraViolence|3 years ago|reply
[+] [-] odo1242|3 years ago|reply