It's worth imagining if curl was invented today...
How large would the team be behind it? How many stars on Github? How many open pull-requests?
How big would the patreon be? The Github sponsors? What Fortune100 would sponsor it?
The open issues count? The build steps? The installer packages? The docker image? The docker-compose? The rust rewrite?
Wiki pages? The reactjs website? The nextjs rewrite? The CSS framework that would be born from the rewrite?
The incomplete documentation, because of the Discord? How big would the Discord be? What about the subreddit born to cry about the lack of documentation because of the Discord?
What about the startup that would offer you curl as a service? Curl as a lambda? Curl one click install in CPanel?
How many milliseconds would curl take to just start up? How much RAM would it take?
How many developer resume would claim to be proficient at curl?
Somethings are amazing because they were made at a time where things were simply made... Not taking a stab at anyone or anything in particular, just reminiscing and imagining...
I'm a business consultant and surprised how Daniel missed a big opportunity here. He should start selling to big enterprises by making curl a subscription-based model. Then he could hire thousands of developers to support him so he never has to do the grunt work himself and completely focus on monetization strategy and a rewrite in nextjs. Then he could start an ad-based model in partnership with Facebook for free use to support the free tier. Daniel, if you're reading this contact me. We can make a good deal.
The simple way to do this is just to declare that a new curl is needed in Rust, and you'll get thousands of people lined up to work on your "breakthrough" project. Then get seed funding for this incredible task and you'll have a business!
So true! All of this makes me feel like I missed out on the golden era of open source. I mean I was there 25 years ago but my programming skills were not good enough to write something like ls command, let alone something like curl!
If I knew as much about programming 25 years ago as I know now, maybe I may have thought of writing something like curl or wget too and make a successful project out of it. But alas! Those days are gone. And developing a new project these days comes with so much baggage. Developing something like curl from scratch today would be a mammoth task. The HTTP/HTTPS protocol spec has become huge! And so many compatibility issues to consider and resolve even for creating an MVP that can get some minimal adoption.
What good problems are there today that curious developers like me can work on? Problems that do not have a huge historical baggage and compatibility issues to worry about? Problems that are still new and tractable enough for a single person to work on and get an MVP out?
What was the most depressing funny post I’ve seen. Ever. Great job. I genuinely feel sad and nostalgic now. I know there’s a lot of love for Discord but oh boy, where do I start…
I wasn't sure what "driven" meant despite hearing it all the time, so I looked in the dictionary:
"Someone who is driven is so determined to achieve something or be successful that all of their behavior is directed toward this aim"
That fits with this early part of the article:
"The concept behind curl development has always been to gradually and iteratively improve all aspects of it. Keep behavior, but enhance the code, add test cases, improve the documentation. Over and over, year after year. It never stops. As the timeline below helps showing.
Similarly, there was no sudden specific moment when suddenly curl became popular and the number of users skyrocketed. Instead, the number of users and the popularity of the tool and library has gradually and continuously grown. In 1998 there were few users. By 2010 there were hundreds of millions."
I love curl. It's become one of my favorite pieces of software partly due to the fact that I use it so often but mostly because when I do run into problems, the documentation is so good. Thanks for making useful stuff!
Open the network tab in your browser's dev-tools. Right click on any interesting request and "Copy as cURL." Now you're well on your way to using cURL to do something interesting.
I would also say both has it's place. Wget is the downloader, curl is for when I really need control over the request, headers, body and so on (so interactions with APIs, testing, ...)
Indeed. I really enjoyed the article, and I am grateful for the tool as well. My favorite section was "Staying relevant" which explains (IMO) how it is possible for this project to have not only kept such staying power but exploded in use.
Also the "2019" section made me lol:
> I started working for wolfSSL, doing curl full-time. It just took 21 years to make curl my job.
Finally: thank you to Daniel Stenberg, Rafael Sagula, and the many, many contributors for making this tool.
HTTP client libraries on the various languages still seems so lousy with excessive boilerplate and all sorts of complexity to handle cases that are beyond rare.
> with almost 150 CVEs since its inception. Just now for the 25nd aniversary half a dozen were added to the list.
These ones?
141. We are not aware of any exploit of this flaw. Severity: Low
140. We are not aware of any exploit of this flaw. Severity: Low
139. We are not aware of any exploit of this flaw. Severity: Low
138. We are not aware of any exploit of this flaw. Severity: Medium
137. We are not aware of any exploit of this flaw. Severity: Low
...
> The dependency list is an absolute security nightmare:
-- Curl claims that it supports so many protocols but I do not see many examples where they are in action, for instance, how to use ssh protocol to perform transfer between two machines? Where can I find such examples?
-- Are there any examples of a fully functional high-level data motion tool that is based on Curl?
-- Has anyone used Curl to perform data transfers at large scale such as terabytes of data? I would love to read more about it if there are any pointers describing such transfers.
> for instance, how to use ssh protocol to perform transfer between two machines? Where can I find such examples?
That's out of scope for cURL, use `scp` for that.
> Are there any examples of a fully functional high-level data motion tool that is based on Curl?
What's data motion?
> Has anyone used Curl to perform data transfers at large scale such as terabytes of data? I would love to read more about it if there are any pointers describing such transfers.
There's a few posts on Stack Overflow about that, but they go up to gigabytes. cURL is not the best solution for transfering terabytes of data. http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html seems to be a good overview of the challenges (and tools) to use to securely and reliably transfer terabytes of data, referencing rsync, bbcp, bittorrent, etc.
> Curl claims that it supports so many protocols but I do not see many examples where they are in action, for instance, how to use ssh protocol to perform transfer between two machines? Where can I find such examples?
SSH file transfers use either SCP:// or SFTP:// urls, not SSH:// urls.
> Curl claims that it supports so many protocols but I do not see many examples where they are in action, for instance, how to use ssh protocol to perform transfer between two machines? Where can I find such examples?
You could try it yourself! The man page is pretty good.
[+] [-] keyle|3 years ago|reply
How large would the team be behind it? How many stars on Github? How many open pull-requests?
How big would the patreon be? The Github sponsors? What Fortune100 would sponsor it?
The open issues count? The build steps? The installer packages? The docker image? The docker-compose? The rust rewrite?
Wiki pages? The reactjs website? The nextjs rewrite? The CSS framework that would be born from the rewrite?
The incomplete documentation, because of the Discord? How big would the Discord be? What about the subreddit born to cry about the lack of documentation because of the Discord?
What about the startup that would offer you curl as a service? Curl as a lambda? Curl one click install in CPanel?
How many milliseconds would curl take to just start up? How much RAM would it take?
How many developer resume would claim to be proficient at curl?
Somethings are amazing because they were made at a time where things were simply made... Not taking a stab at anyone or anything in particular, just reminiscing and imagining...
[+] [-] siva7|3 years ago|reply
[+] [-] latency-guy2|3 years ago|reply
[+] [-] coliveira|3 years ago|reply
[+] [-] matheusmoreira|3 years ago|reply
I don't get it. Those are ethical ways to reward people for their valuable work. Are you implying people should be ashamed of this?
[+] [-] ketanmaheshwari|3 years ago|reply
[+] [-] distcs|3 years ago|reply
If I knew as much about programming 25 years ago as I know now, maybe I may have thought of writing something like curl or wget too and make a successful project out of it. But alas! Those days are gone. And developing a new project these days comes with so much baggage. Developing something like curl from scratch today would be a mammoth task. The HTTP/HTTPS protocol spec has become huge! And so many compatibility issues to consider and resolve even for creating an MVP that can get some minimal adoption.
What good problems are there today that curious developers like me can work on? Problems that do not have a huge historical baggage and compatibility issues to worry about? Problems that are still new and tractable enough for a single person to work on and get an MVP out?
[+] [-] steve1977|3 years ago|reply
[+] [-] b212|3 years ago|reply
[+] [-] bas|3 years ago|reply
[+] [-] subtra3t|3 years ago|reply
[+] [-] bbbbzzz1|3 years ago|reply
[+] [-] benatkin|3 years ago|reply
"Someone who is driven is so determined to achieve something or be successful that all of their behavior is directed toward this aim"
That fits with this early part of the article:
"The concept behind curl development has always been to gradually and iteratively improve all aspects of it. Keep behavior, but enhance the code, add test cases, improve the documentation. Over and over, year after year. It never stops. As the timeline below helps showing.
Similarly, there was no sudden specific moment when suddenly curl became popular and the number of users skyrocketed. Instead, the number of users and the popularity of the tool and library has gradually and continuously grown. In 1998 there were few users. By 2010 there were hundreds of millions."
That very much typifies "driven".
However, there is this at the end of his about page at https://daniel.haxx.se/about.html
LOL!!![+] [-] filleokus|3 years ago|reply
Good to see someone who had such a positive influence get recognised outside of our bubble
[+] [-] ilovecurl|3 years ago|reply
[+] [-] fsckboy|3 years ago|reply
[+] [-] flerchin|3 years ago|reply
[+] [-] jraph|3 years ago|reply
Wget to download stuff in an interactive shell (shorter, and I know its -c switch to continue interrupted downloads), curl for anything else.
[+] [-] atoav|3 years ago|reply
[+] [-] ricardo81|3 years ago|reply
[+] [-] ufmace|3 years ago|reply
[+] [-] quijoteuniv|3 years ago|reply
[+] [-] tomcam|3 years ago|reply
[+] [-] avg_dev|3 years ago|reply
Also the "2019" section made me lol:
> I started working for wolfSSL, doing curl full-time. It just took 21 years to make curl my job.
Finally: thank you to Daniel Stenberg, Rafael Sagula, and the many, many contributors for making this tool.
[+] [-] monksy|3 years ago|reply
[+] [-] davideg|3 years ago|reply
In November, in the middle of Covid, I finally got a US visa.[1]
[1] https://daniel.haxx.se/blog/2020/11/09/a-us-visa-in-937-days...
Edit: formating
[+] [-] asicsp|3 years ago|reply
[+] [-] pbreit|3 years ago|reply
[+] [-] alliao|3 years ago|reply
[+] [-] racl101|3 years ago|reply
[+] [-] fefe23|3 years ago|reply
[deleted]
[+] [-] mrkeen|3 years ago|reply
These ones?
...> The dependency list is an absolute security nightmare:
Is it the thirteen items on this list: https://curl.se/docs/libs.html ? "You can build curl without them"
> Why is anyone trusting software that needs this many security updates for such a comparably trivial task?
At 148299 lines of C, perhaps it has more uses than you give it credit for?
[+] [-] ketanmaheshwari|3 years ago|reply
-- Curl claims that it supports so many protocols but I do not see many examples where they are in action, for instance, how to use ssh protocol to perform transfer between two machines? Where can I find such examples?
-- Are there any examples of a fully functional high-level data motion tool that is based on Curl?
-- Has anyone used Curl to perform data transfers at large scale such as terabytes of data? I would love to read more about it if there are any pointers describing such transfers.
[+] [-] Cthulhu_|3 years ago|reply
That's out of scope for cURL, use `scp` for that.
> Are there any examples of a fully functional high-level data motion tool that is based on Curl?
What's data motion?
> Has anyone used Curl to perform data transfers at large scale such as terabytes of data? I would love to read more about it if there are any pointers describing such transfers.
There's a few posts on Stack Overflow about that, but they go up to gigabytes. cURL is not the best solution for transfering terabytes of data. http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html seems to be a good overview of the challenges (and tools) to use to securely and reliably transfer terabytes of data, referencing rsync, bbcp, bittorrent, etc.
[+] [-] charonn0|3 years ago|reply
SSH file transfers use either SCP:// or SFTP:// urls, not SSH:// urls.
>> curl sftp://public.example.com/file.txt
[+] [-] rjh29|3 years ago|reply
You could try it yourself! The man page is pretty good.