I don't know if this is related or not but back when Flickr was popular in like 2005 or 2006 my friends and I were uploading pictures of our events there (they're all still there AFAIK. We'd upload them to either our own accounts or for certain events to a shared account. One shared account is here: https://www.flickr.com/photos/sourpower/albums
I wanted copies of those pictures and the easiest way to get them was the write a tool to download them rather than have to coordinate with 3 to 15 friends and ask them to copy the images to a CD or USB stick or some other nonsense. Dropbox wasn't a thing and not all my friends were tech heads that would want to setup FTP servers.
Flickr had also come out with an API. APIs for online services seemed kind of new at that point and Flickr was one of the first AFAIK.
So I wrote the app https://blog.greggman.com/blog/flickrdown/
and a few months later it was accused by other users of flickr of being solely for the purpose of downloading copywritten images. Not once did I ever use it for such a purpose nor, AFAIK did any of my friends. None of us had any interest in other people's images on flickr, only shared images of mutually attended parties, bbq, picnics, events.
Those users reported the app to Flickr and the app was banned.
It was banned by the app's id. That meant you could register your own app and then hack in your app's id and still use it. IIRC I continued to use it to download pictures from our events but it always pissed me off they banned it. It also pissed me off because it wasn't accessing anything you couldn't just scrape for. The API made it easy to get a list of URLs, search for albums or people etc but you could easily write a script that just scraped the HTML to find all the same data. Didn't matter, flickr didn't budge.
It further pissed me off that over zealous flickr members accused me of lying about its purpose. Like many topics today, there is often absolutely nothing you can say that will convince someone else your intensions are not bad.
This story clearly illustrates the purpose behind "web APIs". To limit access.
As a user (not a web developer), I personally never saw the practical point of web APIs; I have always just "scraped the HTML". Many times the solutions I write outlive the corresponding "API"; IME, often the non-API method of data retrieval is more robust and reliable than using the so-called API.
YouTube used to have a freely accessible search API. Not anymore. However "scraping" the YT search result pages continues to work fine.
> Like many topics today, there is often absolutely nothing you can say that will convince someone else your intensions are not bad.
Sometimes it's better to not even try. Acting is always the more powerful move. Just do your thing. You wrote an awesome program that downloads stuff, they got offended and banned it. You've already accomplished your goal so you let it go... But if you cared enough you could just write a scraper for the website itself. What are they gonna do about it?
In an ideal world, downloading copywritten data would be so easy and ubiquitous that the intellectual property laws would be unenforceable. Sure, they would get mad but who cares? There's very little they can do about it.
I maintain a similar project for SoundCloud called SoundScrape: https://github.com/Miserlou/soundscrape which I started for a similar reason, to save my own 'likes' and tracks that I've made and my friends have made.
SoundCloud made this very easy, as they had an API which exposed the endpoint MP3/WAV location in a field. The tool used an API key provided by SoundCloud to fetch the response.
Overnight and without warning, they removed that field from responses, changed the terms of service, banned my application for terms of service violations, and deleted all of my personal music and likes because I had used my own account to create the API key.
I was very angry at the time since all my my music got deleted, but these days I'm just sad. Things like this have little by little destroyed all of my enthusiasm for technology.
While I can understand your frustration, I also understand the frustration of photographers and other creators trying to make a living who have their work stolen. Their anger was wrongly directed at you, but it is a real problem.
Back when Udacity just launched with their first course (Thrun's self driving car intro) I wanted to watch the videos on a big screen. I had a somewhat smart TV that could play files off a USB stick but did not have Internet access. Udacity hosted all their videos on YouTube at the time and there was no convenient way to download them. So I spent an afternoon hacking together a Chrome extension that would modify the Udacity website DOM and give me download links for the lectures (it had to be an extension to get around same origin restrictions). Udacity had some weird naming convention for their videos so I had to make some calls to their API as well as YouTube and correlate videos with titles. The YouTube part was "influenced" by youtube-dl. This was much faster than reverse engineering, even though I was writing new JavaScript and not using the original Python. Anyway, trivial stuff.
After watching the videos from my couch for a few days I decided to post a link to my extension on the Udacity message board... and it absolutely blew up! My dinky little extension had thousands of users all over the world seemingly overnight.
But the absolute highlight was getting an email from a student from Iran. Iran just blocked YouTube because of https://en.wikipedia.org/wiki/Innocence_of_Muslims and there was a whole group of students who could no longer participate in the course. Apparently they had some friends at a US university use my extension to download the videos and reupload to a VPS they ran. I was blown away - my quest to sit on a couch ended up accidentally helping fight censorship.
I maintained the extension until Udacity added a native video download feature and then took it down. But it was an interesting experience and definitely shaped my perception of fair use laws. They are important. People have way more legitimate uses for information than lawyers can imagine.
> my quest to sit on a couch ended up accidentally helping fight censorship
That's brilliant! We can never predict the impact our tools will have on people's lives.
Your idea of partially porting youtube-dl to the browser gives me an idea... would it be feasible to port it fully? I think the biggest hurdle would be ffmpeg, but a few days ago I saw "A pure WebAssembly / JavaScript port of FFmpeg": https://news.ycombinator.com/item?id=24987861
Thanks for this. It is weird how long it’s been. I totally forgot that Wireshark was once called Ethereal.
I hope that the DMCA takedown issue can be resolved reasonably, but it’s starting to seem more and more like a move off of Github is overdue. Especially in a world where anyone can stand up a Gitea or Gitlab CE instance.
> I hope that the DMCA takedown issue can be resolved reasonably,
I don't think it can be solved on Github.
"GitHub’s CEO suggested that YouTube-DL won’t be reinstated in its original form. But, the software may be able to return without the rolling cipher circumvention code and the examples of how to download copyrighted material."
This pretty much makes youtube-dl useless, since the "rolling cipher" is just downloading the same bit of js, inspecting it, and executing it, almost the way a web browser does (AIUI, the difference is that yt-dl inspects the js and picks out the function to run from it instead of just running it all verbatim). This counts as circumvention according to the DMCA, which leaves yt-dl little legal standing in the US.
Also note that the "examples of how to download copyrighted material" in the yt-dl tests were just code for getting the first few bytes of a number of RIAA-sequestered music videos. Small excerpts are usually allowed under Fair Use. The RIAA didn't really look into that detail.
On the plus side, this fork is active and not DMCA'ed, for now. I just turned to it because I needed a fix for Bandcamp that upstream yt-dl doesn't have:
gitlab isn't evil. One of my short life lessons is that, in general, when I do business with evil, sketchy, or nasty organizations or people, I generally come out behind.
If the good guys have an inferior product and charge double, I'll sometimes pick the bad guy. And more often than not, I get burned, costing me tenfold what it would have cost to just go with the high-integrity choice in the first place.
I'm not leaving github over this, but I'm mostly starting new project on gitlab instead.
My first tech book after Mike Meyers "All in one A+ 4th Ed" was a book written on Wireshark by Laura. I never got too deep as it was taken from the desk I pulled CQ duty at.
> I hope that the DMCA takedown issue can be resolved reasonably, but it’s starting to seem more and more like a move off of Github is overdue.
It's a risky move, dabbling with stuff that is targeted under DMCA. Anything hosted in the US is liable for takedowns - including domain names that are under the control of US-based companies. You'll need to deal with acquiring hosting and DDoS protection yourself, plus keeping track of security updates. And to be honest Europe isn't exactly a legal safe haven either, we also have nasty laws (e.g. in Germany the infamous "Störerhaftung") exposing you to liability.
Been a longtime fan of YouTube-dl, so much better and faster than any alternative. I remember being in high school in a group project with some theater kids. They needed a ton of free stock footage from Vimeo and they were going through links one-by-one to download each video individually. I just compiled a list of the links, ran youtube-dl, and had it done in 5 minutes when they'd already wasted three hours on it.
What a wonderful little retrospective. It's a nice walk down memory lane, and I really appreciate how clear and sensible his thinking was around handing off the project when the time was right.
Wow, I did not know youtube-dl had existed for so long.
I remember coding my own YouTube downloader because of similar reasons. My internet connection was way too slow to stream videos, even at the lowest quality, so I'd make a list of videos, download them in the background the entire day and then watch it offline at the end of the day. When I finally discovered youtube-dl, I was relieved that I no longer had to keep maintaining my script... and it supported almost every other video website.
Also, I just realized that it can still be downloaded from the official website and updated using --update argument.
What a wonderful writeup that touches on both the original author's personal history, little technical tidbits, and so many ethical and societal aspects of computing!
I've had a personal playlist of videos I maintained since 2008, a collection really, with multiple fads I once used to share with the various social circles I had during the last decade at college and university. But now more than half of the videos are gone due to the various ban episodes YouTube had (FU cancel culture, FU Carlos Maza). Fortunately with YTdl I can save at least half of what has been my good fairly big portion of my adult life.
greggman3|5 years ago
I wanted copies of those pictures and the easiest way to get them was the write a tool to download them rather than have to coordinate with 3 to 15 friends and ask them to copy the images to a CD or USB stick or some other nonsense. Dropbox wasn't a thing and not all my friends were tech heads that would want to setup FTP servers.
Flickr had also come out with an API. APIs for online services seemed kind of new at that point and Flickr was one of the first AFAIK.
So I wrote the app https://blog.greggman.com/blog/flickrdown/ and a few months later it was accused by other users of flickr of being solely for the purpose of downloading copywritten images. Not once did I ever use it for such a purpose nor, AFAIK did any of my friends. None of us had any interest in other people's images on flickr, only shared images of mutually attended parties, bbq, picnics, events.
Those users reported the app to Flickr and the app was banned.
It was banned by the app's id. That meant you could register your own app and then hack in your app's id and still use it. IIRC I continued to use it to download pictures from our events but it always pissed me off they banned it. It also pissed me off because it wasn't accessing anything you couldn't just scrape for. The API made it easy to get a list of URLs, search for albums or people etc but you could easily write a script that just scraped the HTML to find all the same data. Didn't matter, flickr didn't budge.
It further pissed me off that over zealous flickr members accused me of lying about its purpose. Like many topics today, there is often absolutely nothing you can say that will convince someone else your intensions are not bad.
1vuio0pswjnm7|5 years ago
As a user (not a web developer), I personally never saw the practical point of web APIs; I have always just "scraped the HTML". Many times the solutions I write outlive the corresponding "API"; IME, often the non-API method of data retrieval is more robust and reliable than using the so-called API.
YouTube used to have a freely accessible search API. Not anymore. However "scraping" the YT search result pages continues to work fine.
matheusmoreira|5 years ago
Sometimes it's better to not even try. Acting is always the more powerful move. Just do your thing. You wrote an awesome program that downloads stuff, they got offended and banned it. You've already accomplished your goal so you let it go... But if you cared enough you could just write a scraper for the website itself. What are they gonna do about it?
In an ideal world, downloading copywritten data would be so easy and ubiquitous that the intellectual property laws would be unenforceable. Sure, they would get mad but who cares? There's very little they can do about it.
Mizza|5 years ago
I maintain a similar project for SoundCloud called SoundScrape: https://github.com/Miserlou/soundscrape which I started for a similar reason, to save my own 'likes' and tracks that I've made and my friends have made.
SoundCloud made this very easy, as they had an API which exposed the endpoint MP3/WAV location in a field. The tool used an API key provided by SoundCloud to fetch the response.
Overnight and without warning, they removed that field from responses, changed the terms of service, banned my application for terms of service violations, and deleted all of my personal music and likes because I had used my own account to create the API key.
I was very angry at the time since all my my music got deleted, but these days I'm just sad. Things like this have little by little destroyed all of my enthusiasm for technology.
I want to be a carpenter now.
michaelmior|5 years ago
nzmsv|5 years ago
After watching the videos from my couch for a few days I decided to post a link to my extension on the Udacity message board... and it absolutely blew up! My dinky little extension had thousands of users all over the world seemingly overnight.
But the absolute highlight was getting an email from a student from Iran. Iran just blocked YouTube because of https://en.wikipedia.org/wiki/Innocence_of_Muslims and there was a whole group of students who could no longer participate in the course. Apparently they had some friends at a US university use my extension to download the videos and reupload to a VPS they ran. I was blown away - my quest to sit on a couch ended up accidentally helping fight censorship.
I maintained the extension until Udacity added a native video download feature and then took it down. But it was an interesting experience and definitely shaped my perception of fair use laws. They are important. People have way more legitimate uses for information than lawyers can imagine.
andai|5 years ago
That's brilliant! We can never predict the impact our tools will have on people's lives.
Your idea of partially porting youtube-dl to the browser gives me an idea... would it be feasible to port it fully? I think the biggest hurdle would be ffmpeg, but a few days ago I saw "A pure WebAssembly / JavaScript port of FFmpeg": https://news.ycombinator.com/item?id=24987861
jchw|5 years ago
I hope that the DMCA takedown issue can be resolved reasonably, but it’s starting to seem more and more like a move off of Github is overdue. Especially in a world where anyone can stand up a Gitea or Gitlab CE instance.
jordigh|5 years ago
I don't think it can be solved on Github.
"GitHub’s CEO suggested that YouTube-DL won’t be reinstated in its original form. But, the software may be able to return without the rolling cipher circumvention code and the examples of how to download copyrighted material."
https://torrentfreak.com/riaas-youtube-dl-takedown-ticks-of-...
This pretty much makes youtube-dl useless, since the "rolling cipher" is just downloading the same bit of js, inspecting it, and executing it, almost the way a web browser does (AIUI, the difference is that yt-dl inspects the js and picks out the function to run from it instead of just running it all verbatim). This counts as circumvention according to the DMCA, which leaves yt-dl little legal standing in the US.
Also note that the "examples of how to download copyrighted material" in the yt-dl tests were just code for getting the first few bytes of a number of RIAA-sequestered music videos. Small excerpts are usually allowed under Fair Use. The RIAA didn't really look into that detail.
On the plus side, this fork is active and not DMCA'ed, for now. I just turned to it because I needed a fix for Bandcamp that upstream yt-dl doesn't have:
https://github.com/blackjack4494/yt-dlc
woofie11|5 years ago
If the good guys have an inferior product and charge double, I'll sometimes pick the bad guy. And more often than not, I get burned, costing me tenfold what it would have cost to just go with the high-integrity choice in the first place.
I'm not leaving github over this, but I'm mostly starting new project on gitlab instead.
llsf|5 years ago
nosmokewhereiam|5 years ago
mschuster91|5 years ago
It's a risky move, dabbling with stuff that is targeted under DMCA. Anything hosted in the US is liable for takedowns - including domain names that are under the control of US-based companies. You'll need to deal with acquiring hosting and DDoS protection yourself, plus keeping track of security updates. And to be honest Europe isn't exactly a legal safe haven either, we also have nasty laws (e.g. in Germany the infamous "Störerhaftung") exposing you to liability.
humford|5 years ago
wpietri|5 years ago
AntiImperialist|5 years ago
I remember coding my own YouTube downloader because of similar reasons. My internet connection was way too slow to stream videos, even at the lowest quality, so I'd make a list of videos, download them in the background the entire day and then watch it offline at the end of the day. When I finally discovered youtube-dl, I was relieved that I no longer had to keep maintaining my script... and it supported almost every other video website.
Also, I just realized that it can still be downloaded from the official website and updated using --update argument.
gman83|5 years ago
gspr|5 years ago
vagrantJin|5 years ago
saghul|5 years ago
TeeMassive|5 years ago
Sektor|5 years ago
rg3|5 years ago
kzrdude|5 years ago
unknown|5 years ago
[deleted]