Personally, I find it fascinating that this band of merry pirates is so organized, so rules-based. Out of a purely anarchic environment, they've managed to accept this kind of self-discipline. If only web-standards committees had that sort of power...
It's because there's a huge amount of competition in the "scene" to get first and best copies of a show. In order to declare a "winner" in the competitions, there have to be rules.
The (movie/audio/warez/demo) "scene" is not anarchic at all but rather a very well organised distributed processing machine. You probably know these guys organise into "groups" but there is lots of other stuff you likely do not know. I will describe below aspects of the warez scene, as it was some 10 years ago. I believe most observations continue to hold today and are probably mirrored in the movie-release scene. So:
"The scene" is made up of groups and each group is made up of specialists: suppliers provide material for release, crackers (or rippers, depending on context) remove copy protection and sometimes optional "filler" material, coders write useful helper software for the group and couriers distribute releases around the internet. Not all groups fill every role and some may have other roles I did not describe here (hackers, carders, hardware suppliers to name a few). There are sometimes partnerships between groups with different specialities; usually this occurs when there is an overlap in membership. For example: a release group may have close links with a particular courier group and those guys handle all their releases.
The distribution process is actually quite interesting. Each successful release group aims to have a small list of affiliations with well known and prestigious underground boards/ftp-sites. The quality and prestige of a board depends on the speed of its connection, its capacity, its group affiliations, the standing of its admins in the community and the speed with which new releases are uploaded to it. Each well regarded site is allowed only one particular kind of affiliation: a games group affil, an apps group affil, a courier affil, an ISO affil -- you get the idea. The important thing is that there is only one of each. Affiliations carry prestige for a site and being affiliated with the best sites raises a group's prestige, so there is often heated competition between sites and groups for affils. This is the case at least in the so called "zero second" scene where releases are "traded" (i.e. re-uploaded) by couriers within seconds of being first uploaded. Once a release has been traded among all zero-second sites, they slowly filter down to lesser and lesser sites until eventually making their way to the Internet at large.
So why would you go this all this trouble? The reasons are myriad but usually it comes down to prestige: for example, among couriers there is a longstanding weekly and monthly competition between groups that prove their chops by trying to dominate each other. Similar competitions exist among release groups. There is huge pressure to be the first to release a highly anticipated game or well known application. Additionally, there are strict rules about how to release something -- what to keep, what can (and should) be omitted, how to package everything up, the inclusion of .nfo and .diz files and so on. Failure to comply with these strict standards renders a release invalid and causes it to be "nuked" (deleted) among the top sites. This process amounts to a public shaming of the group responsible and makes it possible for competing groups to snatch the credit by doing a "proper" release.
I could go on and on but I think that's enough for now. Hopefully I've convinced you that pirates aren't an anarchic bunch ;)
Competition forces rules to be born. Not just competition for prestige, a competition for access.
Topsites are fuelled by racers. These are the couriers, who transfer data between multiple topsites they have access to. All topsites have a multiplier. For every 1MB you upload, you can download ~3MB. If you don't each a quota or are not in the top ~15 racers by the end of the week your access will be revoked.
With access comes prestige, ability to race to other topsites, contacts, and leech access to the topsite. These things are many terabytes big and can have archives in excess of 60TB. A huge repository of constantly refreshing and up to date material.
Without rules this would become a free-for-all, with poor quality. That's no good for anyone.
I wonder if the producers of the content they distribute will ever get that such a high level of quality.
DVDs are often full of interlaced content (even for short movies), broken subtitles, letterboxed pictures, jerking menus; sometimes they even resize the video to 4:3. The rare downloadable videos are often over compressed or barely compressed (thank you for the 2GB file to this 20 minute show). Similar things happens with bought music: I have paid for many albums distributed as badly-encoded MP3s in zip files with __MACOSX directories and ._DS* files.
That's because there is no competition for those releases, so there is little care for quality.
If content producers released their raw sources and content monetizers had to compete to produce the best possible downloadable and distributable versions, you'd see much better commercial releases. ;-)
Anime groups can afford to be much more aggressive about technology because they have special requirements that older tech doesn't allow (styled softsubs, multiple audio/sub tracks, ordered chapters), and because their audience is overwhelming using high-performance PCs because nothing else commonly supports those technologies. In fact, modern anime releases are basically targeted towards one platform (Windows PC with MPC-HC and CCCP[1] without DXVA) with coincidental support for Mplayer on Linux because it uses the same basic open source codecs and splitters. Hi10p is even more aggressive - when they started releasing Hi10p releases, the only decoder that supported it was ffdshow alphas, and even now software support for it is very sparse. Hardware support is non-existent, and will continue to be so unless a mainstream content provider like Apple decides to start releasing Hi10p video content.
On the other hand, people who watch non-anime scene releases are much more likely to use mobile devices, set-top boxes and other A/V equipment that relies on hardware decoding support. One of the major reasons that scene releases are standardized is so that they'll play on a wide variety of hardware, and this results in a much more conservative approach than anime groups. The main reason they're moving forward with this change is because H.264 has acceptable support in hardware (primarily in the MP4 container, MKV support is awful) and XviD is really showing its age compared to modern H.264 encoders like x264.
For the longest time, xvid was the standard for these guys. They still have standards requiring that all releases be packaged into multipart RAR files, "in 15, 20, or multiples of 50 MB". In other words, they are kind of inflexible and hidebound.
Anime fansubbing groups, in contrast, have no common standards except for -- to paraphrase the IETF motto -- rough consensus and regular releases. If some group wants to encode with Hi10P to get the same video quality at 2/3 the file size, they don't have to ask permission from anyone. Competition between groups gradually pushes forward the rough consensus, which is how they switched from xvid to x264, from container formats like AVI to the more flexible MKV, from SD to HD, and so on. The freedom to not live up to a community standard of quality gave them the ability to exceed it.
Yeah. It was the same thing when bittorrent became big. It was the anime groups that were the first big adopters. Mainstream releases took a lot longer.
Thank you. I was a bit surprised by the mp4 container (instead of matroska), as well as DTS not being allowed. Now it all makes sense, and it's not really a big deal at all.
SD content is mostly being used on smartphones and tablets. For iOS users, this would require a lot of extra work to convert the files to the limited container choices.
Internal ffmpeg aac encoder is still experimental and tends to create broken AAC streams at times. The other options:
A) libfaac / libvo_aacenc are pretty bad in terms of quality and have problems with multi-channel audio
B) libaacplus only encodes up to 64kbps in less supported AAC+
So Nero and Apple are pretty much only quality free audio encoders for AAC available right now.
Finally, pirates are using ISO standards instead of gross hacks like DivX. RAR is still lame and using the tag "HDTV" to refer to SD downsamples of HD sources is still misleading, though.
HDTV indicates source, not release quality. It's the same with bluray. The naming standards (and everything else for that matter) are highly regulated which is an impressive example of order by consensus since no one can give out orders.
RARs are an artefact of FTP transfers during which files would get corrupted, and not-so-fast connections of the past which made re-downloading the whole release somewhat inconvenient and wasteful.
It seems they had previously mandated Xvid, which is very much a conforming implementation of MPEG-4 part 2 (for that matter, DivX has been since 2001.)
I noticed this today on Usenet. I've switched to downloading all my TV from torrent sites to Usenet because it's much more reliable. Everything old is new again. It's nice to see that the animated shows I enjoy like Family Guy are available in HD resolutions at roughly the same download size as their previously available SD counterparts. I assume animation is really efficiently encoded with h264. Anyway, smaller files and higher quality = happy me.
But requiring mp4 as container is a bad idea in my opinion. To playback an mp4 file you need to have it wholly (please correct me if I am wrong) so you cannot start watching before you downloaded it all. MKV would be so much better.
Please do not blindly reply with "But my XYZ device does not support MKV". Scene rules/standards have a lot of impact and might help making vendors support Matroska.
> To playback an mp4 file you need to have it wholly (please correct me if I am wrong)
You are. By default, many tools put the meta data of mp4 files at the end of the file which means you have to have the whole file before playing. However, such mp4 files can be "fixed" using tools like this:
> To playback an mp4 file you need to have it wholly
720p or better videos on youtube are all MP4 now. You just have to move some atoms around within the video file to prepare them for "streaming" playback. There's open source software to do this.
Then again, the ability to play an incomplete file is not really useful if you're fetching a movie via torrent, which is what I would assume most of these guys are using.
If the MP4 is "fast start" the index is at the beginning so you can "stream" it.
I suspect scene standardization on MKV would probably encourage Apple and Sony to deliberately not support MKV (instead of not supporting it out of laziness, as they currently are).
You don't need the whole file. That would just be silly and it's used for streaming all the time. Also even though I would prefer to see an open standard succeed, the mp4 container does have a lot more support while mkv doesn't have any advantages for this type of use.
What makes it beneficial for ripping/distribution groups to coordinate like this? I've previously seen them agree on a common standard for the number and size of rar files a release is to be broken up into, and other things i can't remember the specifics of. Why do they do it?
From reading many of the curiously well-informed comments, I suspect that the reason for standardization has to do with speed and validation for credit. Since it sounds like "scene" people arbitrage new releases across scene FTP drop sites for upload credit, a standard method of "settling accounts" via scripts/bots would be useful. Arbitrage would be less efficient if files had to be transcoded between FTP sites.
I wondered why I was suddenly seeing x264 instead of AVI for TV files. I thought it had to do with the lockers going down.
Well, in a way this will make things even worse for the CopyNazis. No longer will people have to convert from AVI to watch on their tablets and phones.
I suppose you mean mp4 instead of avi, as avi is only a container. So the discussion is two-fold, the container format (mp4, mkv, avi) and the encoding of audio, subtitles and video, of which the latter, h264 is pretty much the standard, and xviD/Divx are and have been for almost a decade, deprecated.
Although I can just imagine all the pirates whining about "h.264 (or mkv as they call it) doesn't play on my dvd player, bring avi (as they call xvid) back".
I can't believe pirates are still burning crappy SD rips to DVD.
Someone I know stopped doing that when he wanted to watch a movie for the second time and realized it was quicker to leech a 1080p bluray rip than search through his mountains of VCD and SVCD discs for the one he downloaded just a couple of years back.
This pirate friend of mine welcomes the change, for the maybe 1 SD show he watches.
Although he personally wishes people paid more attention to sound codecs, nothing worse than downloading something broadcast in 5.1 and finding out the file only has a stereo downmix.
1) I watch it in the corner of my screen, so I'm fine with 320.
2) Here in Australia, we pay for usage. I'm not going to download something in a high quality if I'm near my limit
[+] [-] mdda|14 years ago|reply
[+] [-] sp332|14 years ago|reply
[+] [-] gradstudent|14 years ago|reply
"The scene" is made up of groups and each group is made up of specialists: suppliers provide material for release, crackers (or rippers, depending on context) remove copy protection and sometimes optional "filler" material, coders write useful helper software for the group and couriers distribute releases around the internet. Not all groups fill every role and some may have other roles I did not describe here (hackers, carders, hardware suppliers to name a few). There are sometimes partnerships between groups with different specialities; usually this occurs when there is an overlap in membership. For example: a release group may have close links with a particular courier group and those guys handle all their releases.
The distribution process is actually quite interesting. Each successful release group aims to have a small list of affiliations with well known and prestigious underground boards/ftp-sites. The quality and prestige of a board depends on the speed of its connection, its capacity, its group affiliations, the standing of its admins in the community and the speed with which new releases are uploaded to it. Each well regarded site is allowed only one particular kind of affiliation: a games group affil, an apps group affil, a courier affil, an ISO affil -- you get the idea. The important thing is that there is only one of each. Affiliations carry prestige for a site and being affiliated with the best sites raises a group's prestige, so there is often heated competition between sites and groups for affils. This is the case at least in the so called "zero second" scene where releases are "traded" (i.e. re-uploaded) by couriers within seconds of being first uploaded. Once a release has been traded among all zero-second sites, they slowly filter down to lesser and lesser sites until eventually making their way to the Internet at large.
So why would you go this all this trouble? The reasons are myriad but usually it comes down to prestige: for example, among couriers there is a longstanding weekly and monthly competition between groups that prove their chops by trying to dominate each other. Similar competitions exist among release groups. There is huge pressure to be the first to release a highly anticipated game or well known application. Additionally, there are strict rules about how to release something -- what to keep, what can (and should) be omitted, how to package everything up, the inclusion of .nfo and .diz files and so on. Failure to comply with these strict standards renders a release invalid and causes it to be "nuked" (deleted) among the top sites. This process amounts to a public shaming of the group responsible and makes it possible for competing groups to snatch the credit by doing a "proper" release.
I could go on and on but I think that's enough for now. Hopefully I've convinced you that pirates aren't an anarchic bunch ;)
[+] [-] shocks|14 years ago|reply
Topsites are fuelled by racers. These are the couriers, who transfer data between multiple topsites they have access to. All topsites have a multiplier. For every 1MB you upload, you can download ~3MB. If you don't each a quota or are not in the top ~15 racers by the end of the week your access will be revoked.
With access comes prestige, ability to race to other topsites, contacts, and leech access to the topsite. These things are many terabytes big and can have archives in excess of 60TB. A huge repository of constantly refreshing and up to date material.
Without rules this would become a free-for-all, with poor quality. That's no good for anyone.
[+] [-] donniezazen|14 years ago|reply
[+] [-] gioele|14 years ago|reply
DVDs are often full of interlaced content (even for short movies), broken subtitles, letterboxed pictures, jerking menus; sometimes they even resize the video to 4:3. The rare downloadable videos are often over compressed or barely compressed (thank you for the 2GB file to this 20 minute show). Similar things happens with bought music: I have paid for many albums distributed as badly-encoded MP3s in zip files with __MACOSX directories and ._DS* files.
[+] [-] sbarre|14 years ago|reply
If content producers released their raw sources and content monetizers had to compete to produce the best possible downloadable and distributable versions, you'd see much better commercial releases. ;-)
[+] [-] rmc|14 years ago|reply
If only all content like this was open source.
[+] [-] GvS|14 years ago|reply
[+] [-] tomku|14 years ago|reply
On the other hand, people who watch non-anime scene releases are much more likely to use mobile devices, set-top boxes and other A/V equipment that relies on hardware decoding support. One of the major reasons that scene releases are standardized is so that they'll play on a wide variety of hardware, and this results in a much more conservative approach than anime groups. The main reason they're moving forward with this change is because H.264 has acceptable support in hardware (primarily in the MP4 container, MKV support is awful) and XviD is really showing its age compared to modern H.264 encoders like x264.
[1] - Media Player Classic - Home Cinema (http://mpc-hc.sourceforge.net) and the Combined Community Codec Pack (http://www.cccp-project.net)
[+] [-] pjscott|14 years ago|reply
Anime fansubbing groups, in contrast, have no common standards except for -- to paraphrase the IETF motto -- rough consensus and regular releases. If some group wants to encode with Hi10P to get the same video quality at 2/3 the file size, they don't have to ask permission from anyone. Competition between groups gradually pushes forward the rough consensus, which is how they switched from xvid to x264, from container formats like AVI to the more flexible MKV, from SD to HD, and so on. The freedom to not live up to a community standard of quality gave them the ability to exceed it.
There's probably a broader lesson in all this.
[+] [-] CoolGuySteve|14 years ago|reply
[+] [-] yxhuvud|14 years ago|reply
[+] [-] dfc|14 years ago|reply
http://scenerules.irc.gs/t.html?id=2011_X264.2.nfo
[+] [-] okamiueru|14 years ago|reply
SD content is mostly being used on smartphones and tablets. For iOS users, this would require a lot of extra work to convert the files to the limited container choices.
[+] [-] ars|14 years ago|reply
What's wrong with FFmpeg?
[+] [-] Mavrik|14 years ago|reply
A) libfaac / libvo_aacenc are pretty bad in terms of quality and have problems with multi-channel audio B) libaacplus only encodes up to 64kbps in less supported AAC+
So Nero and Apple are pretty much only quality free audio encoders for AAC available right now.
[+] [-] tonfa|14 years ago|reply
[+] [-] naner|14 years ago|reply
https://torrentfreak.com/bittorrent-pirates-go-nuts-after-tv...
[+] [-] danberger|14 years ago|reply
[+] [-] wmf|14 years ago|reply
[+] [-] spindritf|14 years ago|reply
RARs are an artefact of FTP transfers during which files would get corrupted, and not-so-fast connections of the past which made re-downloading the whole release somewhat inconvenient and wasteful.
[+] [-] dfc|14 years ago|reply
[+] [-] 0x09|14 years ago|reply
[+] [-] jonmaddox|14 years ago|reply
[+] [-] stevenp|14 years ago|reply
[+] [-] aw3c2|14 years ago|reply
But requiring mp4 as container is a bad idea in my opinion. To playback an mp4 file you need to have it wholly (please correct me if I am wrong) so you cannot start watching before you downloaded it all. MKV would be so much better.
Please do not blindly reply with "But my XYZ device does not support MKV". Scene rules/standards have a lot of impact and might help making vendors support Matroska.
[+] [-] felixge|14 years ago|reply
You are. By default, many tools put the meta data of mp4 files at the end of the file which means you have to have the whole file before playing. However, such mp4 files can be "fixed" using tools like this:
https://github.com/danielgtaylor/qtfaststart
[+] [-] getsat|14 years ago|reply
720p or better videos on youtube are all MP4 now. You just have to move some atoms around within the video file to prepare them for "streaming" playback. There's open source software to do this.
Then again, the ability to play an incomplete file is not really useful if you're fetching a movie via torrent, which is what I would assume most of these guys are using.
[+] [-] wmf|14 years ago|reply
I suspect scene standardization on MKV would probably encourage Apple and Sony to deliberately not support MKV (instead of not supporting it out of laziness, as they currently are).
[+] [-] newman314|14 years ago|reply
No more transcoding from xvid/divx.
[+] [-] asdfaoeu|14 years ago|reply
[+] [-] thadeus_venture|14 years ago|reply
PS: About time with dropping xvid for h264.
[+] [-] politician|14 years ago|reply
[+] [-] mikecane|14 years ago|reply
Well, in a way this will make things even worse for the CopyNazis. No longer will people have to convert from AVI to watch on their tablets and phones.
[+] [-] okamiueru|14 years ago|reply
[+] [-] MrKurtHaeusler|14 years ago|reply
Although I can just imagine all the pirates whining about "h.264 (or mkv as they call it) doesn't play on my dvd player, bring avi (as they call xvid) back".
I can't believe pirates are still burning crappy SD rips to DVD.
Someone I know stopped doing that when he wanted to watch a movie for the second time and realized it was quicker to leech a 1080p bluray rip than search through his mountains of VCD and SVCD discs for the one he downloaded just a couple of years back.
This pirate friend of mine welcomes the change, for the maybe 1 SD show he watches.
Although he personally wishes people paid more attention to sound codecs, nothing worse than downloading something broadcast in 5.1 and finding out the file only has a stereo downmix.
[+] [-] rmccue|14 years ago|reply
1) I watch it in the corner of my screen, so I'm fine with 320. 2) Here in Australia, we pay for usage. I'm not going to download something in a high quality if I'm near my limit
[+] [-] digitalwingx|14 years ago|reply
[+] [-] janogonzalez|14 years ago|reply
[+] [-] ktizo|14 years ago|reply
[+] [-] jrockway|14 years ago|reply
[+] [-] glomph|14 years ago|reply
[deleted]
[+] [-] unknown|14 years ago|reply
[deleted]