Sigh. TL;DR version - scaling pixels makes them blocky, scaling vectors doesn't, for some small range of scaling we can keep upscaled pixels not blocky.
Upscaling has a lot of research behind it. Creating a 'vectorized' version of an image (see their paper here: http://eprints.gla.ac.uk/47879/1/ID47879.pdf) certainly helps avoid dumb scaling issues. But when you blow it up 7x (like the example) it looks like a cartoon or a bad paint by numbers painting. The issue is that you lose information about the picture when you capture it, and without that information you have to 'make up' what you think would have been there had you been able to capture it. People get a lot of mileage out of the fact that for the most part real world imagery has a lot of linear transitions but there are limits to that as well.
I think the work is interesting but as they say in their paper: "What we have shown already is that there is a viable continuous image format and that it can be used for some conventional operations which are not handled easily in
sampled formats." which I agree with, but the pixel is not yet in danger of extinction :-)
If JPG and video codecs already store information in terms of wavelets, frequencies, or whatever have you --
Are there any programs that will render JPG or videos at a higher-than-native resolution?
I know you're not gaining any extra signal. But if you have a 320x240 video, and you want to play it full-screen at 1280x960 -- instead of rendering at 320x240 and then naively upscaling 4x, would some kind of direct codec rendering to 1280x960 possibly produce more accurate results?
If the codec/format is doing a good job at extracting "meaning" from the original image for its compression, it seems like it ought to.
For audio, there's spectral band replication which works well in a large part because harmonics make audio rather correlated across multiple frequencies. Images have much less correlation, and thus processing in the frequency domain is less useful.
Additionally, especially with codecs with a small DCT like JPEG, it's been shown that once you reach a certain quantizer (throwing away all higher frequencies, effectively downscaling in the frequency domain), it becomes better to do a traditional downscale + upscale around the lossy coding.
Or, to put it another way, you can exploit the properties of a how a specific lossy codec discards information to get closer to the original image (deblocking filters are the most common example). There is a significant amount of research on this topic for JPEG. But if you could exploit the properties of the DCT or wavelets for better upscaling, it would work equally well as a codec-agnostic algorithm. Frequency transforms are perfectly reversible assuming infinite precision math, after all.
My guess would be that those upscalers in TVs and whatnot do that..
edit:
While googling i didn't find good information on how those hardware upscalers work, but it seems to be the case that it's mostly some pixel based stuff with edge detection and interpolation based on the resulting pixel map.
It's a Sample. There are many ways to filter a collection of samples to reconstruct an image. Drawing bigger squares with nearest-neighbor sampling is only one of them. Depending on the context of your application, you might even go as far as the linked article: http://www.extremetech.com/gaming/84040-depixelizing-pixel-a...
TLDR: Imagine a CRT/Osciloscope not sweeping row by row but instead line by line, where line doesn't necessarily mean horizontal but can be diagonal and even turn midway.
The method they use for vectorization is cool (vectorizing Y, U and V independently) and also illustrates why subsampling UV yields such good results (for standard raster-based codecs).
Ultimately however vectors are far more expensive to rasterize than bitmaps, especially if you want antialiasing along edges. There was a bunch of noise a while back about NVIDIA's path rendering extension -- they do antialiasing via 16x multisampling (!!!) NOT a mobile friendly approach.
>There was a bunch of noise a while back about NVIDIA's path rendering extension -- they do antialiasing via 16x multisampling (!!!) NOT a mobile friendly approach.
Do you have any links? I've seen the NVIDIA talk about the extension, but I didn't hear any of the noise about issues with it.
I'm struggling to understand what the point is. Video codecs don't store data in "pixel" format either, they store wavelet coefficients, data from the frequency domain.
From the article, the source data is in pixel format. No codec format can be better than the source data. And the output is displayed on a screen with pixels. Pixels in, pixels out. The intermediate format can only make things worse, not better. It's not surprising that they're able to do better than current formats: it sounds like they're using a huge amount of CPU for encoding. You could get better compression if you created a new MPEG4 profile that required more CPU too.
The intermediate format may be tweaked to provide better results while scaling, but you could probably add scaling hints to more conventional codecs, too. Not sure what the point would be either unless the source data had a lot more pixels than the output format...
>From the article, the source data is in pixel format. No codec format can be better than the source data.
Not necessarily true. For example if I have a bitmap picture of a printed circle, I can use software to find the shape and convert it to a vector representation of a circle (which would be infinitely better than the source image).
It works very well with scanned line art, for example.
Summary: Video Codec based on vectorization of images rather than pixel encoding. Slow, but highly parallel. Not much info. Developers hope to have a codec for commercial use in some number of years.
Vector art is not often the holy grail as it seems. Take fonts, for instance. Even though formats like TrueType stores the data in vector format, numerous resolution-dependent "hints" have to be added to make it more readable.
Similarly vector icons appear more blurry than pixelled ones especially at lower resolutions. Extremely scaled images are unimpressive due to loss of detail.
Vector art often works well in a certain range of resolutions. The "infinite scaling" promise should be taken with a grain of salt.
> Vector art is not often the holy grail as it seems. Take fonts, for instance. Even though formats like TrueType stores the data in vector format, numerous resolution-dependent "hints" have to be added to make it more readable.
While true, font hinting is thankfully finally going the way of the dodo. Cell phones and other handheld devices has very crisp fonts without any aliasing and hinting thanks to their high pixel density. The magic threshold is at about 150 dpi which when passed makes hinting redundant. Desktop displays are getting there, albeit slowly. To slowly for my taste. OS X already doesn't use hinting which is why if you are used to Windows fonts, os x fonts can look blurry. But with then new retina displays os x's text rendering simply looks amazing.
Vector art can look very good when optimised for smaller sizes. The issue is that the current formats and engines for vector use on the web.
I've spent a lot of time trying to tweak my vector icons to work well in various browsers, and its next to impossible at the moment. More often than not SVG graphics are larger than their bitmap counterparts, almost always crash WebKit browsers, or take 5-10 seconds to render. Safari on iOS will crash if you zoom in on even the most simple graphics.
I love vector artwork, I truly do. It's the rendering engines that are letting us down.
It depends on the nature of the work. I could make a beautiful illustration, then scale it from here to the moon and, at least to me in my hypothetical situation, it would still look good.
Like all things, it depends. Some people crave detail, some crave resolution, some crave anything beyond the monotony of everyday life - regardless of said work's mediocrity.
Vector art can be good. Pixel art can be good. Neither are going away any time soon. I think that's the only rational conclusion to any of this.
ChuckMcM|13 years ago
Upscaling has a lot of research behind it. Creating a 'vectorized' version of an image (see their paper here: http://eprints.gla.ac.uk/47879/1/ID47879.pdf) certainly helps avoid dumb scaling issues. But when you blow it up 7x (like the example) it looks like a cartoon or a bad paint by numbers painting. The issue is that you lose information about the picture when you capture it, and without that information you have to 'make up' what you think would have been there had you been able to capture it. People get a lot of mileage out of the fact that for the most part real world imagery has a lot of linear transitions but there are limits to that as well.
I think the work is interesting but as they say in their paper: "What we have shown already is that there is a viable continuous image format and that it can be used for some conventional operations which are not handled easily in sampled formats." which I agree with, but the pixel is not yet in danger of extinction :-)
crazygringo|13 years ago
If JPG and video codecs already store information in terms of wavelets, frequencies, or whatever have you --
Are there any programs that will render JPG or videos at a higher-than-native resolution?
I know you're not gaining any extra signal. But if you have a 320x240 video, and you want to play it full-screen at 1280x960 -- instead of rendering at 320x240 and then naively upscaling 4x, would some kind of direct codec rendering to 1280x960 possibly produce more accurate results?
If the codec/format is doing a good job at extracting "meaning" from the original image for its compression, it seems like it ought to.
brigade|13 years ago
For audio, there's spectral band replication which works well in a large part because harmonics make audio rather correlated across multiple frequencies. Images have much less correlation, and thus processing in the frequency domain is less useful.
Additionally, especially with codecs with a small DCT like JPEG, it's been shown that once you reach a certain quantizer (throwing away all higher frequencies, effectively downscaling in the frequency domain), it becomes better to do a traditional downscale + upscale around the lossy coding.
Or, to put it another way, you can exploit the properties of a how a specific lossy codec discards information to get closer to the original image (deblocking filters are the most common example). There is a significant amount of research on this topic for JPEG. But if you could exploit the properties of the DCT or wavelets for better upscaling, it would work equally well as a codec-agnostic algorithm. Frequency transforms are perfectly reversible assuming infinite precision math, after all.
joeld42|13 years ago
buster|13 years ago
edit: While googling i didn't find good information on how those hardware upscalers work, but it seems to be the case that it's mostly some pixel based stuff with edge detection and interpolation based on the resulting pixel map.
geon|13 years ago
corysama|13 years ago
It's a Sample. There are many ways to filter a collection of samples to reconstruct an image. Drawing bigger squares with nearest-neighbor sampling is only one of them. Depending on the context of your application, you might even go as far as the linked article: http://www.extremetech.com/gaming/84040-depixelizing-pixel-a...
kemiller|13 years ago
Too|13 years ago
TLDR: Imagine a CRT/Osciloscope not sweeping row by row but instead line by line, where line doesn't necessarily mean horizontal but can be diagonal and even turn midway.
alexk7|13 years ago
pretoriusB|13 years ago
So, even if the display has pixels it doesn't matter, the interesting thing to eliminate is pixels in media (image/video).
randallu|13 years ago
Ultimately however vectors are far more expensive to rasterize than bitmaps, especially if you want antialiasing along edges. There was a bunch of noise a while back about NVIDIA's path rendering extension -- they do antialiasing via 16x multisampling (!!!) NOT a mobile friendly approach.
asdfs|13 years ago
Do you have any links? I've seen the NVIDIA talk about the extension, but I didn't hear any of the noise about issues with it.
bryanlarsen|13 years ago
From the article, the source data is in pixel format. No codec format can be better than the source data. And the output is displayed on a screen with pixels. Pixels in, pixels out. The intermediate format can only make things worse, not better. It's not surprising that they're able to do better than current formats: it sounds like they're using a huge amount of CPU for encoding. You could get better compression if you created a new MPEG4 profile that required more CPU too.
The intermediate format may be tweaked to provide better results while scaling, but you could probably add scaling hints to more conventional codecs, too. Not sure what the point would be either unless the source data had a lot more pixels than the output format...
pretoriusB|13 years ago
Not necessarily true. For example if I have a bitmap picture of a printed circle, I can use software to find the shape and convert it to a vector representation of a circle (which would be infinitely better than the source image).
It works very well with scanned line art, for example.
jgeralnik|13 years ago
jws|13 years ago
PDF of authors' previous work on still images: http://eprints.gla.ac.uk/47879/1/ID47879.pdf
Still too long?: "Mona Lisa in 50 Polygons[1]" Grows Up.
␄
[1] http://rogeralsing.com/2008/12/07/genetic-programming-evolut...
Florin_Andrei|13 years ago
arocks|13 years ago
Similarly vector icons appear more blurry than pixelled ones especially at lower resolutions. Extremely scaled images are unimpressive due to loss of detail.
Vector art often works well in a certain range of resolutions. The "infinite scaling" promise should be taken with a grain of salt.
bjourne|13 years ago
While true, font hinting is thankfully finally going the way of the dodo. Cell phones and other handheld devices has very crisp fonts without any aliasing and hinting thanks to their high pixel density. The magic threshold is at about 150 dpi which when passed makes hinting redundant. Desktop displays are getting there, albeit slowly. To slowly for my taste. OS X already doesn't use hinting which is why if you are used to Windows fonts, os x fonts can look blurry. But with then new retina displays os x's text rendering simply looks amazing.
nwh|13 years ago
I've spent a lot of time trying to tweak my vector icons to work well in various browsers, and its next to impossible at the moment. More often than not SVG graphics are larger than their bitmap counterparts, almost always crash WebKit browsers, or take 5-10 seconds to render. Safari on iOS will crash if you zoom in on even the most simple graphics.
I love vector artwork, I truly do. It's the rendering engines that are letting us down.
bennyg|13 years ago
Like all things, it depends. Some people crave detail, some crave resolution, some crave anything beyond the monotony of everyday life - regardless of said work's mediocrity.
Vector art can be good. Pixel art can be good. Neither are going away any time soon. I think that's the only rational conclusion to any of this.
czr80|13 years ago
jacquesm|13 years ago
nnnnni|13 years ago