This is a pretty old paper, but I hadn't seen the live demo. That's pretty cool.
A problem with this algorithm is that it's pretty slow. It would be hard to use it for real time scaling.
I think that it's pretty cool though for any precalculated conversion to vector format. It would also be pretty cool if there were software that allowed you to create a pixel art image with a live preview in this format and the ability to export into a vector format.
One of the benefits of pixel art IMO is that it's got a pretty distinct style and its simplicity and restrictions means it's easy to be aware of composition where if you are given an unrestricted canvas that freedom can make it hard to know where to start. If you are given a 8x8 pixel grid to indicate "happy" there's only so many ways to do it, a symbolic heart, a little smiley face, etc. Given a whole sheet of paper you'll have more difficulty.
With the high res displays, there's more to consider than just the upscaling of old programs, the fact is people (especially indie game devs) still want to work with pixel art, and the number of titles with the term "8-bit" in their name that have come out recently. Part of that's because it's in vogue now, but a big part of it is simply that a poor artist can make a passable 24x24 sprite.
But it's still a shame that on high res displays they end up looking like a blocky mess. If you had a program that let you author your sprites under the same conditions, but let you preview an upscaled version and left you with a vector image (or high res texture with mip maps) you can make use of the fidelity while maintaining the simplicity. I think that would be cool.
Well SuperEagle seems even slower for me, and I know for a fact that I can emulate with SuperEagle scaling on and it works fine (>30 FPS). So I don't think their algorithm is out of the question.
> If you had a program that let you author your sprites under the same conditions, but let you preview an upscaled version
Combine Atwood's law with a proposed law of mentioning ideas on HN being implemented shortly thereafter and there may be a HTML5+JS implementation popping up as a "Show HN" soon.
I wish they would use this in Windows 8. As far as I can tell, their scaling algorithm for 4k displays is: scale up 2x using nearest neighbor, then randomly drop pixels to get back to 125% or 150%. The result is horrifyingly bad. Not just kind of bad, but unusably bad. (Adding insult to injury, they ClearType the text before scaling.)
While something similar would probably be better, this is heavily optimized for pixel artwork. Lots of curves, shading, and most importantly - limited colors. UI is generally much straighter, less shaded, with far more colors.
Here's an example of UI that's scaled with potrace (a similar algorithm): http://i.imgur.com/jDq4M7e.png (left side is the converted side, right side is nearest-neighbor. Top is 3x, bottom is 2x)
That's with the highest amount of colors/depth that potrace allows. On an i7 processor, it takes a LONG time to compute all of that. Each icon is no longer just a couple hundred pixels, it's a couple hundred vector paths. Even rendering it takes a number of seconds to refresh the screen in inkscape.
Do you know if MS is working on Mac-Style resolution independence for W9? I'm currently running a Retina MBP, at 2x res I think, and it works _perfectly_. The first few months some apps didn't work well, but the community adapted very quickly.
Only if you look at the first ones. All of Boo, Dolphin, all of SoM, Invaders, Golden Axe, Windows assets and Lost Vikings look great and have vastly reduced artifacts compared to hqx. Also, according to the paper abstract[0], it outputs vector, not scaled up bitmap.
even so. I suspect a live rendering of the vector process would be fairly attractive. I'd love to see this technique used on the textures for n64 games. The rendering times wouldn't be such an issue in that case.
The issue with depixelizing pixel art is that you need an AI that can identify semantically what the features in the sprite are. When you look at Mario you know you're looking at a stout little man with a hat, you can identify his eyes, ears, nose, hair, hat, moustache, shirt, gloves, overalls and shoes. Given that knowledge, if I were to depixelize Mario's sprite, I'll know the rough shape of his shoes and where the shading should go and what shape it should have, i.e. I can reason what the 3D shape is and how the 2D shape was derived from that. I'll also be able to identify which lines were put there to make sure separate elements can be legible and not make those thicker than needed. Or that his eyes blend with his hat simply because there aren't enough pixels to make them separate.
For simple shapes, like the background, their algorithm works really well, but for complex objects it fails, because it distorts details that were put in with very careful thought and completely depend on the resolution. Such small sprites rely a lot on being looked at by someone who can identify semantically what they're looking at, and any really successful depixelization solution will need to be able to understand what basic shapes the sprite is made of based on what it's supposed to represent.
I've been thinking about doing something like this. Compress images down to half resolution, then train a neural network to blow it back up and guess the pixels as accurately as possible.
A C++ LGPL implementation of this has been as been developed as part of last year Inkscape GSoC and is available as a standalone library: https://launchpad.net/libdepixelize
It will be released in the next Inkscape major version, expected soon :)
Perfect! Thanks for the link. I want to run this over the classic SimCity tile set (and the additional ones in the add-on graphics sets like the moon base, etc), to generate high resolution pixel and vector tiles for the JavaScript browser based version of Micropolis (GPL SimCity).
The authors of this paper also have other amazing image processing/optimization algorithms. Eg: Deblurring, Optimizing Color Consistency, Content-aware Automatic Photo Enhancement, Object Discovery and Segmentation in Internet Images and more:
> This paper introduces a novel content-adaptive image downscaling method. The key idea is to optimize the shape and locations of the downsampling kernels to better align with local image features. [...] Besides natural images, our algorithm is also effective for creating pixel art images from vector graphics inputs, due to its ability to keep linear features sharp and connected.
IIRC, this is one of the major reasons that vector graphics based icons have not taken off.
Sometimes the Vector Magic results are truly magical; striking poses become even more striking when the figure is reduced to a collection of simpler shapes.
Count me as another one who likes the pixel aesthetic. The title made me ask myself "why would you want to do this?"
(I also loathe most forms of antialiasing, including ClearType, so that might have something to do with it... I like sharp, clearly contrasted edges, including pixel edges.)
I think it comes down to your point of focus. With peach and peach 2, The eyes change shape quite a bit. Humans notice that much more than other aspects.
In general, I think the "ours" method could benefit by moving the boundaries between colours in and out until they get an average brightness match with the similar area in the nearest neighbour version. A lot of the "ours" images have a higher proportion of black.
Super2xSaI looks better to me at 4x than bicubic, and EPX does an OK job but the others look pretty terrible.
I think the problem is that those sprites use multiple strategies for handling diagonal outlines. Those look fine when the image is blurred together, but when you try to detect the outlines and upscale them, the differences between techniques are exaggerated.
yeah, it seems to handle patches of complex colouring badly (especially where two tones of the same colour were used to good effect in the icon, but scaled up as messy blobs), but boo and the invaders were very impressive.
I'm amazed to find that no one has mentioned that ScummVM has implemented some of these algorithms for years and they work pretty nice with the classical games.
HQ4X and Ours are the clear winners if you must resample like this; the downsides are that HQ4X occasionally has jaggies and Ours makes makes a lot of fine details look "blobby".
Regarding HQ4X, they used the algorithm to create 4X image but used Nearest neighbor to create 8X and 16X images. I think doing HQ4X two times would have resulted into a better 16X images without jags.
I did a GPU version in 2012 for my course project. For small size sprite, it runs roughly the same or even slower than the original paper. Anyone interested in finishing this and maybe try this on realtime emulator? http://yzhwang.github.io/depixelization/
I think the most interesting aspect of this is the internal shading. I haven't seen any other algorithm that manages that while maintaining sharp edges overall.
I imagine it would end up with pretty weird results though when a sprite was on top of a background that with similar enough colors to trigger the shading detection at its edges.
[+] [-] zeidrich|12 years ago|reply
A problem with this algorithm is that it's pretty slow. It would be hard to use it for real time scaling.
I think that it's pretty cool though for any precalculated conversion to vector format. It would also be pretty cool if there were software that allowed you to create a pixel art image with a live preview in this format and the ability to export into a vector format.
One of the benefits of pixel art IMO is that it's got a pretty distinct style and its simplicity and restrictions means it's easy to be aware of composition where if you are given an unrestricted canvas that freedom can make it hard to know where to start. If you are given a 8x8 pixel grid to indicate "happy" there's only so many ways to do it, a symbolic heart, a little smiley face, etc. Given a whole sheet of paper you'll have more difficulty.
With the high res displays, there's more to consider than just the upscaling of old programs, the fact is people (especially indie game devs) still want to work with pixel art, and the number of titles with the term "8-bit" in their name that have come out recently. Part of that's because it's in vogue now, but a big part of it is simply that a poor artist can make a passable 24x24 sprite.
But it's still a shame that on high res displays they end up looking like a blocky mess. If you had a program that let you author your sprites under the same conditions, but let you preview an upscaled version and left you with a vector image (or high res texture with mip maps) you can make use of the fidelity while maintaining the simplicity. I think that would be cool.
[+] [-] glhaynes|12 years ago|reply
Even if it's too slow for realtime on today's hardware, I'd love to see some prerendered sample videos.
[+] [-] ahassan|12 years ago|reply
[+] [-] lloeki|12 years ago|reply
Combine Atwood's law with a proposed law of mentioning ideas on HN being implemented shortly thereafter and there may be a HTML5+JS implementation popping up as a "Show HN" soon.
[+] [-] jrockway|12 years ago|reply
[+] [-] jjcm|12 years ago|reply
Here's an example of UI that's scaled with potrace (a similar algorithm): http://i.imgur.com/jDq4M7e.png (left side is the converted side, right side is nearest-neighbor. Top is 3x, bottom is 2x)
That's with the highest amount of colors/depth that potrace allows. On an i7 processor, it takes a LONG time to compute all of that. Each icon is no longer just a couple hundred pixels, it's a couple hundred vector paths. Even rendering it takes a number of seconds to refresh the screen in inkscape.
[+] [-] rkuykendall-com|12 years ago|reply
[+] [-] balls187|12 years ago|reply
[+] [-] bluedino|12 years ago|reply
[+] [-] frik|12 years ago|reply
backup on archive.org: https://web.archive.org/web/20120630044334/http://www.hiend3...
http://en.wikipedia.org/wiki/Hqx
Source: https://code.google.com/p/hqx/source/browse/trunk/src/hq4x.c
[+] [-] lloeki|12 years ago|reply
[0]: http://research.microsoft.com/en-us/um/people/kopf/pixelart/
[+] [-] GhotiFish|12 years ago|reply
[+] [-] whywhywhy5|12 years ago|reply
[+] [-] anon4|12 years ago|reply
For simple shapes, like the background, their algorithm works really well, but for complex objects it fails, because it distorts details that were put in with very careful thought and completely depend on the resolution. Such small sprites rely a lot on being looked at by someone who can identify semantically what they're looking at, and any really successful depixelization solution will need to be able to understand what basic shapes the sprite is made of based on what it's supposed to represent.
[+] [-] Houshalter|12 years ago|reply
[+] [-] kaeso|12 years ago|reply
It will be released in the next Inkscape major version, expected soon :)
[+] [-] SimHacker|12 years ago|reply
[+] [-] rsiqueira|12 years ago|reply
Dani Lischinski's page: http://www.cs.huji.ac.il/~danix/
Johannes Kopf page: http://johanneskopf.de/
[+] [-] vanderZwan|12 years ago|reply
http://research.microsoft.com/en-us/um/people/kopf/downscali...
> This paper introduces a novel content-adaptive image downscaling method. The key idea is to optimize the shape and locations of the downsampling kernels to better align with local image features. [...] Besides natural images, our algorithm is also effective for creating pixel art images from vector graphics inputs, due to its ability to keep linear features sharp and connected.
IIRC, this is one of the major reasons that vector graphics based icons have not taken off.
[+] [-] JasonFruit|12 years ago|reply
[+] [-] vsviridov|12 years ago|reply
[+] [-] Orangeair|12 years ago|reply
[+] [-] th0ma5|12 years ago|reply
[+] [-] 10098|12 years ago|reply
[+] [-] userbinator|12 years ago|reply
(I also loathe most forms of antialiasing, including ClearType, so that might have something to do with it... I like sharp, clearly contrasted edges, including pixel edges.)
[+] [-] GhotiFish|12 years ago|reply
[+] [-] Kiro|12 years ago|reply
[+] [-] smackfu|12 years ago|reply
https://news.ycombinator.com/item?id=2601347
https://news.ycombinator.com/item?id=2578706
[+] [-] rkuykendall-com|12 years ago|reply
[+] [-] jonathanj|12 years ago|reply
[+] [-] sp332|12 years ago|reply
[+] [-] Lerc|12 years ago|reply
In general, I think the "ours" method could benefit by moving the boundaries between colours in and out until they get an average brightness match with the similar area in the nearest neighbour version. A lot of the "ours" images have a higher proportion of black.
[+] [-] mistercow|12 years ago|reply
I think the problem is that those sprites use multiple strategies for handling diagonal outlines. Those look fine when the image is blurred together, but when you try to detect the outlines and upscale them, the differences between techniques are exaggerated.
[+] [-] mberning|12 years ago|reply
[+] [-] Trufa|12 years ago|reply
[+] [-] zem|12 years ago|reply
[+] [-] josh-wrale|12 years ago|reply
[+] [-] omegote|12 years ago|reply
[+] [-] baumgarn|12 years ago|reply
[+] [-] gverri|12 years ago|reply
[+] [-] rangibaby|12 years ago|reply
The Vector Magic results reminded me a lot of this: http://www.cowboybooks.com.au/html/acidtrip1.html
[+] [-] jatin085|12 years ago|reply
[+] [-] yzh|12 years ago|reply
[+] [-] mjmahone17|12 years ago|reply
[+] [-] mistercow|12 years ago|reply
I imagine it would end up with pretty weird results though when a sprite was on top of a background that with similar enough colors to trigger the shading detection at its edges.