As a professional photo editor and historian, colorized photos really agitate me. I'm all for the creation of new ways to get people to engage with historical primary documentation, but the nuance that these colorizations are interpretations gets lost immediately.
Do an image search for "D-Day in color" and try to tell me which results are original color negatives and which are colorizations made by teenagers.
I'm also a little confused as to why colorizations always aim to restore color to the equivalent of a faded color negative, with muted tonality and grain. Human logic is funny.
I'm also a little confused as to why colorizations always aim to restore color to the equivalent of a faded color negative, with muted tonality and grain. Human logic is funny.
Well, as a professional photo editor, you know that if the original b+w photo captures an image with say, a 50% grey value, you don't know if the original color was bright red, or closer to that 50% gray. Bright red has a much higher chroma value, but chroma isn't recorded in a b+w photo - color saturation is lost. Easily demonstrated by making an image in your favorite image editing program that's just straight up red, then changing the image mode to grayscale, then asking someone else entirely to guess the original color.
That, and I think the style is to mimic hand tinting photographs, where you would paint right on a b+w photo. The colors would look "faded" because whatever was used to tint the photograph needed to have a transparent medium, for the information of the photograph itself to shine through. That, and there's just so many colors you could use when hand-tinting. Back to our 50% gray. What if that was... bright yellow? You can't tint "bright yellow" onto a 50% gray area of a photograph. Yellow is highly transparent, and the grey would be too powerful to let its chroma value shine through.
I like colorized historical photos. It brings them to life in a remarkable way. I'd like to see the old B+W movies colorized (the Ted Turner ones don't count, as they were done very poorly).
Sure, the colors will never be exact, because we don't know what the original colors were. But that doesn't matter in any material way.
> the nuance that these colorizations are interpretations gets lost immediately.
True for all AI; neural networks are doing amazing things, but the output is a synthesis, it's a complex interpolation of it's training inputs that may seem "good" or reliable, but it is never to be taken as truth or fact, and it can be arbitrarily wrong with unbounded errors.
> I'm also a little confused as to why colorization always aim to restore color to the equivalent of a faded color negative, with muted tonality and grain. Human logic is funny.
This isn't a human logic problem. Normally colorizations don't affect tonality and grain much, they are putting color splashes on top of a B/W image. This is true of hand-painted colorization, as well as the digital colorization here. You can't get rid of grain or adjust tone by adding color.
One can adjust tone and grain, but then you're doing more than colorizing, and going even further down the road of "interpretation" you're concerned about.
In this particular case, the author did mention "A more diverse dataset makes the pictures brownish". Brown is the average color in natural photos, so minimizing error tends to make things browish. That is separate from leaving faded tone & grain in tact, but it's a second reason why AI based colorization will tend toward muted color.
I'm also a little confused as to why colorizations always aim to restore color to the equivalent of a faded color negative, with muted tonality and grain. Human logic is funny.
I always assumed that if you tried to use "full color" it would look weird, since the photos themselves usually are quite faded and grainy.
As a professional photo editor and historian, colorized photos really agitate me.
As a photographer, collector and enthusiast of vintage prints and photos and amateur historian, I deeply appreciate the aesthetic of colorized photos and understand the motivation to reproduce and master it. There is more to art than truth.
The artist is not the transcriber of the world, he is its rival. - L'Intemporel (Third volume of 'The Metamorphosis of the Gods'), André Malraux (1957)
> I'm also a little confused as to why colorizations always aim to restore color to the equivalent of a faded color negative, with muted tonality and grain.
Many movies set in the past, or that have flashbacks to the past, will often mute the colors. Most modern movies muck around with the colors in post production, too. The worst is when they go for the blue/orange palette.
Another thing that is rarely mentioned, is that it’s very common to use color filters in B&W photography to tune the contrast.
That means the luminance values can be completely different from what they actually were, and skew the color choices. On the other hand, this might be something that a good ML algorithm can detect and compensate for.
I think there was a cable TV channel a longtime ago which might have been a Turner property --they would get rights to some classic B&W movie and colorize it --I'm sure the tech used back then was somewhat primitive but the results were horrific --but of course "Color!"
This is refreshing. I’ve been learning machine learning through Kaggle. Recently and I’m a bit tired with the “tuning hyperparameter” culture. It rewards people that have the pockets to spend on computing power and the time to try every parameter. I’m starting to find problems that don’t have a simple accuracy metric more interesting. It forces me to understand the problem and think in new ways, instead of going down a checklist of optimizations.
You can be a little less brute force if you use something like hyperopt (http://hyperopt.github.io/hyperopt/) or hyperband (https://github.com/zygmuntz/hyperband) for tuning hyperparameters (Bayesian and multi-armed bandit optimization, respectively). If you're more comfortable with R, then caret supports some of these types of techniques as well, and mlr has a model-based optimization (https://github.com/mlr-org/mlrMBO) package as well.
These types of techniques should let you explore the hyperparameter space much more quickly (and cheaply!), but I agree - having money to burn on EC2 (or access to powerful GPUs) will still be a major factor in tuning models.
Ha, it reminds me of what Andrej Karpathy said "Kaggle competitions need some kind of complexity/compute penalty. I imagine I must be at least the millionth person who has said this." It would be interesting to collaborate/compete on more creative tasks and have different metrics for success.
The averaging problem in colorization is interesting. If it learns that an apple can be red, green and even yellow - how does it know how to color it?
A HN user in an earlier thread suggested to use a fake/real colorization classifiers as a loss function. [1] But I still feel that it would not solve the averaging problem. It would hop between different colors and probably converge to brown. I haven’t come across a plausible solution so far.
[1] https://news.ycombinator.com/item?id=10864801
>But I still feel that it would not solve the averaging problem. It would hop between different colors and probably converge to brown.
At least to the extent that GANs work, it works. They will alternate between the observed colours based on the noise vector. They do not simply converge to averages, because the discriminator easily recognizes brown apples as fakes.
It could try to classify the apple tree or the context, but it would require a lot of training data. If it's out of context, it should select a color based on probability. But it's hard to solve this with just input and output data. The simple solution is to use noncontradictory training data, i.e. only having green apples.
I have an urge to teach it simple logic. Instead of making it brown, it selects the color with the highest probability from a range of colors. However, I haven't come across a deep learning implementation like this to mimic.
Although it is quite egregious here - this is not a problem inherent to colorization but rather to generative models in general.
Using something akin to a variational autoencoder would solve this problem, because it learns a distributional approximation rather than a single point estimate of the color, and then the random noise vector input allows one to sample from this output distribution.
Similarly, Mixture Density Networks allow you to model a distribution and then sample from it.
You could adjust the error function. The common Root Mean Square error pushes predictions to the average. If you use absolute errors, or even a logistic function instead, you'll encourage the model to commit to a decision on a multimodal distribution.
Alternatively, use a discrete colour space and consider colours as categorical data not implying any ordinal scale.
Interesting! If nowadays, pictures are colorized by hand in photoshop, it wouldn't be practical to colorize a full black and white movie. I guess this deep learning approach would solve this problem and colorize old black and white classic movies.
Agreed. I imagine this has applications in compression as well. You could stream a movie (or a football game) in black and white and enable each device to color it on the spot. A similar technique could also be done for HD/3D/VR.
I'd like to see more than colorization. Consider the silent movie "Wings". Very high quality blu-rays are available of it. I would colorize it, remove the dialog cards and dub the dialog, then add foley sound effects and a music soundtrack!
I'd like to train this on color comic strips and then run something traditionally black and white like xkcd through it. Seems like it could make the colorization part of hand drawn animation much easier.
What's even cooler is adding support for human annotations so users can selectively give colorization hints for different parts of the image to customize the output.
A GAN style approach to learning and generating variants could be interesting as well. It could generate a couple of hundred plausible versions. Then you have another network that is trained to differentiate between fake and real colored photos which picks the best version.
[+] [-] vwcx|8 years ago|reply
Do an image search for "D-Day in color" and try to tell me which results are original color negatives and which are colorizations made by teenagers.
I'm also a little confused as to why colorizations always aim to restore color to the equivalent of a faded color negative, with muted tonality and grain. Human logic is funny.
[+] [-] justinator|8 years ago|reply
Well, as a professional photo editor, you know that if the original b+w photo captures an image with say, a 50% grey value, you don't know if the original color was bright red, or closer to that 50% gray. Bright red has a much higher chroma value, but chroma isn't recorded in a b+w photo - color saturation is lost. Easily demonstrated by making an image in your favorite image editing program that's just straight up red, then changing the image mode to grayscale, then asking someone else entirely to guess the original color.
That, and I think the style is to mimic hand tinting photographs, where you would paint right on a b+w photo. The colors would look "faded" because whatever was used to tint the photograph needed to have a transparent medium, for the information of the photograph itself to shine through. That, and there's just so many colors you could use when hand-tinting. Back to our 50% gray. What if that was... bright yellow? You can't tint "bright yellow" onto a 50% gray area of a photograph. Yellow is highly transparent, and the grey would be too powerful to let its chroma value shine through.
[+] [-] WalterBright|8 years ago|reply
Sure, the colors will never be exact, because we don't know what the original colors were. But that doesn't matter in any material way.
[+] [-] dahart|8 years ago|reply
True for all AI; neural networks are doing amazing things, but the output is a synthesis, it's a complex interpolation of it's training inputs that may seem "good" or reliable, but it is never to be taken as truth or fact, and it can be arbitrarily wrong with unbounded errors.
> I'm also a little confused as to why colorization always aim to restore color to the equivalent of a faded color negative, with muted tonality and grain. Human logic is funny.
This isn't a human logic problem. Normally colorizations don't affect tonality and grain much, they are putting color splashes on top of a B/W image. This is true of hand-painted colorization, as well as the digital colorization here. You can't get rid of grain or adjust tone by adding color.
One can adjust tone and grain, but then you're doing more than colorizing, and going even further down the road of "interpretation" you're concerned about.
In this particular case, the author did mention "A more diverse dataset makes the pictures brownish". Brown is the average color in natural photos, so minimizing error tends to make things browish. That is separate from leaving faded tone & grain in tact, but it's a second reason why AI based colorization will tend toward muted color.
[+] [-] nerdponx|8 years ago|reply
I always assumed that if you tried to use "full color" it would look weird, since the photos themselves usually are quite faded and grainy.
[+] [-] contingencies|8 years ago|reply
As a photographer, collector and enthusiast of vintage prints and photos and amateur historian, I deeply appreciate the aesthetic of colorized photos and understand the motivation to reproduce and master it. There is more to art than truth.
The artist is not the transcriber of the world, he is its rival. - L'Intemporel (Third volume of 'The Metamorphosis of the Gods'), André Malraux (1957)
[+] [-] WalterBright|8 years ago|reply
Many movies set in the past, or that have flashbacks to the past, will often mute the colors. Most modern movies muck around with the colors in post production, too. The worst is when they go for the blue/orange palette.
[+] [-] ricardobeat|8 years ago|reply
That means the luminance values can be completely different from what they actually were, and skew the color choices. On the other hand, this might be something that a good ML algorithm can detect and compensate for.
[+] [-] mc32|8 years ago|reply
[+] [-] terrabytes|8 years ago|reply
[+] [-] emilwallner|8 years ago|reply
[1] http://colah.github.io/
[2] https://iamtrask.github.io/
[3] https://distill.pub
[4] https://experiments.withgoogle.com/ai
[+] [-] perturbation|8 years ago|reply
These types of techniques should let you explore the hyperparameter space much more quickly (and cheaply!), but I agree - having money to burn on EC2 (or access to powerful GPUs) will still be a major factor in tuning models.
[+] [-] wrangler99|8 years ago|reply
[1] https://twitter.com/karpathy/status/913619934575390720
[+] [-] iluvmylife|8 years ago|reply
A HN user in an earlier thread suggested to use a fake/real colorization classifiers as a loss function. [1] But I still feel that it would not solve the averaging problem. It would hop between different colors and probably converge to brown. I haven’t come across a plausible solution so far. [1] https://news.ycombinator.com/item?id=10864801
[+] [-] zardo|8 years ago|reply
At least to the extent that GANs work, it works. They will alternate between the observed colours based on the noise vector. They do not simply converge to averages, because the discriminator easily recognizes brown apples as fakes.
[+] [-] emilwallner|8 years ago|reply
I have an urge to teach it simple logic. Instead of making it brown, it selects the color with the highest probability from a range of colors. However, I haven't come across a deep learning implementation like this to mimic.
[+] [-] zan2434|8 years ago|reply
Using something akin to a variational autoencoder would solve this problem, because it learns a distributional approximation rather than a single point estimate of the color, and then the random noise vector input allows one to sample from this output distribution. Similarly, Mixture Density Networks allow you to model a distribution and then sample from it.
[+] [-] matt4077|8 years ago|reply
Alternatively, use a discrete colour space and consider colours as categorical data not implying any ordinal scale.
[+] [-] taneq|8 years ago|reply
I dunno, does it look more like a red apple, a green apple or a yellow apple?
[+] [-] fizx|8 years ago|reply
[+] [-] strin|8 years ago|reply
[+] [-] flsantos|8 years ago|reply
[+] [-] zimpenfish|8 years ago|reply
https://en.wikipedia.org/wiki/List_of_black-and-white_films_...
[+] [-] saip|8 years ago|reply
[+] [-] WalterBright|8 years ago|reply
[+] [-] whatrocks|8 years ago|reply
[+] [-] web007|8 years ago|reply
An example implementation would look something like edges2cats https://affinelayer.com/pixsrv/
[+] [-] quotemstr|8 years ago|reply
I love how we can, in general, elevate the sophistication of ML models by having different models interact and train each other.
[+] [-] bootcat|8 years ago|reply
[+] [-] ReDeiPirati|8 years ago|reply
[+] [-] houqp|8 years ago|reply
[+] [-] saip|8 years ago|reply
[+] [-] doppenhe|8 years ago|reply