Extract the "overall colors" of an image to apply another?

I've been meaning to show my students a way to "borrow" the color palette of an image to apply to another?

Take the Mona Lisa, for example. It has its own "tones" and "mood". Suppose there was a way to load that image into Matlab and extract its "most vital" color characteristic? And then be able to "apply" this "personality" to any other existing image? I know, it's a crazy idea and I've been thinking what could possibly be the "linking mechanism" between the two images or between any two sets of images, for that matter, that would make this possible? Perhaps the "Brightness" levels? To me, it's clearly a problem of "mapping tones" between imiages but I have no idea where to start. But one thing is sure, the students would immediately understand the idea and, from a pedagogical point of view, it would go a long way towards building an "abstract" view of "image processing" which is what the class is about.

Answer 1

[DGM]

The paper that @Image Analyst linked is probably the ideal, but nobody posted a MATLAB implementation yet. I think there are a couple image recoloring examples on the FEX, but if I recall, they're based on semantic segmentation and don't use a reference image as the color source.

MIMT has two somewhat-applicable tools, neither of which are very good for photographic images, but both of which are better than using an unsorted color table.

MIMT imrecolor() uses histogram matching to transfer color information from one image to another. This is accomplished by sorting the pixels into a specified number of bins by brightness and then performing histogram matching on the hue and chroma/saturation data within each of those bins. The result is a brightness-correlated color distribution match between the images. This tool tends to produce patchy results with photographic content, but for simple cases it's sometimes passable. It was originally intended for use with abstract images.

ref = imread('flowers.png');
inpict = imread('shoes.png');

outpict = imrecolor(ref,inpict,'colormodel','hsly');

imshow(outpict)

Compare that to a simple histogram matching in RGB using imhistmatch():

While imhistmatch() does do a better job of reproducing the color distribution, it does so with no regard to preserving the brightness or contrast of the original image.

MIMT also has a simpler way that's similar to the example you gave. MIMT im2ct() was written as a response to this FEX submission, and all it does is create a color table of specified length using the "dominant" colors of an image. The CT is sorted and optionally linearized in a specified color space.

Bear in mind that this is a simple tool, and it should be expected that there are many configurations of images and options which will yield completely garbage results. Often what people call the "dominant colors" are not representative of the dominant fraction of the image, and there's no reason to expect that there is a monotonic and visually pleasing trajectory between them. It's usually best to use only a small number of sample points (nbreaks) to avoid banding.

Instead of applying this color table directly using ind2rgb(), we reduce the image to its corresponding brightness component and quantize that. In this manner, we're maintaining some brightness relationship. In MIMT, the tool to use would be gray2pcolor().

ref = imread('flowers.png');
inpict = imread('shoes.png');

CT = im2ct(ref,'ncolors',256,'nbreaks',5,'uniform',true, ...
    'cspace','ypbpr','fullrange',true,'minsat',0.15);
outpict = gray2pcolor(inpict,CT,imclassrange(class(inpict)),'default');

% display both the CT and the recolored image
subplot(8,1,1)
image(rot90(ctflop(CT)))
subplot(8,1,2:8)
imshow(outpict)

[Roger Breton]

Thank you so much for your excellent suggestions. I have been researching the whole question of "color palettes" and "harmonious colors" for a while. Ultimately, I'd like to use these "case studies" to introduce students to Munsell colors. I know, it sounds crazy but wish I could get them away from thinking in "PANTONE" meaningless numbers. I have gathered quite a collection of colored images from the web, many paintings but also natural sceneries and animals such as birds which I find the color of the plumage to offer lots of starting points for graphic design students. I experimented with k-means which offers some level of image's color analysis, based on frequencies. Problem is, it's based on "counts"? So, if I want to study the relationship between colors on a bird, I'd have first to do some kind of pre-processing to get rid on background and other meaningless content. See this bird:

I manually extracted the "important/dominant" colors based on my judgement using Adobe Illustrator Color Picker tool. I could have accomplished the similar result in Photoshop but, in an "ideal" world, I would have love for a K-means or other similar tool to give me this "color palette" algorithmically. Adobe has a site called https://color.adobe.com/ which offers an automated color extraction function, based on their analysis of the image but it has limitations which I'm trying to work around in my research.

I was thinking about, after having read the image in memory, to convert to CIE Lab and CIE Lch to sort pixels by Chroma. And try to "regroup" colors by "groups", similar to the idea of a histogram.

I tried this https://1drv.ms/b/s!AkD78CVR1NBqm8N15yiF9PYUTKjNDQ?e=ra30da

I also tried this approach:

As you can see, on the horizontal axis, I plotted all CIE Hues (Lch) from 0 to 360degrees and on the vertical axis, I plotted the frequencies, to show where the colors of an image are "concentrated". My idea was to try to isolate the "major hues" to plot them on a CIE ab plot to try to show underlying "rules of harmonies".

My latest efforts, as you can see, is using Index colors to try to extract the important/dominant colors in an image. I thought I'd show the students how extracting dominant colors in an image, using an indexed approach, could then be applied to the problem of "recoloring artwork". I know Adobe Illustrator has a tool dedicated to just that but it does not far enough for my taste and is based on an RGB/HSB model. I'd like to show students how to use Munsell colors to get them to "think" colors in a more intuitive way...

[DGM]

I'm not the color sci guy around here, but even as much as I've tinkered with it, I don't have a good solution to the "thematic colors" extraction problem.

For the purposes of color transfer, I'd think that a method used for extraction would have a dual that could be used to apply them to the working image. The problem that always stumped me was that once we get a good color table by finding characteristic groups in some color space, how do we arbitrarily reshape the color distribution of the working image to match without causing local inversions or banding? The problem is that the grouping of colors in LAB or RGB tends to be only weakly associated with the representative colors of each particular object in an image, and they tend to overlap anyway. Moving the group dominated by one object will often create bands and blobs elsewhere where once-smoothly graduated shadows or highlights move away from their spatial neighbors, simply due to where the group boundaries were chosen in the color space.

The Grundland paper (if I recall) addresses this, but that was quite a bit over my head.

Answer 2

[Richard Burnside]

The imread command will return the indexed image and the 256x3 colormap of the image:

[X,cmap] = imread('corn.tif');

You can work with that colormap to apply it to another image using the colormap command.

[Roger Breton]

Thanks so much! That ought to get me started :-)

[Roger Breton]

I wanted to share my code, in case it would help others who are looking to do the same thing I want to do. First, here is a screen capture of the result (I have a 4K screen):

And here is the code:

RGB = imread('FleursLilac.png');
[X,cmap] = rgb2ind(RGB,32);
% Sort colormap by saturation value
hsv_cmap = rgb2hsv(cmap);
sorted_hsv_cmap = sortrows(hsv_cmap, -2);
sorted_rgb_cmap = hsv2rgb(sorted_hsv_cmap);
% Read another image
RGB2 = imread('Brown.jpg');
% Apply colormap to new image
[X2,cmap2] = rgb2ind(RGB2,32);
RGB2_indexed = uint8(X2);
RGB2_colored = ind2rgb(RGB2_indexed, cmap); % convert back from Index to RGB
A = reshape(1:36, 6, 6);  % A = reshape(1:256, 16, 16);
subplot(1,4,1)
imshow(RGB)
title('Original RGB Image')
subplot(1,4,2)
imshow(RGB2)
title('Target Image')
subplot(1,4,3)
% Display new image with colormap
imshow(RGB2_colored)
title('Aplied Colormap')
subplot(1,4,4)
imagesc(A)
colormap(cmap)
title('Colormap Q=36')

---------------------------

The result, so far, is nothing to write home about...

I see a number of "problems" which I'll try to "break down" with a cup of coffee... But it's clear I have no control over the "mapping"?

[Roger Breton]

Thinking about the result out loud... It is clear that the mapping is applied without any consideration to the original "tones" in the Target Image. Clearly, I need to do a better job of matching tones using some kind of 'criteria' OR, which could be not too trivial, come up with an interface that would allow me to manually match Input tones to those in the cmap, like an Input/Output table. How to do that in Matlab? No idea how to do that... I'd have to display a "table" of some kind with initial values? Another cup of coffee...

[Richard Burnside]

The answer supplied below by Image Analyst seems to have that process worked out. Pretty amazing results.

Answer 3

[Image Analyst]

The best algorithm I've seen for transferring the color gamut is Mark Grundland Color Histogram Specification by Histogram Warping

Sorry, I don't have code for that algorithm.

[Roger Breton]

I'll take a look at it, thanks a million!

[Image Analyst]

If youi can do it, let me know. And post it to the File Exchange. A couple of years ago I made a s tab at it and I got the first few steps done but then it got into some pretty sophisticated math that I didn't know how to translate into MATLAB so I gave it up.

[Roger Breton]

I don't think I can pull this, no :(