CDR Tickets

Issue Number 3431
Summary Service for resizing images
Created 2011-10-14 11:12:36
Issue Type Improvement
Submitted By Beckwith, Margaret (NIH/NCI) [E]
Assigned To alan
Status Closed
Resolved 2012-01-19 13:25:27
Resolution Fixed
Path /home/bkline/backups/jira/ocecdr/issue.107759
Description

BZISSUE::5125
BZDATETIME::2011-10-14 11:12:36
BZCREATOR::Margaret Beckwith
BZASSIGNEE::Alan Meyer
BZQACONTACT::Kevin Broun

Kevin Broun requested that we develop a service for resizing images. Bob has already done an estimate of time required and determined that it would be fairly quick. Here is the email exchange:

Awesome, thanks very much! Copying Chuck as he is PM'ing our work with NIH in this area.

Thanks,
Kevin

---Original Message---
From: Beckwith, Margaret (NIH/NCI) [E]
Sent: Wednesday, October 05, 2011 4:31 PM
To: Broun, Kevin (NIH/NCI) [E]; Kline, Robert (NCI)
Subject: RE: image resizing server?

Hi Kevin,

I think it is fine for Bob to do this. He didn't think it will take too long. I will put an issue in bugzilla for this.

Thanks,

Margaret

---Original Message---
From: Broun, Kevin (NIH/NCI) [E]
Sent: Wednesday, October 05, 2011 2:06 PM
To: Kline, Robert (NCI)
Cc: Beckwith, Margaret (NIH/NCI) [E]
Subject: RE: image resizing server?

Did you guys conclude whether this would be a feasible effort for Bob to undertake? Thanks!

---Original Message---
From: Broun, Kevin (NIH/NCI) [E]
Sent: Friday, September 09, 2011 6:32 PM
To: Kline, Robert (NCI)
Cc: Beckwith, Margaret (NIH/NCI) [E]
Subject: Re: image resizing server?

On a CDR server if possible. Thanks.

          • Original Message -----
            From: Kline, Robert (NCI)
            Sent: Friday, September 09, 2011 06:17 PM
            To: Broun, Kevin (NIH/NCI) [E]
            Cc: Beckwith, Margaret (NIH/NCI) [E]
            Subject: Re: image resizing server?

On 09/09/2011 05:55 PM, Broun, Kevin (NIH/NCI) [E] wrote:
> On the traffic, looking at 2009-2011, the current VOL averages about 40-50 new images per month, with a high of 161 in a month (presumably there would be 3 image resizing queries for each new image). We're also looking at deploying an NIH version of the VOL site - NIH's "Image Bank" is a smaller volume image library at this point, but would presumably add a similar load going forward.

Do you have an existing server with Python and a web server on it, or were you hoping we'd deploy it on one of the CDR servers?

--
Bob Kline

Comment entered 2011-10-21 15:01:26 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2011-10-21 15:01:26
BZCOMMENTOR::Bob Kline
BZCOMMENT::1

This has been implemented on Bach. The url is

http://bach.nci.nih.gov/cgi-bin/cdr/ResizeImage.py

Send a POST request to this service with the following parameters:

image
bytes for the original JPEG image; required
percent
floating point value (without "%") for percentage to scale image; optional
width
integer representing new width of scaled image; optional
quality
integer representing JPEG compression quality; optional
sharpen
non-negative floating point value for sharpening image; optional

Returns the bytes for the rescaled image, with a ContentType header of image/jpeg.

If neither percent nor width is specified the image will be returned at the original scale. If both percent and width are specified, the width value will be ignored. If quality is not specified, the default value of 85 will be used. Values less than 0.5 for sharpen will blur the image (lower values blur more); values greater than 0.5 will sharpen edges.

Ready for testing. Please let me know if you run into any problems.

Comment entered 2011-10-24 09:54:22 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2011-10-24 09:54:22
BZCOMMENTOR::Bob Kline
BZCOMMENT::2

[pasted in from offline email exchange with Kevin]

Should "image" be sent as a file, i.e. using "multipart/form-data" encoding type, or as a variable/string? If it's a string, how should the file be encoded, Base 64 maybe?

I just tried both ways and it wasn't happy with either:

A problem occurred in a Python script.

d:\cdr\Log\tmpmidhs9.html contains the description of this error. D:\Python\lib\cgitb.py:173: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6 value = pydoc.html.repr(getattr(evalue, name))

[response:]

Url encoding works for me. Here's a script I just used to confirm that it works with that encoding:

#!/usr/bin/python

import urllib2, urllib

bytes = open('wiws_types_of_cancer_pic.jpg', 'rb').read()
url = 'http://bach.nci.nih.gov/cgi-bin/cdr/ResizeImage.py'
request = urllib2.Request(url)
request.add_header('User-Agent', 'Image Resize Tester')
request.add_data(urllib.urlencode({'image': bytes, 'percent': '50'}))
response = urllib2.urlopen(request)
fp = open('50percent.jpg', 'wb')
fp.write(response.read())
fp.close()

Bob

Comment entered 2011-10-24 10:43:48 by Broun, Kevin (NIH/NCI) [E] [X]

BZDATETIME::2011-10-24 10:43:48
BZCOMMENTOR::Kevin Broun
BZCOMMENT::3

Making some progress, but my script is confused by the response from Bach - looks like the mime type is text/html rather image/jpeg. Can you force the response to be image/jpeg unless there's an error?

Comment entered 2011-10-24 11:26:16 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2011-10-24 11:26:16
BZCOMMENTOR::Bob Kline
BZCOMMENT::4

(In reply to comment #3)
> Making some progress, but my script is confused by the response from Bach -
> looks like the mime type is text/html rather image/jpeg. Can you force the
> response to be image/jpeg unless there's an error?

It already does that (as noted in the docs). Here's a modified test script, which shows everything returned by the server:

!/usr/bin/python

import httplib, urllib

bytes = open('test.jpg', 'rb').read()
params = urllib.urlencode({'image': bytes, 'percent': '50'})
headers = {"Content-type": "application/x-www-form-urlencoded"}
conn = httplib.HTTPConnection("bach.nci.nih.gov")
conn.request("POST", "/cgi-bin/cdr/ResizeImage.py", params,headers)
response = conn.getresponse()
print response.status, response.reason
print repr(response.getheaders())
print repr(response.read())

The output is attached to this comment. The interesting part is the second line, which shows the list of headers returned by the server (each header is represented by a (name, value) tuple), including the content-type header, which is "image/jpeg."

What tools/language are you using for your client? Perhaps it would help if I took a look at your client code.

Comment entered 2011-10-24 11:26:16 by Kline, Bob (NIH/NCI) [C]

Attachment test.out has been added with description: Output from the server, captured by debugging version of test client

Comment entered 2011-10-24 14:50:57 by Broun, Kevin (NIH/NCI) [E] [X]

BZDATETIME::2011-10-24 14:50:57
BZCOMMENTOR::Kevin Broun
BZCOMMENT::5

Form on my dev server is here: http://visuals-dev.nci.nih.gov/admin/resizeimage.html
Try /web/visuals/images/2330-150.jpg as an image file path and 50 as the scale (all other params not used).

The action page ColdFusion code:

------

<cftry>
<cfif isdefined("imagefilepath") and trim(imagefilepath) neq "">
<cffile action="read" file="#imagefilepath#" variable="imagefilecontents">
<cfelse>
<cfthrow message="File required" detail="Specify an image file path.">
</cfif>

<cfhttp url="http://bach.nci.nih.gov/cgi-bin/cdr/ResizeImage.py" method="post" resolveurl="no" throwonerror="yes" getasbinary="yes">
<cfhttpparam type="formfield" name="image" value="#URLEncodedFormat(imagefilecontents)#">
<cfif isdefined("percent") and isnumeric(percent)>
<cfhttpparam type="formfield" name="percent" value="#percent#">
</cfif>
<cfif isdefined("width") and isnumeric(width)>
<cfhttpparam type="formfield" name="width" value="#width#">
</cfif>
<cfif isdefined("quality") and isnumeric(quality)>
<cfhttpparam type="formfield" name="quality" value="#quality#">
</cfif>
<cfif isdefined("sharpen") and isnumeric(sharpen)>
<cfhttpparam type="formfield" name="sharpen" value="#sharpen#">
</cfif>
</cfhttp>

<cfif FileExists("/web/visuals/wwwroot/temp/test.jpg")>
<cffile action="delete" file="/web/visuals/wwwroot/temp/test.jpg">
</cfif>
<cffile action="write" addnewline="no" file="/web/visuals/wwwroot/temp/test.jpg" output="#CFHTTP.FileContent#" mode="644">

<html>
<head>
<title>Resize Image Action</title>
</head>
<body>
<h3>Resize Image Action</h3>
<p>Resize completed. See <a href="/temp/test.jpg">/temp/test.jpg</a>.</p>
</body>
</html>

<!— <cfthrow message="testing - cfhttp response length: #Len(CFHTTP.FileContent)#"> --->
<!— <cfif CFHTTP.MIMEType neq "image/jpeg">
<cfthrow message="Wrong MIME type" detail="Resizing server returned #CFHTTP.MIMEType# instead of image/jpeg.">
</cfif>

<cfsetting enablecfoutputonly="no"><cfheader name="Content-Disposition" value="inline; filename=nci-vol-test.jpg">
<cfcontent type="image/jpeg" variable="#CFHTTP.FileContent#">
--->
<cfcatch>
<cfoutput>
<html>
<head>
<title>Error</title>
</head>
<body>
<h3>Error</h3>
<h4>#CFCATCH.Message#</h4>
<p>Detail: #CFCATCH.Detail#</p>
<cfif isdefined("resizeoutput")>
<p>Resizeoutput:<br />#resizeoutput#</p>
</cfif>
<cfif isdefined("CFHTTP.FileContent")>
<p>CFHTTP.FileContent:<br />#CFHTTP.FileContent#</p>
</cfif>
</body>
</html>
</cfoutput>
</cfcatch>
</cftry>

-------

The output file ended up with these contents:

<body bgcolor="#f0f0f8"><font color="#f0f0f8" size="5"> ->
<body bgcolor="#f0f0f8"><font color="#f0f0f8" size="5"> -> -->
</font> </font> </font> </script> </object> </blockquote> </pre>
</table> </table> </table> </table> </table> </font> </font> </font><p>A problem occurred in a Python script.
<p> d:\cdr\Log\tmp59m2zw.html contains the description of this error.
D:\Python\lib\cgitb.py:173: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6

value = pydoc.html.repr(getattr(evalue, name))

Comment entered 2011-10-24 15:40:01 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2011-10-24 15:40:01
BZCOMMENTOR::Bob Kline
BZCOMMENT::6

(In reply to comment #5)
> Form on my dev server is here:

Thanks, this helps a lot. Looks like we've got two problems.

> ....
> <cfhttpparam type="formfield" name="image"
> value="#URLEncodedFormat(imagefilecontents)#">

The first is that your url-encoding of the image bytes is redundant, as Cold Fusion is doing it for you, so what I get is double-encoded.

The second is that the image, after rolling back the double encoding, is still not recognized as a JPEG image. The JPEG Interchange Format [1] expects the first two bytes to be FF and D8, but the first two bytes in what you're sending (after decoding) are EF and BF.

[1] http://en.wikipedia.org/wiki/JPEG

Comment entered 2011-10-24 15:46:25 by Broun, Kevin (NIH/NCI) [E] [X]

BZDATETIME::2011-10-24 15:46:25
BZCOMMENTOR::Kevin Broun
BZCOMMENT::7

OK, I removed the URLEncodedFormat function, but don't actually have any other control of what CF sends in that variable. The updated error returned in the "test.jpg" file is now:

Invalid image: IOError('cannot identify image file',)

which makes sense with what you are reporting. Any chance you can test for those first couple of bytes and toss them out if found?

Comment entered 2011-10-24 15:57:13 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2011-10-24 15:57:13
BZCOMMENTOR::Bob Kline
BZCOMMENT::8

(In reply to comment #7)
> OK, I removed the URLEncodedFormat function, ...

Yes, I can see that this problem is gone now.

> ... but don't actually have any other control of what CF sends in that
> variable. The updated error returned in the "test.jpg" file is now:
>
> Invalid image: IOError('cannot identify image file',)
>
> which makes sense with what you are reporting. Any chance you can test for
> those first couple of bytes and toss them out if found?

Don't think that's going to help. There isn't an FF byte in the entire string I get for the image bytes, and unless the image conforms to the JPEG format, the library I'm using to manipulate the file doesn't know what to do with it, nor does any of the image manipulation or display programs I've tried to open it with. If you want to see the bytes that are coming into the service from your test client, you can pull down http://bach.nci.nih.gov/kb-test.jpg .

Comment entered 2011-10-24 16:17:25 by Broun, Kevin (NIH/NCI) [E] [X]

BZDATETIME::2011-10-24 16:17:25
BZCOMMENTOR::Kevin Broun
BZCOMMENT::9

OK, cffile has both "read" and "readbinary" variations - I think using "read" tends to munge up a binary file like an image. But if I use "readbinary" I get an error when trying to send the results as a formfield in the cfhttp step - that requires a string, e.g. Base64 encoding. Can I give you the image file Base64'd?

Comment entered 2011-10-24 16:40:36 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2011-10-24 16:40:36
BZCOMMENTOR::Bob Kline
BZCOMMENT::10

(In reply to comment #9)
> OK, cffile has both "read" and "readbinary" variations - I think using "read"
> tends to munge up a binary file like an image. But if I use "readbinary" I get
> an error when trying to send the results as a formfield in the cfhttp step -
> that requires a string, e.g. Base64 encoding. Can I give you the image file
> Base64'd?

Sure, let's give it a try.

[You can leave the status of the issue as "Assigned"; no need to keep setting it back to "New."]

Comment entered 2011-10-24 17:23:17 by Broun, Kevin (NIH/NCI) [E] [X]

BZDATETIME::2011-10-24 17:23:17
BZCOMMENTOR::Kevin Broun
BZCOMMENT::11

Success! Base64 is working from my end very well, thank you. Sharpen parameter testing also successful.

Marking as resolved/fixed... FYI I never intentionally reset this to "new" - that was some form-caching oddity on my browser, I think.

Comment entered 2011-10-24 17:56:11 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2011-10-24 17:56:11
BZCOMMENTOR::Bob Kline
BZCOMMENT::12

Excellent! BTW, I left the original approach intact. If you come in with a client that hands me the image bytes directly (which is what I get with the url encoding, since the web server decodes the parameters before I get them) I proceed as before. I've wrapped that code in an exception handler, and if an exception is caught I fall back on an attempt to base64-decode before loading the image into the image processing module. So both types of client will succeed. Only if neither attempt to load the image is successful do I return an error string (with MIME type of text/plain) showing the information the exception gives me.

Comment entered 2011-10-26 12:16:05 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2011-10-26 12:16:05
BZCOMMENTOR::Bob Kline
BZCOMMENT::13

You can close this issue after you are satisfied that sufficient successful testing has been done.

Comment entered 2011-11-07 16:15:31 by Broun, Kevin (NIH/NCI) [E] [X]

BZDATETIME::2011-11-07 16:15:31
BZCOMMENTOR::Kevin Broun
BZCOMMENT::14

I'm noticing some color shifts in certain situations, particularly when viewing scaled images on color-profile-aware browsers, e.g. Firefox and Safari on the Mac. Is it possible that the ResizeImage.py script is either changing or stripping off color profiles attached to images?

Example 1:
Unscaled - http://visuals-dev.nci.nih.gov/retrieve.cfm?imageid=8804&dpi=300&fileformat=jpg
Scaled 50% - http://visuals-dev.nci.nih.gov/retrieve.cfm?imageid=8804&dpi=150&fileformat=jpg

Example 2:
Unscaled - http://visuals-dev.nci.nih.gov/retrieve.cfm?imageid=8753&dpi=300&fileformat=jpg
Scaled 50% - http://visuals-dev.nci.nih.gov/retrieve.cfm?imageid=8753&dpi=150&fileformat=jpg

Insights appreciated - thanks!

Comment entered 2011-11-07 16:38:42 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2011-11-07 16:38:42
BZCOMMENTOR::Bob Kline
BZCOMMENT::15

I'm afraid I'm out of my depth here, knowing nothing about color profiles in JPEGs, so I calling in the big guns, asking Alan to chime in with what he knows, since he's far more knowledgeable than I am about digital image file technology.

Alan:

Please see the previous comment posted by Kevin.

Comment entered 2011-11-08 10:55:35 by alan

BZDATETIME::2011-11-08 10:55:35
BZCOMMENTOR::Alan Meyer
BZCOMMENT::16

(In reply to comment #15)
> I'm afraid I'm out of my depth here, knowing nothing about color profiles in
> JPEGs, so I calling in the big guns, asking Alan to chime in with what he
> knows, since he's far more knowledgeable than I am about digital image file
> technology.
>
> Alan:
>
> Please see the previous comment posted by Kevin.

Although I am far from being an expert on this topic I will attempt to ramp up my expertise and figure it out.

I'll be looking at this today.

Comment entered 2011-11-08 15:29:01 by alan

BZDATETIME::2011-11-08 15:29:01
BZCOMMENTOR::Alan Meyer
BZCOMMENT::17

I saved the two files from the browser and looked at the 'info'
field in each. It appears that the scaled image has lost the
following information:

icc_profile (the one Kevin thought was lost)
exif
dpi

Now, the good news is that I can take the color profile from the
big image and stuff it into the small image. I think I only want
to do that if both have the same file type (jfif) and mode (RGB).

But there's bad news too. To my eye, saving the color profile
made no difference whatsoever in the colors.

But then the color differences between the full size image and
the scaled image look extremely subtle to me, more so on my machine
than they did on Kevin's.

I'm going to attach three images to this issue so others can see
what I'm seeing - maybe. Or maybe not. Maybe what I'm seeing is
dependent on my software, my hardware, and my own visual acuity -
all of which might well be different on another machine.

Comment entered 2011-11-08 15:43:06 by alan

BZDATETIME::2011-11-08 15:43:06
BZCOMMENTOR::Alan Meyer
BZCOMMENT::18

I saved the two files from the browser and looked at the 'info'
field in each. It appears that the scaled image has lost the
following information:

icc_profile (the one Kevin thought was lost)
exif
dpi

Now, the good news is that I can take the color profile from the
big image and stuff it into the small image. I think I only want
to do that if both have the same file type (jfif) and mode (RGB).

But there's bad news too. To my eye, saving the color profile
made no difference whatsoever in the colors.

But then the color differences between the full size image and
the scaled image look extremely subtle to me, more so on my machine
than they did on Kevin's.

I'm going to attach three images to this issue so others can see
what I'm seeing - maybe. Or maybe not. Maybe what I'm seeing is
dependent on my software, my hardware, and my own visual acuity -
all of which might well be different on another machine.

Comment entered 2011-11-08 15:44:28 by alan

BZDATETIME::2011-11-08 15:44:28
BZCOMMENTOR::Alan Meyer
BZCOMMENT::19

This is the original image before scaling.

Comment entered 2011-11-08 15:44:28 by alan

Attachment nci-vol-8753-300.jpg has been added with description: Full size image

Comment entered 2011-11-08 15:46:24 by alan

BZDATETIME::2011-11-08 15:46:24
BZCOMMENTOR::Alan Meyer
BZCOMMENT::20

This is the one Bob's program is returning, as supplied by Kevin's program.

Comment entered 2011-11-08 15:46:24 by alan

Attachment nci-vol-8753-150.jpg has been added with description: Image scaled by Bob's program and Python Image Library

Comment entered 2011-11-08 15:51:31 by alan

BZDATETIME::2011-11-08 15:51:31
BZCOMMENTOR::Alan Meyer
BZCOMMENT::21

For this one, I started Python, imported the Image library, created an image in memory by loading the original full size image, then created another image in memory by loading the scaled image as found in the previous attachment.

I then saved the scaled image using the same properties that Bob's program uses - JPEG, quality=85, but I added the icc_profile copied from the big image.

As a check to be sure I did it right, I read in this saved image and compared the image's icc_profile to the icc_profile property of the big image. They were identical.

Comment entered 2011-11-08 15:51:31 by alan

Attachment small2.jpg has been added with description: Scaled image with large image's icc_profile

Comment entered 2011-11-08 16:03:28 by alan

BZDATETIME::2011-11-08 16:03:28
BZCOMMENTOR::Alan Meyer
BZCOMMENT::22

I'll stop work on this for now and wait to see what people think should be done. It may be that we want to accept that when the image is re-sized, at least by the library we have, subtle changes occur. If that's not acceptable, it's possible we'll need to rewrite the program using a different image library, for example by calling an external program like Irfanview.

However, before we make any decision to switch to different image software someone with sharper eyes than mine will need to test lots of images with both the Python Image Library and the alternative one. If we don't we might find that one of the images we've converted with PIL looks better using the new software, but something else looks worse.

It may also be necessary to test with different browsers on different computers since, again, we might wind up switching to a conversion program that looks better on one platform but worse on something else. I suspect that writing the program with any desired library will be easier than figuring out what library is best.

Comment entered 2011-11-08 16:09:07 by alan

BZDATETIME::2011-11-08 16:09:07
BZCOMMENTOR::Alan Meyer
BZCOMMENT::23

I see that I managed to double click on comment 17 and 18, or do something to cause it to store twice.

I'd like to think it was a computer error, not mine.

Comment entered 2011-11-08 22:10:17 by alan

BZDATETIME::2011-11-08 22:10:17
BZCOMMENTOR::Alan Meyer
BZCOMMENT::24

(In reply to comment #18)

> But there's bad news too. To my eye, saving the color profile
> made no difference whatsoever in the colors.

I take it back. The news is good. I had a nagging doubt about
the validity of my test. I had looked at the images in Irfanview
and could not tell any difference between the two scaled images,
one with and one without the color profile. However Kevin had
told me that not all programs respect color profiles. I just
assumed that Irfanview, that paragon of image viewing programs,
did the right thing. But what if it didn't?

So I just looked again using Safari - which Kevin said treats
color profiles properly. To my eye, in Safari, the scaled image
with a color profile correctly matches the large image with a
color profile and the other scaled image looks different. So the
experiment of copying in the color profile was a success.

I copied Bob's existing program to "ResizeImageNoICC.py", then
modified "ResizeImage.py" as follows:

If the image to resize has mode='RGB' and format='JPEG':

Save the resized image with the icc_profile value taken
from the original image.

Else:

Discard the profile and save the resized image without a
profile (as was done in the original program.)

Everything else is unchanged.

My understanding of color profiles is that most of them are
referenced to the format (I think) and color mode (for sure.) To
use them with another format and mode requires a conversion,
which might or might not be possible with our image library, but
which I don't plan to research unless there is a need for that.

[From my reading it appears that conversion involves converting
the profile from a mode relative one to an absolute, mode
independent one, then converting that to the profile specific to
the target mode. That way each color profile can be designed for
the needs of specific modes, formats and applications. But by
using an absolute intermediary, only one conversion program is
required for it, and one for the target of the conversion.]

I read the code and ran it through a syntax checker but I don't
have a handy program to test it. Since Kevin has one, I'll let
him test it. If it's broken, let me know and I'll fix it.

I can perhaps take credit for the mechanics of this change to the
program, but credit for the intellectual work goes to Kevin -
whose idea it was to just copy the color profile from the
original to the scaled image, and of course to Bob for the basic
program.

Comment entered 2011-11-09 08:26:08 by Broun, Kevin (NIH/NCI) [E] [X]

BZDATETIME::2011-11-09 08:26:08
BZCOMMENTOR::Kevin Broun
BZCOMMENT::25

Yes, the files with the copied/retained profile do have the same apparent color has the original - yeah!

So now I'm testing the updated script, but not getting a JPEG back from the script.

Try the tester here: http://visuals-dev.nci.nih.gov/admin/resizeimageaction.cfm
You can use /web/visuals/images/8753-300.jpg as the image file path.

Comment entered 2011-11-09 09:12:17 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2011-11-09 09:12:17
BZCOMMENTOR::Bob Kline
BZCOMMENT::26

(In reply to comment #25)
> Yes, the files with the copied/retained profile do have the same apparent color
> has the original - yeah!
>
> So now I'm testing the updated script, but not getting a JPEG back from the
> script.
>
> Try the tester here: http://visuals-dev.nci.nih.gov/admin/resizeimageaction.cfm
> You can use /web/visuals/images/8753-300.jpg as the image file path.

(You'd need to start with http://visuals-dev.nci.nih.gov/admin/resizeimage.html, which puts up the form in which you can enter the parameters.)

Found and fixed a couple of bugs in the new script. Please give it another shot.

Comment entered 2011-11-09 11:01:01 by Broun, Kevin (NIH/NCI) [E] [X]

BZDATETIME::2011-11-09 11:01:01
BZCOMMENTOR::Kevin Broun
BZCOMMENT::27

Sorry about the bad link in my last comment.

Anyway, yes, the script is now working. But I'm back to seeing the same color shifts (or dropping of the profile) as before.

Compare:
http://visuals-dev.nci.nih.gov/retrieve.cfm?imageid=8753&dpi=300&fileformat=jpg
http://visuals-dev.nci.nih.gov/retrieve.cfm?imageid=8753&dpi=150&fileformat=jpg

Comment entered 2011-11-09 11:35:59 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2011-11-09 11:35:59
BZCOMMENTOR::Bob Kline
BZCOMMENT::28

Alan:

One of the bugs I fixed was the following line:

copyICC = image.info['icc_profile']

which triggered an exception in the test case Kevin gave us, because "icc_profile" wasn't present as a key value in the image.info dictionary. I changed the line to:

copyICC = image.info.get("icc_profile")

which avoids the exception, but means that copyICC now has the value None if the key "icc_profile" isn't present in the image's info dictionary. Hence the reversion to the original behavior (the profile isn't preserved). I better avoid any further mucking around in the script until you've had a chance to take a look.

Is it possible that we're dealing with the same problem reported at http://www.imagemagick.org/discourse-server/viewtopic.php?f=3&t=17900 ?

Comment entered 2011-11-09 12:05:00 by alan

BZDATETIME::2011-11-09 12:05:00
BZCOMMENTOR::Alan Meyer
BZCOMMENT::29

(In reply to comment #28)

> Is it possible that we're dealing with the same problem reported at
> http://www.imagemagick.org/discourse-server/viewtopic.php?f=3&t=17900 ?

Could be. It looks like different cameras might put the ICC profile in different places. If the Python Image Library gives us full access to the ImageMagick API, then we probably want to do something along the lines of:

If there is a color profile in the image.info['icc_profile'] field:
Get it.
Else if there is a color profile in the EXIF data:
Get it
Else
Proceed with no profile.

This could turn into a trial and error development process since different camera manufacturers and models might do different things within the EXIF.

We're learning more about digital images than we expected to learn.

Comment entered 2011-11-14 11:53:25 by Broun, Kevin (NIH/NCI) [E] [X]

BZDATETIME::2011-11-14 11:53:25
BZCOMMENTOR::Kevin Broun
BZCOMMENT::30

So at this point we are not retaining the color profile, correct? That's the behavior I'm seeing anyway.

Comment entered 2011-11-15 17:26:01 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2011-11-15 17:26:01
BZCOMMENTOR::Bob Kline
BZCOMMENT::31

Alan's going to follow up with Kevin.

Comment entered 2011-11-15 17:31:27 by alan

BZDATETIME::2011-11-15 17:31:27
BZCOMMENTOR::Alan Meyer
BZCOMMENT::32

(In reply to comment #31)
> Alan's going to follow up with Kevin.

Kevin,

If you're in on Thursday, let's meet to talk about it. We need to know first if it's not working for all images or just not working for some images. It may be that some images don't have color profiles and/or some have them but they're not in the same place in the data where found them in the MRI machine image. Or it may be that what I did worked to save images but wasn't successfully returning the color profile in the data that went back out over the net (less likely I think.)

I'm changing the status of the issue to re-opened.

Comment entered 2011-11-28 17:27:02 by alan

BZDATETIME::2011-11-28 17:27:02
BZCOMMENTOR::Alan Meyer
BZCOMMENT::33

I would like to so some experiments with this this evening using Kevin's program to access Bob's. The ResizeImage program will be broken while I'm doing it. I'll hold off until 6 pm before starting. Someone please let me know if there's a problem with that.

Comment entered 2011-11-28 21:38:39 by alan

BZDATETIME::2011-11-28 21:38:39
BZCOMMENTOR::Alan Meyer
BZCOMMENT::34

I have not solved the problem but I know more about it.

Using the image of the MRI machine, I found a color profile in
the EXIF data - which is not where I expected it from the last
time I looked at this.

There are two problems we have to solve:

1. Finding the color profile wherever it is:

Parsing EXIF is not for the faint of heart. It's a tag/value
binary format in which both tags and values are binary.
Furthermore, it looks like different camera manufacturers use
tags slightly differently, so we might have to know something
about the cameras that produce the images - not something we
want to get into or maintain.

There are stout hearted programmers out there who have done
it and I found at least four packages that can get the data
for us. They include "pexif" - a Python EXIF parser,
"libexif" a parser written in C, "exiftool" - a Perl based
parser that can be used as a Perl library or as a stand alone
command line tool, and the most recent versions of Irfanview
(I had an older one when I tested in the past) which has to
be used as a stand alone command line tool.

Of the four packages, the one that looks to me to be most
complete and mature is exiftool, the Perl package. To make
use of it we would either have to rewrite ResizeImage.py in
Perl, which might give us the most options, or spawn the Perl
program from Python and suck the results back in.

2. Putting it some place where it belongs in the output image.

I tried taking the entire exif data as I found it in the
input image and putting it in the same place in the output
image, i.e., in image.info['exif']. That was accepted by the
Python Image Library (PIL) that we're using, i.e., there were
no complaints from the software. However the actual output
file was unchanged as compared to the output file where I had
not done that.

So the quick and dirty techniques I've been using to try to
make this work don't work. It will be necessary to get back
to first principles and understand what is happening inside
the image library in Python, or in Perl if we switch to that,
and figure out how to output what we want as well as find it
in the input data.

If we can't solve this problem using the Python Image
Library, we may be committed to switching to a different
programming language.

I believe this no longer meets the criteria that Margaret
described at the top of this Bugzilla issue when she said:

"Bob has already done an estimate of time required and
determined that it would be fairly quick."

I'm going to stop work for now and go back to other tasks. I'll
wait for management to tell me whether to continue or not. I
think it will be at least a day's work, but we'll be in uncharted
territory. It could be a lot longer if we don't find a
straightforward way to use the PIL to output the data.

Comment entered 2011-11-28 21:40:48 by alan

BZDATETIME::2011-11-28 21:40:48
BZCOMMENTOR::Alan Meyer
BZCOMMENT::35

I've restored ResizeImage.py to the form that Bob left it in.

The debug version that I had is ResizeImage.py.Alan in the same directory.

Comment entered 2011-11-29 16:39:44 by Broun, Kevin (NIH/NCI) [E] [X]

BZDATETIME::2011-11-29 16:39:44
BZCOMMENTOR::Kevin Broun
BZCOMMENT::36

Can we go with an option something like this?

If there is a color profile in the image.info['icc_profile'] field:
Get it.
[ Else if there is a color profile in the EXIF data:
Get it ] - add this logic if/when resources allow
Else
Proceed with no profile.

Comment entered 2011-11-29 18:45:36 by alan

BZDATETIME::2011-11-29 18:45:36
BZCOMMENTOR::Alan Meyer
BZCOMMENT::37

(In reply to comment #36)
> Can we go with an option something like this?
>
> If there is a color profile in the image.info['icc_profile'] field:
> Get it.
> [ Else if there is a color profile in the EXIF data:
> Get it ] - add this logic if/when resources allow
> Else
> Proceed with no profile.

Yes, I believe that we have this now. It's only if it's in the EXIF data that we're not getting it.

Comment entered 2012-01-19 13:24:56 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2012-01-19 13:24:56
BZCOMMENTOR::Bob Kline
BZCOMMENT::38

I'm going to close this, after talking with Kevin.

Comment entered 2012-01-19 13:25:27 by Kline, Bob (NIH/NCI) [C]

BZDATETIME::2012-01-19 13:25:27
BZCOMMENTOR::Bob Kline
BZCOMMENT::39

Kevin said we could close this.

Attachments
File Name Posted User
nci-vol-8753-150.jpg 2011-11-08 15:46:24
nci-vol-8753-300.jpg 2011-11-08 15:44:28
small2.jpg 2011-11-08 15:51:31
test.out 2011-10-24 11:26:16

Elapsed: 0:00:00.000780