Cannot read YCbCr JPEG-compressed images

[Craig Bruce]

The TIFF library runs into an internal problem when I try to use it to read
YCbCr JPEG-compressed images. I'm currently using version 4.0.0beta6,
but 3.9 and 3.7 had the same problem. Calling TIFFReadTile() inside my
program for the first tile produces the following error:

JPEGDecodeRaw: application buffer not large enough for all data.

I have put a sample image (it's 41 MB) at:

http://csbruce.com/temp/q09050_006_nir.tif

$ tiffinfo q09050_006_nir.tif
TIFF Directory at offset 0x2705581
Image Width: 11500 Image Length: 7500
Tile Width: 256 Tile Length: 256
Resolution: 0, 0 (unitless)
Bits/Sample: 8
Compression Scheme: JPEG
Photometric Interpretation: YCbCr
YCbCr Subsampling: 2, 2
Samples/Pixel: 3
Planar Configuration: single image plane
[...]

I get the same problem with TIFFs that aren't tiled or multi-image.
Any help is appreciated.

--------------------------+----------------------+--------------------------
Dr. Craig S. Bruce | Ph 819-771-8303 x205 | CubeWerx Inc.
Senior Software Developer | Fax [phone] | Gatineau, Québec, Canada
***@cubewerx.com | http://csbruce.com/ | http://www.cubewerx.com/
--------------------------+----------------------+--------------------------
"Oh, please. You can't expect politicians to have any comprehension
of the technical issues they make world-changing decisions about."
_______________________________________________
Tiff mailing list: ***@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/ti

[j***@gmail.com]

Hi Craig,

Post by Craig Bruce
The TIFF library runs into an internal problem when I try to use it to read
YCbCr JPEG-compressed images.  I'm currently using version 4.0.0beta6,
but 3.9 and 3.7 had the same problem.  Calling TIFFReadTile() inside my
JPEGDecodeRaw: application buffer not large enough for all data.

Are you reading the image a scanline at a time? You can't read YCbCr
images this way, since they have sub-sampled chrominance and therefore
don't divide neatly into lines. You need to read a strip or a tile at
a time, see TIFFReadEncodedStrip() and TIFFReadTile().

Post by Craig Bruce
http://csbruce.com/temp/q09050_006_nir.tif

I was able to read your image without problems here.

John
_______________________________________________
Tiff mailing list: ***@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/

[Charles Auer]

Using TIFFReadEncodedTile, I was able to decode all 7 images in the sample file without any errors.

Date: Thu, 20 Jan 2011 19:51:33 -0500
Subject: [Tiff] Cannot read YCbCr JPEG-compressed images
The TIFF library runs into an internal problem when I try to use it to read
YCbCr JPEG-compressed images. I'm currently using version 4.0.0beta6,
but 3.9 and 3.7 had the same problem. Calling TIFFReadTile() inside my
JPEGDecodeRaw: application buffer not large enough for all data.
http://csbruce.com/temp/q09050_006_nir.tif
$ tiffinfo q09050_006_nir.tif
TIFF Directory at offset 0x2705581
Image Width: 11500 Image Length: 7500
Tile Width: 256 Tile Length: 256
Resolution: 0, 0 (unitless)
Bits/Sample: 8
Compression Scheme: JPEG
Photometric Interpretation: YCbCr
YCbCr Subsampling: 2, 2
Samples/Pixel: 3
Planar Configuration: single image plane
[...]
I get the same problem with TIFFs that aren't tiled or multi-image.
Any help is appreciated.
--------------------------+----------------------+--------------------------
Dr. Craig S. Bruce | Ph 819-771-8303 x205 | CubeWerx Inc.
Senior Software Developer | Fax [phone] | Gatineau, Québec, Canada
--------------------------+----------------------+--------------------------
"Oh, please. You can't expect politicians to have any comprehension
of the technical issues they make world-changing decisions about."
_______________________________________________
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/

[Bob Friesenhahn]

Using TIFFReadEncodedTile, I was able to decode all 7 images in the sample file without any errors.

Likewise, GraphicsMagick was able to read this file (via libtiff)
without any problems.

Bob

[Craig Bruce]

Post by Charles Auer
Using TIFFReadEncodedTile, I was able to decode all 7 images in the
sample file without any errors.

I put together the following test program:

#include <stdio.h>
#include <stdlib.h>
#include <tiffio.h>

main()
{
TIFF *tiff;
ttile_t tileNum;
tsize_t tileSize;
int readSize;
tdata_t buf;

tiff = TIFFOpen( "Q09050_006_NIR.tif", "r" );
if (tiff == NULL) goto ERROR;
tileNum = TIFFComputeTile( tiff, 0, 0, 0, 0 );
tileSize = TIFFTileSize( tiff );
buf = malloc( tileSize );
printf( "tileNum=%ld, tileSize=%ld, buf=%p\n", (long) tileNum,
(long) tileSize, (long) buf );
readSize = TIFFReadEncodedTile( tiff, tileNum, buf, tileSize );
if (readSize == -1) goto ERROR;
printf("successful read, size=%d\n", readSize );
exit( 0 );

ERROR:
printf( "ERROR!\n" );
exit( 1 );
}

If I build it using the system libraries on my computer (Linux x86_64,
Fedora 13, libtiff 3.9.4), it works:

$ gcc tile.c -ltiff -ljpeg -lz -lm -o tile_test
$ ./tile_test
TIFFReadDirectory: Warning, Q09050_006_NIR.tif: unknown field with tag 50742 (0xc636) encountered.
tileNum=0, tileSize=98304, buf=0x2124a60
successful read, size=98304

However, if I build and link with my own libtiff 4.0.0beta6 with the
default configuration, I get the buffer error:

$ gcc -I ~/tiff-4.0.0beta6/libtiff tile.c ~/tiff-4.0.0beta6/libtiff/.libs/libtiff.a -ljpeg -lz -lm -o tile_test
$ ./tile_test
TIFFReadDirectory: Warning, Unknown field with tag 50742 (0xc636) encountered.
TIFFFetchNormalTag: Warning, ASCII value for tag "ImageDescription" does not end in null byte.
TIFFFetchNormalTag: Warning, ASCII value for tag "Software" does not end in null byte.
tileNum=0, tileSize=98304, buf=0x22092b0
JPEGDecodeRaw: application buffer not large enough for all data..
ERROR!

The behaviour is consistent across a couple different versions of Fedora-
built and personally-build libtiffs. Is there some special libtiff
configuration needed to make YCbCr/JPEG reading work properly? I need
to use a custom-built libtiff.

--------------------------+----------------------+--------------------------
Dr. Craig S. Bruce | Ph 819-771-8303 x205 | CubeWerx Inc.
Senior Software Developer | Fax [phone] | Gatineau, Québec, Canada
***@cubewerx.com | http://csbruce.com/ | http://www.cubewerx.com/
--------------------------+----------------------+--------------------------
The Free Market is why Canadian flags are made in China instead of Singapore.
_______________________________________________
Tiff mailing list: ***@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff

[Olivier Paquet]

Post by Craig Bruce
If I build it using the system libraries on my computer (Linux x86_64,
$ gcc tile.c -ltiff -ljpeg -lz -lm -o tile_test
$ ./tile_test
TIFFReadDirectory: Warning, Q09050_006_NIR.tif: unknown field with tag 50742 (0xc636) encountered.
tileNum=0, tileSize=98304, buf=0x2124a60
successful read, size=98304
However, if I build and link with my own libtiff 4.0.0beta6 with the
$ gcc -I ~/tiff-4.0.0beta6/libtiff tile.c ~/tiff-4.0.0beta6/libtiff/.libs/libtiff.a -ljpeg -lz -lm -o tile_test
$ ./tile_test
TIFFReadDirectory: Warning, Unknown field with tag 50742 (0xc636) encountered.
TIFFFetchNormalTag: Warning, ASCII value for tag "ImageDescription" does not end in null byte.
TIFFFetchNormalTag: Warning, ASCII value for tag "Software" does not end in null byte.
tileNum=0, tileSize=98304, buf=0x22092b0
JPEGDecodeRaw: application buffer not large enough for all data..
ERROR!

I tried both my system libtiff (3.9.4 too but on slackware) and the
CVS head of 4.0 and got a segfault in jpeg_read_raw_data. This seems
to be called from libtiff just after the code which generates the
error you got above so it's probably related.

Post by Craig Bruce
The behaviour is consistent across a couple different versions of Fedora-
built and personally-build libtiffs.  Is there some special libtiff
configuration needed to make YCbCr/JPEG reading work properly?  I need
to use a custom-built libtiff.

I suspect the different results we're seeing between people here might
have something to do with the libjpeg involved. Mine is
libjpeg.so.8.0.1. Oh and for what it's worth, I get the segfault no
matter how large I make the buffer in your test program.

Olivier
_______________________________________________
Tiff mailing list: ***@lists.maptools.org
http://lists.maptools.org/mailman/listinfo/tiff
http://www.remotesensing.org/libtiff/

[Craig Bruce]

Post by Olivier Paquet
I suspect the different results we're seeing between people here might
have something to do with the libjpeg involved. Mine is
libjpeg.so.8.0.1. Oh and for what it's worth, I get the segfault no
matter how large I make the buffer in your test program.

Whatever the cause, I was able to get my application to work using the
TIFFReadRGBATile() and TIFFReadRGBAStrip() calls. I don't know why these
work when the other calls blow up, but they also save me the trouble of
decoding the YCbCr samples to RGB.

--------------------------+----------------------+--------------------------
Dr. Craig S. Bruce | Ph 819-771-8303 x205 | CubeWerx Inc.
Senior Software Developer | Fax [phone] | Gatineau, Québec, Canada
***@cubewerx.com | http://csbruce.com/ | http://www.cubewerx.com/
--------------------------+----------------------+--------------------------
"I don't suffer from stress. I'm a carrier."
_______________________________________________
Tiff mailing list: ***@lists.maptools.org
http://lists.maptools.org/mailman/lis

[Bob Friesenhahn]

Post by Craig Bruce
JPEGDecodeRaw: application buffer not large enough for all data..
ERROR!
The behaviour is consistent across a couple different versions of Fedora-
built and personally-build libtiffs. Is there some special libtiff
configuration needed to make YCbCr/JPEG reading work properly? I need
to use a custom-built libtiff.

The bug you are reporting has been reported several times before.
Please let us know if you find a correct fix.

The reason why our readers are working is because we tell libtiff to
use JPEGCOLORMODE_RGB like:

(void) TIFFSetField( tiff, TIFFTAG_JPEGCOLORMODE, JPEGCOLORMODE_RGB );

And then libjpeg helps by returning upsampled RGB data to libtiff
using reader code designed for PHOTOMETRIC_RGB.

If your software really was capable of dealing with all of the nuances
associated with YCbCr (filtering, subsampling, etc.) then you would
probably prefer to deal directly with it. Very little software in the
TIFF world is currently capable of doing that. Horizontal sampling is
not too bad to deal with, but things become more complex when vertical
sampling is added. JPEG samples in an 8x8 or 16x16 tile size.

Bob