tiff is evil, but not as evil as I thought.

@pavel the evil will grown on you, especially when random pieces of software will start rejecting your tiffs. Or even worse, make a piece of software that loads a tiff, modifies it and saves it again...

@pavel hdf5 is my poison of preference. Smaller filesize than tiff, opens faster and is more versatile. Compatibility in regular software is basically non-existent though

@papiris This is for camera images, and I wanted simple format...

@martijnbraam Yeah I guess compatibility will be a problem. I was surprised that writing tiffs is not hard. Reading tiffs I wrote should not be too bad, either, but obviously reading random tiffs is bad.

I wanted to store raw camera data to .pnm. That can work for bayer8 data, but starts to be problem for little-endian bayer10, and I guess tiff is really best option there.

@pavel main issue with tiff is that it's impossible to read a tiff file correctly if you don't understand all the tags in it since the format itself is not complete enough to be self-describing. This means it's not possible to edit TIFF files easily since it's one big mess of pointers.

Also I wrote libdng specifically to make this easier :)

@martijnbraam Ok, I may need some help. In dng, I see idf with preview parameters. How do I find out _real_ idf in dng? The one that contains raw image width, for example?

And is tiff or dng actually descriptive enough so that I can take common raw data (say grbg14) and just prepend tiff header before that? dng says something about data always being high-endian, and the purpose of this **** is me not wanting to do the byte swapping. Can it also store yuv data by chance?

And yes, it looks like tiff is as evil as I thought after all :-(.

@pavel a DNG is actually a tiff with nested IFDs where one is always a thumbnail.

Basically the structure is that the thumbnail is the "main" RGB image and it has a SubIFD (not a second IFD) with the type `0` to signify it's the "original" data. All the metadata that's not specific to only raw must be on the thumbnail IFD, like most exif data. The resolution of the raw data must be on the SubIFD as the width/length metadata. In combination with stripe offsets to fix padding it's possible to...

@pavel generate a "generic" tiff header you can paste on top of raw data to make it work for one specific sensor. But in most cases you'd also want to adjust the color-specific for every picture like sensor gains to make the images decode correctly.

It's not possible to store YUV raw data in DNG files sadly, that's why Megapixels has a jpeg exporter now...

@pavel technically TIFF is flexible enough that all your data can be _anywhere_ in the file. The only thing that needs to be at a fixed position is the TIFF magic header. It's technically possible to make a tiff where the main exif metadata block is halfway in the middle of your raw sensor dump and the raw sensor dump data comes before the thumbnail data, because everything is a pointer to arbitrary locations.

@martijnbraam Yep, understood so far. And got the bayer data width now, thanks. And yes, dng is more evil than plain tiffs.

I can't get "10-bits-in-two-bytes" to work with ImageMagick, and the endianness support is ... strange. You can't have LE headers and BE bayer data, for example. I wonder what else is common in hw formats and tiff can't do..

@pavel I think it might be necessary to unpack 10-bit bayer to 16-bits for DNG and then set the whitelevel metadata for the raw IFD to 2**10 to make the decoding for it correct.

@martijnbraam I could not get that whitelevel thing to work.

I guess better question to ask was: do you process pixel data before writing them to dng in megapixels? I had to do byteswapping for pnm write, I am wondering if dng/tiff can avoid that.

@pavel Yeah it's required for a lot of cases to actually shuffle the raw data around to make it fit into something that's valid in TIFF. None of the packed formats are possible in TIFF which is the most common format. I don't think I'm byteswapping the data anywhere.

@pavel Oh for 10-bit data I'm unpacking that to 16-bit and then setting TIFFTAG_BITSPERSAMPLE to 10 to make the level correct.

@pavel Aye, I used hdf5 for storing raw and debayered image arrays from video captures, with per-frame metadata and sensor measurements.
It's not too bad, but definitely not the first go-to image format, but if you'll be dealing with sensor fusion in post-processing or find tiffs take too much space, I'd recommend giving it a look

@papiris I need speed. I'm looking for something where I can just add header before raw data and write it to RAM disk... Then some other process will take it, debayer, and pass it to gstreamer for compression. Ideally at 30fps. I'd like something that can be viewed by common tools for debugging, but apparently that's hard.

@pavel I see. I reckon you've already had a look at what they do in [rpicam-raw](https://www.raspberrypi.com/documentation/computers/camera_software.html#rpicam-raw) ?

@papiris Thanks for pointer. I have not seen that really, but yes, I'm trying to develop something similar.