Other pixel-level meta data you could put in an image format

Image files are a grid of pixels - each pixel contains colour information⁰. But they don't just have to contain colour information. Here are some thoughts on other things that a future image format might contain.

What exists already?

A typical bitmap image looks like this under the hood:

 TEXT  0      1      2     3

0 Black  Red    Red   Blue

1 Red    White  Blue  Yellow

2 Orange Purple Green Brown

...

That is a grid of pixels, each with a colour value.

Modern image formats can also contain pixel-level detail about transparency. When an image is displayed above a different image, the computer calculates how much of the lower image's colour is mixed with the upper image's colour.

Some image formats also allow pixel-level detail about brightness¹. A device might be able to set the brightness on individual parts of the image.

So you end up with an image which looks like:

 TEXT  0                    1                             2     3

0 Black                Red - but 50% transparent     Red   Blue

1 Red - but really dim White - as bright as possible Blue  Yellow

...

Thermal / Infrared

Most image formats only deal with visible light. But there are a bunch of cameras which can capture infrared. Would it be useful to capture the heat of scene being photographed?

A future image format could contain details about visible and invisible light.

Invisible Wavelengths

Objects in the real worlds don't just give off thermal signatures - they often have information in the ultraviolet wavelengths as well.

For example, flowers often have "landing guides" which are only visible to bees and other insects.

Being able to capture beyond visible light may be a useful property for an image format.

Depth

Modern cameras often fire out an invisible infrared grid to allow for quick focussing and to aid with depth effects like bokeh. An image could store the depth map of its subject.

Here's a series of depth maps I made from analysing 3D movies:

This could allow for better 3D displays, or for easier image editing.

Angle

Images are typically 2D. The present pixels on a flat rectangular plane. But they aren't limited to that. Image panoramas and "spherical" images allow for display on a non-flat surface.

Here's an equirectangular 2D image which is rendered as a sphere:

An image could contain pixel-level data showing the angle away from the viewer. That would allow for a more accurate rendering in VR or other non-flat displays.

Object Detection

Now we start to get funky! AI is getting pretty good at detecting specific objects in photographs. What happens if we start marking them up within the image itself?

Either something as basic as this:

Or as detailed as this:

That would make it possible to easily remove or replace objects from images.

Combined with depth-sensing, it could be a powerful way to edit images.

Extra Texture

Colour replication depends heavily on the material which is imbued with that colour. That's what Pantone is all about². Suppose an object has a sky-blue colour - is it useful to know that it is a heavy woollen jumper rather than a thin nylon t-shirt?

Wheels within wheels and layers within layers

It's probably sensible to implement something like this as multiple layers, rather than saying every pixel must be RGBA+Depth+Heat+Angle+Object+etc+etc.

Different layers could have different resolution (Infrared is typically lower res than the image it is overlayed on). Different layers might be compressed more efficiently by different algorithms.

We're used to layered images in formats like PSD and XCF - so why not in a future version of JPEG or AVIF?

What else?

What other things might be useful for an image to store at an individual pixel-level? Drop a note in the comments?