The original article is published on the website and published on 3DNews with the author's permission. We present the full text of the article, except for the numerous links — they will be useful for those who are seriously interested in the topic and would like to explore the theoretical aspects of computational photography more deeply, but for the general audience, we deemed this material excessive.
Today, no smartphone presentation avoids flaunting its camera. Every month we hear about the latest success in mobile cameras: Google teaches the Pixel to shoot in the dark, Huawei zooms like binoculars, Samsung adds LiDAR, and Apple makes the roundest corners in the world. Innovations are flowing abundantly in this area.
DSLRs, on the other hand, seem to be stuck in place. Sony showering everyone with new sensors every year, while manufacturers lazily update only the last digit of the version and continue to relax on the sidelines. I have a $3000 DSLR on my desk, but I take my iPhone on trips. Why?
As the classic said — I went online with this question. There, people discuss some 'algorithms' and 'neural networks,' having no idea how exactly they impact photography. Journalists loudly recite the number of megapixels, bloggers chorus out paid unboxings, and aesthetics indulge in the 'sensory perception of the color palette of the sensor.' All as usual.
I had to sit down, spend half my life, and figure everything out myself. In this article, I will share what I learned.
What is computational photography?
Everywhere, including Wikipedia, you find roughly this definition: computational photography — any techniques for capturing and processing images where digital computations are used instead of optical transformations. It is all well and good, except that it explains nothing. Autofocus fits this definition, but plenoptic optics, which has already brought us many benefits, does not fit. The vagueness of official definitions somehow suggests that we have no idea what we're talking about.
Pioneer of computational photography, Stanford professor Marc Levoy (who is currently in charge of the camera in Google Pixel) provides a different definition — a set of computer vision methods that enhance or extend the capabilities of digital photography, resulting in an ordinary photograph that could not have been technically captured by this camera in a traditional manner. In this article, I adhere to his view.
So, smartphones were to blame for everything.
Smartphones had no choice but to give birth to a new form of photography — computational.
Their tiny noisy sensors and small, low-light lenses should have, by all laws of physics, only brought pain and suffering. And they did, until their developers cleverly figured out how to harness their strengths to combat their weaknesses — fast electronic shutters, powerful processors, and software.
Most groundbreaking research in computational photography took place between 2005 and 2015, a mere yesterday in the world of science. Right now, before our eyes and in our pockets, a new field of knowledge and technology is developing that has never existed before.
Computational photography isn't just selfies with neural bokeh. The recent photograph of a black hole would not have come to light without computational photography methods. To capture such a photo with a regular telescope, we would have to make it the size of Earth. However, by combining data from eight radio telescopes located at different points on our globe and writing a few scripts in Python, we obtained the world's first photograph of the event horizon. It works for selfies too.
Beginning: Digital Processing
Let's imagine we've returned to 2007. Our mom is anarchy, and our photographs are noisy 0.6-Mp JPEGs taken on a skateboard. Around this time, we get our first irresistible urge to sprinkle them with presets to hide the ugliness of mobile sensors. Let’s not deny ourselves.
Math and Instagram
With the advent of Instagram, everyone became obsessed with filters. As someone who once reverse-engineered X-Pro II, Lo-Fi, and Valencia for, of course, research (heh) purposes, I still remember that they consisted of three components:
- Color settings (Hue, Saturation, Lightness, Contrast, Levels, etc.) — simple digital coefficients, just like in any presets that photographers have used since ancient times.
- Tone Mapping — vectors of values, each of which told us: "Red with a tint of 128 should be turned into a shade of 240."
- Overlays — semi-transparent images with dust, grain, vignette, and everything else that can be layered on top to achieve a decidedly non-banal effect of old film. It wasn’t always present.
Modern filters have not strayed far from this trio, though they've become a bit more complex mathematically. With the advent of hardware shaders and OpenCL on smartphones, they were quickly rewritten for GPUs, which was considered incredibly cool — for 2012, of course. Today, any school kid can create the same using CSS, and they won’t even get extra credit in graduation.
However, the progress of filters has not stopped today. Guys from Dehancer, for instance, are diving deep into nonlinear filters — instead of proleterian tone mapping, they use more complex nonlinear transformations, which, they say, open up far more possibilities.
Nonlinear transformations can do a lot, but they are incredibly complex, and we, humans, are incredibly limited. Once it comes to nonlinear transformations in science, we prefer to resort to numerical methods and stuff everything with neural networks to have them create masterpieces for us. The same was true here.
Automation and dreams of a 'masterpiece' button.
When everyone got used to filters, we began embedding them directly into cameras. History hides who was the first among manufacturers, but just to understand how long ago this was — by iOS 5.0, released back in 2011, there was already a public API for Auto Enhancing Images. Only Jobs knows how long it was used before being released to the public.
Automation did the same thing each of us does when opening a photo in an editor — it pulled up shadows and highlights, ramped up the saturation, removed red-eye, and fixed skin tones. Users didn’t even suspect that the 'dramatically improved camera' in the new smartphone was merely a result of a few new shaders. It would be another five years before the launch of Google Pixel and the hype around computational photography.
Today, the battle for the 'masterpiece' button has moved to the field of machine learning. After experimenting with tone mapping, everyone rushed to train CNNs and GANs to adjust sliders instead of users. In other words, using the input image to determine a set of optimal parameters that would bring this image closer to a subjective understanding of a 'good photograph.' This has been implemented in Pixelmator Pro and other editors. As you can guess, it doesn't always work very well.
Stacking — 90% of the success of mobile cameras
True computational photography began with stacking — layering several photographs on top of each other. For a smartphone, snapping a dozen shots in half a second is no problem. Their cameras lack slow mechanical parts: the aperture is fixed, and instead of a moving shutter, there’s an electronic one. The processor simply tells the sensor how many microseconds to capture wild photons, and reads the result.
Technically, the phone can take photos at video speed and video at photo resolution, but it all comes down to the speed of the bus and the processor. That's why software limits are always set.
Stacking has been around for a long time. Even our grandparents used plugins on Photoshop 7.0 to assemble several photos into eye-popping HDRs or to stitch together a panorama of 18000 × 600 pixels, but… no one has really figured out what to do with them afterward. Those were rich times, though wild.
Now we are adults and we call it 'epsilon photography' — when by changing one of the camera parameters (exposure, focus, position) and stitching together the resulting frames, we obtain something that could not be captured in a single shot. But that's a term for theorists; in practice, another name has taken hold — stacking. Today, in fact, 90% of all innovations in mobile cameras are based on it.
One thing that many don't think about, but which is important for understanding all mobile and computational photography: a camera in a modern smartphone starts taking photos as soon as you open its app. This makes sense, as it needs to display the image on the screen somehow. However, in addition to the display, it saves high-resolution frames in its own circular buffer, where it retains them for another couple of seconds.
When you press the 'take photo' button, the photo has actually already been taken; the camera simply pulls the last photo from the buffer.
This is how any mobile camera works today. At least in all flagship models, not from garbage. Buffering allows for not just zero shutter lag, which photographers have long dreamed of, but even negative lag—when you press the button, the smartphone looks into the past, retrieves the last 5-10 photos from the buffer, and starts furiously analyzing and stitching them together. There's no need to wait for the phone to snap pictures for HDR or night mode; just take them from the buffer, the user won't even know.
By the way, it is precisely through negative shutter lag that Live Photo is implemented on iPhones, and HTC had something similar back in 2013 under the strange name Zoe.
Exposure stacking — HDR and combating brightness fluctuations
Whether camera sensors can capture the full range of brightness available to our eyes is an old hot topic for debate. Some say no, as the eye can see up to 25 f-stops, whereas even the best full-frame sensor pulls in a maximum of 14. Others claim that the comparison is incorrect since the brain aids the eye by automatically adjusting the pupil and completing the image with its neural networks, and the instant dynamic range of the eye is actually no more than 10-14 f-stops. Let's leave these debates to the best sofa thinkers of the internet.
The fact remains: when taking pictures of friends against a bright sky without HDR on any mobile camera, you either get a normal sky with black faces of friends or well-drawn friends but a completely overexposed sky.
The solution has long been found — to expand the brightness range using HDR (High Dynamic Range). You need to take several shots with different exposures and stitch them together. One will be 'normal,' another a bit lighter, and a third darker. We take the dark areas from the light shot and fill the highlights from the dark one — profit. The only task left is to solve the automatic bracketing problem—how much to shift the exposure of each shot to avoid overdoing it, but a second-year technical college student can now manage to determine the average brightness of the image.
On the latest iPhones, Pixels, and Galaxies, the HDR mode turns on automatically when a simple algorithm inside the camera detects that you are shooting something contrasting on a sunny day. You can even notice how the phone switches to a buffer recording mode to store frames shifted in exposure — the fps drops, and the image itself becomes richer. The moment of switching is quite noticeable on my iPhone X if shooting outside. Take a closer look at your smartphone next time as well.
The downside of HDR with exposure bracketing is its utter helplessness in low light. Even under the light of a room lamp, the frames turn out so dark that the computer cannot align and merge them. To solve the lighting issue in 2013, Google introduced a different approach to HDR in the Nexus smartphone released at that time. It used time stacking.
Time stacking is a simulation of long exposure and time-lapse.
Time stacking allows you to achieve a long exposure effect using a series of short exposures. The pioneers were enthusiasts who filmed star trails in the night sky, finding it inconvenient to keep the shutter open for two hours straight. It was hard to calculate all the settings in advance, and even the slightest shake would ruin the entire frame. They decided to open the shutter for just a couple of minutes but many times, and then they would go home and stitch the resulting frames together in Photoshop.
So, in fact, the camera never really shot with a long exposure, but we obtained an effect of its simulation by stacking several consecutive shots. There have been numerous applications written for smartphones that employ this trick, but they became unnecessary once the feature was added to almost all standard cameras. Today, even an iPhone can easily stitch a long exposure from Live Photo.
Let's return to Google and its night HDR. It turned out that with time bracketing, a decent HDR effect can be achieved in the dark. The technology first appeared in the Nexus 5 and was called HDR+. Other Android phones received it as a sort of gift. The technology is still so popular that it is even highlighted in the presentations of the latest Pixels.
HDR+ works fairly simply: by recognizing that you are shooting in low light, the camera pulls 8-15 of the most recent photos in RAW from its buffer to layer them on top of each other. This algorithm gathers more information about the dark areas of the frame to minimize noise—pixels where, for various reasons, the camera could not collect all the information and failed.
It's like if you didn't know what a capybara looked like and asked five people to describe it—their accounts would be fairly similar, but each would mention some unique detail. This way, you would gather more information than just asking one person. The same applies to pixels.
Overlaying shots taken from the same point produces the same fake effect of long exposure as with the stars mentioned above. The exposure of dozens of frames is summed up, while errors in one are minimized in others. Just imagine how many times you would have to click the shutter of a DSLR to achieve this.
The only problem left was automatic color correction—shots taken in low light typically turn out to be overly yellow or green, while we actually want the vibrancy of daylight. Early versions of HDR+ tackled this with simple adjustments, similar to Instagram filters. Later, they called in neural networks for help.
Thus, Night Sight was born—the 'night photography' technology in Pixel 2 and 3. The description states: 'Machine learning techniques built on top of HDR+ that make Night Sight work.' Essentially, this automates the color correction process. The machine was trained on a dataset of 'before' and 'after' photos to turn various sets of dark, curved images into one beautiful photo.
The dataset, by the way, has been made publicly available. Maybe the folks at Apple will take it and finally teach their glass slabs to take decent shots in low light.
Additionally, Night Sight uses motion vector computation of objects in the frame to normalize any blurring that inevitably occurs with long exposure. Thus, the smartphone can take sharp sections from other frames and stitch them together.
Motion stacking—panorama, super zoom, and noise reduction.
Panorama is a popular activity among rural residents. There are no known cases of a sausage photo being interesting to anyone other than its author, but it cannot be overlooked — for many, this is how stacking began.
The first useful application of panorama is capturing a higher resolution photo than the camera's sensor allows by stitching together multiple frames. Photographers have long used various software for so-called super-resolution images — where slightly shifted photos complement each other between pixels. This allows for images up to hundreds of gigapixels, which is quite useful if you need to print it on a billboard the size of a house.
Another, more interesting approach is Pixel Shifting. Some mirrorless cameras like Sony and Olympus started supporting it back in 2014, but they still required manual stitching of the results. Typical innovations of larger cameras.
Smartphones, however, excelled in this area for a funny reason — when you take a photo, your hands shake. This seemingly trivial problem formed the basis for implementing native super-resolution on smartphones.
To understand how this works, one must recall how the sensor of any camera is structured. Each pixel (photodiode) can only register light intensity — that is, the number of photons that hit it. However, a pixel cannot measure color (wavelength). To obtain an RGB picture, one had to create a workaround — covering the entire sensor with a grid of colored glass filters. The most popular implementation is the Bayer filter, which is used today in most sensors. It looks like the image below.
It turns out that each pixel of the sensor only captures the R, G, or B component, as the other photons are mercilessly reflected by the Bayer filter. The missing components are inferred through a crude averaging of the values of neighboring pixels.
There are more green cells in the Bayer filter — this was designed based on the analogy of the human eye. Thus, from 50 million pixels on the sensor, the green color will be captured by 25 million, while red and blue will each account for 12.5 million. The rest will be averaged — this process is called debayering or demosaicing, and it's a rather crude workaround that everything relies on.
In reality, each sensor has its own clever patented demosaicing algorithm, but for the purpose of this story, we will overlook that.
Other types of sensors (like Foveon) have not really taken off yet. Although some manufacturers are trying to use sensors without a Bayer filter to improve sharpness and dynamic range.
When light is low or the object's details are very tiny, we lose a lot of information because the Bayer filter shamelessly cuts off photons with unwanted wavelengths. This is why Pixel Shifting was invented — to shift the sensor by 1 pixel up, down, right, and left to capture them all. The photograph doesn't actually end up being four times larger, as it might seem; the processor uses this data to more accurately record the value of each pixel. It averages not with neighbors, so to speak, but with its own four values.
The shaking of our hands when taking photos with a phone naturally leads to this process. In the latest versions of Google Pixel, this feature is implemented and is activated whenever you use zoom on the phone — it’s called Super Res Zoom (yes, I also like their ruthless naming). The Chinese have copied it into their phones too, although it turned out a bit worse.
Overlaying slightly shifted images allows us to collect more color information about each pixel, thus reducing noise, increasing sharpness, and elevating resolution without increasing the physical number of megapixels on the sensor. Modern Android flagships do this automatically while their users don’t even think about it.
Focus stacking — any depth of field and refocusing in post-production.
The method originated in macro photography, where a shallow depth of field has always been a challenge. To keep the entire subject in focus, photographers had to take multiple shots with focus shifted back and forth, later stitching them into one sharp image. The same technique was frequently used by landscape photographers to achieve sharpness both in the foreground and background.
This technology has also migrated to smartphones, albeit without much hype. In 2013, the Nokia Lumia 1020 was released with the 'Refocus App', and in 2014, the Samsung Galaxy S5 introduced the 'Selective Focus' mode. They operated on the same principle: with the press of a button, they quickly captured three photographs—one with 'normal' focus, one with focus shifted forward, and the last one with focus shifted backward. The software aligned the frames and allowed the user to select one, which was marketed as 'true' focus control in post-production.
No further processing was needed, as this simple hack was enough to drive another nail in the coffin of Lytro and similar devices with their genuine refocusing capabilities. Speaking of which, let's discuss them (transition master level 80).
Computational matrices — light fields and plenoptic
As we understood earlier, our matrices are a horror show on stilts. We have simply adapted and tried to live with them. In terms of structure, they have barely changed since the dawn of time. We have only refined the technology — reducing the distance between pixels, battling noise interference, and adding special pixels for phase detection autofocus. However, even with the most expensive DSLR, if you try to capture a running cat in indoor lighting, the cat, to put it mildly, will win.
For a long time, we have been trying to invent something better. Many attempts and research in this area can be found under the queries 'computational sensor' or 'non-Bayer sensor', and even the example with Pixel Shifting above can be considered an attempt to enhance matrices using calculations. However, the most promising stories in the last twenty years have emerged from the world of what's known as plenoptic cameras.
To keep you awake amid the anticipation of upcoming complex terms, here’s an insider tip: the camera in the latest Google Pixel is somewhat 'plenoptic'. Just two pixels, but that’s enough for it to calculate genuine optical depth in the frame without a second camera like others.
Plenoptic technology is a powerful tool that has yet to be fully realized. Here’s a link to one of my favorite recent , from which I borrowed examples.
A plenoptic camera will soon be in every household.
Invented in 1994 and assembled at Stanford in 2004. The first consumer camera, Lytro, was released in 2012. Similar technologies are now being actively experimented with in the VR industry.
The only difference between a plenoptic camera and a regular camera is a modification — the sensor is covered with a grid of lenses, each of which covers several actual pixels. Something like this:
if you calculate the distance from the grid to the sensor and the aperture size correctly, the final image will have clear clusters of pixels — like mini-versions of the original image.
It turns out that if you take one central pixel from each cluster and stitch the image together only from them — it will be indistinguishable from a photograph taken with a regular camera. Yes, we lose some resolution, but we can simply ask Sony to add more megapixels in new sensors.
But the fun is just beginning. If you take another pixel from each cluster and stitch the image again — you'll get a normal photograph again, but as if taken with a one-pixel shift. Thus, having clusters of 10 × 10 pixels, we will get 100 images of the object from 'slightly' different angles.
The larger the cluster size, the more images we get, but with lower resolution. In a world of smartphones with 41-megapixel sensors, we can afford to overlook resolution somewhat, but there is a limit to everything. We must maintain balance.
Okay, we've assembled a plenoptic camera, and what does that give us?
True refocusing
The feature that all journalists buzzed about in articles about Lytro is the ability to genuinely adjust focus in post-production. By 'genuine', it means that we do not apply any deblurring algorithms, but only use the existing pixels, choosing or averaging them from clusters in the necessary order.
A RAW photograph from a light field camera looks strange. To transform it into a familiar sharp JPEG, we first need to assemble it. This involves selecting each pixel of the JPEG from one of the RAW clusters. Depending on how we choose them, the result will vary.
For instance, the farther a cluster is from the original ray's impact point, the more out of focus that ray becomes. This is due to optics. To obtain a focus-shifted image, we just need to select pixels at the desired distance from the original—either closer or farther.
Focusing closer was more challenging—physically, there were fewer such pixels in the clusters. Initially, developers didn't want to give users the option to focus manually—the camera decided this programmatically. Users were unhappy with that future, so the feature was added in later firmware under the name 'creative mode', but real focus was heavily restricted for that very reason.
Depth map and 3D from a single camera
One of the simplest operations in light field photography is obtaining a depth map. This just requires capturing two different frames and calculating how shifted the objects are between them. The more shift, the farther from the camera.
Recently, Google acquired and shut down Lytro but used their technologies for its VR and… for the camera in Pixel. Starting with Pixel 2, the camera became 'somewhat' light field-based for the first time, albeit with clusters of only two pixels. This allowed Google to avoid adding a second camera, unlike everyone else, and to compute the depth map solely from a single photograph.
The depth map is constructed from two frames offset by one subpixel. This is sufficient to compute a binary depth map and separate the foreground from the background, blurring the latter in the now-popular bokeh. The result of this layer separation is further smoothed and 'enhanced' by neural networks trained to improve depth maps (not blur, as many believe).
The point is that light field technology in smartphones was almost free for us. We were already placing lenses on these tiny sensors to increase light flow somehow. In upcoming Pixels, Google plans to go further and cover four photodiodes with a lens.
Source: 3dnews.ru
