How machine learning is changing your smartphone camera
Let’s say you are a mobile photographer. Or to make it easier- you like to take photos on your phone, and when you see a new phone that’s coming to the market, a good camera is one of your topmost priorities. Nowadays, there are phones with 108 Megapixel cameras (Samsung Galaxy S20 Ultra) and when you see a phone like that, you are blown away on the spot, right at the moment, you see it. “108 megapixels”, you utter in amazement and then see the price; a hefty £1,048 ($1294.16), and then you ask yourself- is it worth to pay such a huge sum, nearly a month’s income for a young student for a phone? Maybe. Maybe a £1100 phone like Galaxy S20 Ultra is worthy enough to buy, and that’s always a personal preference, I’m not going to interfere on that. I’m not going to talk about the worthiness of buying a phone with a thousand pounds. What I’m going to talk about is- how machine learning affects your photography experience on a smartphone and how it fares to a far older phone of the same calibre.
In recent years, Google took the smartphone world by storm by launching its phones- Pixel. The obvious name suggests it will be a phone greatly focused on the camera, and it beats out every competitor by their camera. What’s more fascinating is, even though Samsung and other phones were implying 2-3 cameras, Google did it by one camera only- their one camera was enough to beat the combined power of dual or triple cameras (their last phone had a dual camera, but still not 4 or 5 like others). How is Google doing it? Enter “Computational Photography”- the term which is getting lots of press lately as Apple and Google roll out their latest devices.
Computational photography is nothing new. In fact it came a long way after the term was re-coined and given a broader meaning by Marc Levoy for a course he taught at Stanford in 2004. All the digital cameras come with some form of ‘computational’ image processing, in that they have to take data coming from the digital image sensor and render it into a format which will then be used (such as JPEG, or maybe some RAW formats). Some cameras do little, while some cameras like Fujifilm & Sony mirrorless cameras, do amazing image enhancements that mimic old films types and other impressive effects, all performed directly in-camera. So why all the hype surrounding pre-existing technology? It’s because of the implementation of new technology within the latest generation of smartphones where machine learning is directly combined with traditional image processing which has everyone excited, and for good reason. The great images of the Pixel phones are the result of clear beneficiaries of Google’s AI prowess—specifically, Google’s Visual Core, a co-processor that Google developed with Intel. Now, machine learning can ‘learn’ and differentiate using algorithms to classify photos into various scenes such as; landscape, portrait, night, daytime, etc. And this algorithm would perform this classification very accurately, giving the camera exactly what it must correctly set all of the colour information, white balance, exposure and sharpening automatically when you take the photo.
Low-light photo sample taken by Google Pixel 4. (taken from Mr Mobile YouTube channel)
The sample photo on the left (taken from Mr Mobile- the YouTube channel) exhibits none of those common problems, all of the lighting has been captured exactly as it looked to our human eye. Not any magic trick, just the combination of that per-pixel machine learning algorithm and multiple images captures done directly in the camera’s hardware. The machine learning model, trained to understand what the proper luminescence value should be for every pixel within the image chooses the closest match from the set of images taken which is then combined to make the ultimate image. By combining several images, the image processing system can eliminate significant amounts of noise for areas of the photo that aren't moving. For those areas where the motion will be detected (by checking pixel data between each frame that was captured), a machine learning algorithm was used to remove the noise. Think about how this technology might perform when used with a way larger image sensor, presumably during a dedicated camera like those previously mentioned. A magical and beautiful result indeed.
Google’s camera capability became so good by the use of AI that it has become the new industry standard. After a release of a new phone, people see it, Youtubers review it and then they conclude by saying, “Okay yeah that’s good, let’s see how well you fare with the Pixel.” Some people even root their Android phones and use GCam- Google’s camera app to reinforce their phone camera performance. This may come across like a piece exclusively praising Google but these are the facts- because a lot of other phones have similar features, but the Pixel executes each one of them well. And a lot of them work in the background without requiring any extra effort. These new features combined with Google's powerful image processing help the Pixel maintain its status because it has the best smartphone camera around. The other manufacturers should take notes on how they can improve their basic cameras on paper just by using AI and machine learning and how they can weigh in on the yet unexplored realm of computational photography to keep up to the booming multi-billion dollar smartphone industry and give us great cameras in the years ahead.
Photography is a form of art, a medium of protest. With the huge advancement of the smartphone camera, the art of expression will take a step further. The rise of e-commerce requires good product photography for your business, and what another way can you promote your small business other than taking photos of your great products? Just think, the possibilities are endless.
Happy Photography!
Thumbnail Credit: Pixabay.com