UC Berkeley researchers develop artificial intelligence to remove shadows from photos

Whether eating out at a restaurant or taking a hike in nature, UC Berkeley doctoral candidate Cecilia Zhang always has a camera at hand.

As a lover of visual media, Zhang noticed that individuals are becoming increasingly reliant on mobile phones to take photos and wanted to find a way to bridge the gap between casual portraits and those produced in a professional studio.

In order to fulfill this need and push casual photography forward, Zhang and researchers at the Massachusetts Institute of Technology, Google and UC Berkeley have developed a way to minimize natural and facial shadows from portraits using artificial intelligence, or AI.

“After going through thousands of casual portraits in the internet, I realized there’s a large issue with lighting and shadows,” Zhang said. “Most people don’t have access to professional equipment and can’t get the environment to bend to their needs.”

From the beginning, Zhang knew she wanted to enhance casual photography and scoured the internet for thousands of portraits. By looking at these images, she found that lighting and shadows were some of the most prevalent challenges for smartphone photography.

After isolating her primary research interest, Zhang and her colleagues embarked on a six-month research project to improve lighting and eliminate two types of shadows from photographs that had been taken previously.

One type of identified shadow is a “foreign shadow,” or a shadow created by environmental factors surrounding a portrait’s subject. The other type of shadow is caused naturally by one’s facial features, according to the study.

After Zhang and her teammates identified the two types of shadows, they developed a dataset of portraits. The researchers then used neural networks, or computing systems that resemble biological neurological networks, to create tools that can teach computers to generate the ideal image by either lightening shadows or removing them altogether.

In addition to addressing the two types of shadows, the research team focused on improving ratios of lighting within photos that had been taken previously.

“Discovering this problem of neutralizing portrait shadows and lighting is really important,” Zhang said. “It’s quite unique and also quite new.”

According to Zhang, while the algorithm can adjust photos after they have been taken, the recently developed AI technique does not have the ability to process photos in real time. She added that developing such a system could be feasible in the future, especially for mobile phones.

Looking ahead, Zhang hopes to use a similar computational approach to address other challenges associated with casual photography and is interested in applying the team’s findings to video.

“It’s not yet done,” Zhang said. “There are a lot of things that we can still improve upon with casual photography.”

Contact Mallika Seshadri at [email protected] and follow her on Twitter at @SeshadriMallika.