When computer vision blends physics with biology…

Nature really is amazing, and computer vision scientists are learning from the abundance of wonders that are found in the natural world.

The human brain processes flashes of light up to 60Hz, anything above this we see as a solid stream of light. This is why we can see perfectly when we’re walking or running but the world around us blurs when we’re driving fast.

Some birds of prey have evolved to see up to 100Hz – this skill allows a Peregrine Falcon to grab prey right out of the sky while diving at 180 miles per hour. Additionally, the presence of two foveae in the eye compared to a human’s single one enables the bird to focus on two objects at once – one close up and one far away. Add to this the Falcon’s ability to see UV light, so a much wider spectrum than is available to us mere humans, and it’s clear why such predators are so feared in nature. Elements of these attributes are now being encompassed in computer vision to develop superior sensors and improved lenses.

Sensor development

Dynamic Vision Sensors (DVS) made their way into computer vision a few years ago. These differ from standard vision sensors as each individual pixel operates independently and is only triggered into action if it is required to capture a change in what’s being viewed, and will display this as a change in light intensity not in color. Event cameras use such sensors to allow them to operate in low-light and sometimes fast-moving environments, conditions which traditionally throw up challenges for drone operation. A drone-based camera being developed at the University of Zurich could, using bio-retina inspiration, assist in challenging search-and-rescue work.

In 2016, Samsung adapted IBM’s TrueNorth neuromorphic chip into its sensor to create cameras that can capture video at 2,000 frames per second while remaining massively power-efficient and cool-running – just like a human eye[1]. Companies more closely aligned with the machine vision sector are now also adopting such technologies – Prophesee, for example, formerly known as Chronocam.

Medical imaging benefits from nature’s ingenuity

Last year, scientists from Washington University and University of Illinois demonstrated how understanding the complex structure of the eye of the Mantis Shrimp could advance imaging for early cancer detection[2]. They constructed a camera based around the ability of the crustacean to see both color and polarized light. It’s believed the shrimp interprets these images to assist in communication and in mating, as certain organs become more visually pronounced, and female fertility changes with tidal cycles, meaning awareness of moon phases is an advantage for the males.

The shrimp has 16 color receptors which are stacked on top of each other and arranged in such a way as to enable a wider variety of wavelengths to be absorbed; the researchers used this concept in a camera and stacked multiple photodiodes on top of each other to capture short, shallow blue wavelengths as well as longer, deeper red wavelengths. Furthermore, certain species can see six types of polarization – vertical, horizontal, two diagonals and two circular polarizations.

This trait was captured by the team by adding miniaturized polarized lenses to the camera. In addition to medical imaging, the low-cost camera, named Mantis Cam, promises enhancements in underwater observation and understanding of marine worlds, and improvements in the sight and safety of self-driving cars.

The researchers behind the innovation, Viktor Gruev and Missael Garcia, have also proposed a version of the camera for image-guided surgery based on technology inspired by the ability of morpho butterflies to see near-infrared fluorescence[3]. This camera simplifies the image capture process and avoids the beam splitters and relay lenses which typically cause misalignment in traditional imaging, supporting more accurate and precise surgery. By detecting fluorescence deep within tissue, the camera is sensitive to fluorescence labels that cannot usually be seen in a brightly-lit operating theatre.

The march of the Antbots

Ants are able to use polarized light to navigate when extreme temperatures mean pheromones would evaporate – this was proved in laboratory tests as far back as 1976 by experimenting with Cataglyphis ants and cheese rewards! Bees, locusts and other insects are believed to employ the same piloting skills.

This year, researchers at the French National Center for Scientific Research (CNRS) unleashed Antbot – a six-legged creation that uses an optical compass which detects polarized light to navigate instead of GPS, supposedly making its exploration and safe return to base more precise. While the 50cm-wide robot took a meandering course to investigate its surroundings, it always took a direct, straight line home. This technology is being developed with space exploration and autonomous vehicles in mind, although, ironically, over-heating issues are limiting its current range.

Giving back to nature

So what are scientists giving back to nature in return for these wonderous inspirations? A research team combining experts from the University of Science and Technology of China and Gang Han and the University of Massachusetts Medical School has found a way to give mice infrared vision[4]. It’s thought that the theory behind the experiment – injecting nanoparticles into the mice’s eyes – could be adapted for humans. So, leave the heat-vision goggles at home and forget eating all your carrots; science could give us “natural” hyperspectral vision after all!

Active Silicon have a range of frame grabbers, camera interface boards and embedded vision products to support state-of-the-art vision systems. View our product range here or contact us to understand more.

[1] https://www.cnet.com/news/samsung-turns-ibms-brain-like-chip-into-a-digital-eye/
[2] Mantis shrimp visual system leads to cancer detecting camera | Electrical & Computer Engineering | UIUC (illinois.edu)
[3] https://www.osa.org/en-us/about_osa/newsroom/news_releases/2018/new_camera_inspired_by_butterfly_eyes_improves_ima/
[4] https://gizmodo.com/incredible-experiment-gives-infrared-vision-to-mice-and-1832940986