BreakThrough Study Idea

Speed up the entire system using high-speed vision. Application to the human interface is possibleMasatoshi Ishikawa

<Series 2 / complete>

2019.04.04

SHARE
  • facebook
  • twitter

Previously, in Part 1, Mr. Ishikawa explained the creation of 21st-century technology. The importance of mindset transformation of researchers and proper evaluation of technologies by the society became evident.
In Part 2, Mr. Ishikawa spoke about cutting-edge technologies, which is representative of his research achievements. The topic is “Application of high-speed vision to medical and welfare fields”.


New Value Called “High-speed Vision”

I started the research on “High-speed Vision” from around 1993. Many of the researchers working in image processing still aim at recording more clear videos, such as 4K and 8K, but my research is different from the research done by others. I set my goal as “Capture fast, process fast”, which no one was thinking about at that time. It is original research that gives rise to synthesis, which I spoke about last time.
An ordinary video camera records 30 frames per second (30 fps). This rate matches the speed of the human eye. I started the development of high-speed vision, aiming at image processing that can capture 1000 frames per second (1000 fps).

Let me explain the difference in speed using the figure below.
Let’s imagine that a baseball pitcher throws a fastball at a speed of 150 km/h. If this is captured using a conventional 30 fps video camera that captures 30 frames per second, the distance between the images recorded will be approximately 140 cm. On the other hand, with 1000 fps high-speed vision, which captures 1000 frames per second, the distance between the images recorded will be approximately 4 cm.
When I started my research, capturing images at speeds higher than 30 fps was not considered necessary as that is the speed of the human eye. However, when I recorded the video, I could see the trajectory of the ball that I had not seen before. A new value was created from the images captured at high speed.

Difference between video and high-speed vision
If you look at the video captured using high-speed vision, you come to know that the trajectory is that of an Eephus pitch.

Though it is a convincing theory if you think about it now, it is amazing that the research focused on something that no one else had noticed before. How did the new value called high-speed vision evolve after this?

Speed Up the Entire System and Not Just the Recording Speed

Even if the recording and processing speed of a camera increases using high-speed vision, the speed of the entire system remains unchanged if the computer that controls or actuator that moves the camera is slow. Even if one component is slow, the whole system has to be adjusted to the speed of the element that is slow.
Therefore, I started working on the development to speed up the three components, the sensor that is equivalent to the human sensory system, the computer that is equivalent to the brain, and the actuator that is equivalent to the muscles.
In 1993, when the sensor and processing speed improved to 1000 fps, a large enclosure of size 1.2 m × 1.2 m was required, even though the output was only 4000 pixels. The “Vision chip”, which was gradually miniaturized and commercialized in May 2018, is a general-purpose 1.27 million pixel stacked vision chip measuring 5.678 mm diagonally, which is a product of collaborative research with Sony Corporation. With miniaturization and weight reduction, low power consumption of 0.363 W was also achieved. The chip can be mounted, for example, even on drones. The results of research utilizing high-speed vision chips are expected to be released gradually, and I am looking forward to it.

Vision chip was miniaturized in a short period

High-speed vision has evolved to the stage where stacked vision chips are now commercially available. Is it possible to use high-speed vision in medical care and welfare fields?

Human Interface to Draw on Images in Space with Gestures

High-speed vision is being used in various research fields. Particularly in medical care, the human interface is attracting attention.
It is assumed that the human interface will be used to capture actions of humans with high-speed vision, and use it to move robots, displays, and machines with the movements.
In this case, we start by capturing human movements first. Human actions are surprisingly fast, and the speed is several tens of kilometers per hour even when waving hands. Capturing with the conventional rate of 30 fps is not sufficient, and the 1000 fps high-speed vision is used for detailed recording and processing of images.
When the video is recorded, processed and played at high speeds, the video of the hand actions can be shown in real time. The speed of recording, image processing, and playback can be implemented to the extent that the person's eye cannot recognize the delay between their motion and the recording shown.
Application of this technology has made it possible to draw on images in space. Images in space is a technology developed by Mr. Hirotsugu Yamamoto, associate professor at Utsunomiya University, in which a high-speed display is placed under the table, and mirrors and reflective sheets are used to display images in space.
For example, an image from a PC display can be reproduced in space and a picture drawn in it. Drawing a picture is just a gesture, and by capturing the act of drawing the picture with high-speed vision and processing the video at high speeds, images can be drawn in space in real time. If there is no lag between your hand and drawing movements, the operability improves, and you can draw neat pictures.
This feature is being researched and developed for applications so that during surgery, physicians can see the data of patients as images in space and mark the affected areas.
Other research such as moving 3D displays with gestures are also in progress.

Interview Date: March 11, 2019

 

Series

Transition to a society that correctly evaluates new values created by the transformation of mindset from analysis to synthesis

Speed up the entire system using high-speed vision. Application to the human interface is possible

High-speed tracking assists procedures such as working using microscopes or robotic surgeries

SMEs determine social acceptability using low-cost Proof Of Concept (POC) and attempt a breakthrough


Masatoshi Ishikawa
Dean, Graduate School of Information Science and Technology, The University of Tokyo

1977 Graduated from Department of Mathematical Engineering and Information Physics, School of Engineering, The University of Tokyo 1979 Completed diploma and master’s course from the Department of Mathematical Engineering and Information Physics, School of Engineering, The University of Tokyo, and became Dr. of Engineering in 1988 (University of Tokyo). He was a senior researcher at the Agency of Industrial Science and Technology, Ministry of International Trade and Industry (now National Institute of Advanced Industrial Science and Technology) in 1979, and he worked as an assistant professor at the Department of Mathematical Engineering and Information Physics, School of Engineering, The University of Tokyo since 1989. Next, he served as a director and associate dean of the University of Tokyo and has been in his current post since 2016.
His field of specialization is systems informatics (sensor engineering, robotics, image processing, perception and behavior systems, bio-information processing). He is engaged in the research on sensor fusion, massively parallel high-speed vision, high-speed robots, visual feedback, meta-perception, optics in computing, intelligent tactile sensors, and circuit models of bioinformation.
In addition to numerous awards at domestic and international conferences, he was awarded the Medal with Purple Ribbon, in November 2011.

◇Major Publications as Co-author
“Robot Control Handbook” (Kindai Kagaku Sha Co., Ltd,) Published in December 2017
“Introduction to Information Network Science” (Corona Publishing Co., Ltd) Published in October 2015, and various other books

share by SNS

related article