BreakThrough Study Idea

True Role Expected of Service Robots in the Age of IoT<Series 3 / 3>

Study Idea

2017.02.17

SHARE
  • facebook
  • twitter
  • Communication Robot Installed and Used at 600 Elder Care Facilities Nationwide -FUJISOFT-

The communication robot, PALRO, uses artificial intelligence (AI) equipped on the main unit and a cloud-based AI to communicate with people. Based on the conversation, it analyzes the mood of the other person through his/her facial expressions and quality of voice, allowing it to gather data that can be linked to new proposals.

FUJISOFT leverages its software development technology nurtured over many years and intelligence technology research jointly conducted with research institutions and educational institutions such as universities. It started developing PALRO since around 2008, releasing the Academic Series for use at educational institutions in March 2010. It also put the PALRO Business Series for B2B on the market in June 2012, delivering PALRO to 600 elder care facilities nationwide up until now.

Hybrid structure of AI on PALRO’s main unit and cloud-based AI

PALRO features a compact size of 40cm in length and 1.8kg in weight. When placing it on a table, it naturally takes a position of looking up at the person’s face, and it always talks to the other person looking straight into his/her face. It is a basic manner to talk to the other person looking at the person’s eyes in a conversation between human beings, and averting one’s face or eyes while talking may offend the other person. Due to this, PALRO always talks while looking straight at the other person.

PALRO also constantly follows the face of the other person who is moving. That is to say, people constantly move while they are talking, and in only 0.3 seconds they are said to move horizontally about the length of the neck. If PALRO cannot follow such movements, it will lose track of the other person’s face and will be looking all over the place, resulting in an awkward conversation. In order to make up for its narrower field of view than human beings, PALRO conducts 3D mapping of its surroundings. It also uses a mic and ultrasonic wave sensor to determine the position of the other person as well as the direction and distance of the sound source, making it possible to quickly capture the other person’s face at the center of its field of view.

Because it is equipped with an artificial intelligence (AI) called “front-end AI” on the main unit, it can remember people’s face and name and call people by their names, as well as follow their faces and start a conversation.

The system supplementing this front-end AI is the cloud-based AI. For example, the cloud-based AI handles complex analysis, interpretation, and arithmetic processing of words and data.

In this way, PALRO features a hybrid AI structure utilizing both front-end AI and cloud-based AI.

PALRO has a hybrid AI structure in which a cloud-based AI supplements the AI of the main unit

Front-end AI that controls natural conversations

The front-end AI, which is an original technology developed by FUJISOFT, controls PALRO’s communication behavior toward people in an optimal manner. In other words, it makes the robot face the other person properly, keep the appropriate interval and inflection when speaking, and conduct gestures at the timing and speed that matches the content of the conversation. Furthermore, it enables the robot to memorize the other person’s hobbies and preferences, and hold conversations based on past knowledge and experience. These communication behaviors bring a feeling of security to the other person and also generate a human-like feeling of conversation. The system presiding over these features is the front-end AI, enabling PALRO to carry out conversations that do not feel strange, like a conversation between human beings.

For example, when PALRO is asked, “What’s the weather today?” it responds in 0.4 second, “Let me see.” In a conversation, the length of time required for response determines whether the communication will be smooth. If the robot spends too much time until it returns a response in a conversation, then the person will grow tired of the conversation and leave. However, the front-end AI realizes a short response time to inquiries by the other person. Additionally, it enables the robot to continue talking while accessing the cloud-based AI for detailed information considered to be required in the conversation as needed.

PALRO Business Series is used at elder care facilities

PALRO Business Series has most frequently been installed at elder care facilities, and it is being used at 600 locations nationwide. PALRO’s basic functions include talking, walking, memorizing, and connecting to the Internet and the cloud, and it is also equipped with functions such as singing, dancing, quizzing, games, exercise, recreation, and MC.

PALRO can become friends with the other person through conversation. It individually recognizes the user and says, “Good morning, _____.” It can register over 100 friends, and it also records the person’s information such as gender, age, birthday, hobbies, and preferences, so it can hold conversations sprinkled with topics that are suitable for each person.

The recreation function consists of a wide variety of contents to the extent that users can enjoy a different content every day for 365 days. The exercise function includes exercises that have been developed according to the demonstration experiment using PALRO, called “Fall Prevention and Physical Fitness Improvement Exercise Program,” and have been verified to have an effect on prevention nursing care.

Masaki Shibuya, Executive Operating Officer and Director of Product & Service Business Division at FUJISOFT, states the reason FUJISOFT has developed a hybrid structure AI as follows: “An optimal AI service cannot be provided unless the front-end AI equipped on the main unit of PALRO and the cloud-based AI our properly linked and working as a hybrid.”

PPALRO is a robot that listens, synchronizes, and empathizes with the other person. To serve this purpose, its important task is to first accurately understand the other person. And it needs to start various conversations with the other person while following the person in real-time and understanding the person. Thus, it matches the face and name of the other person through high-precision mapping of unique parts of the other person such as the eyes, nose, mouth, and contour. And it understands who it is communicating with before starting conversations on various topics, taking into consideration attribute information such as the other person’s gender, age, birthday, hobbies, and preferences.

Understands who it is talking to based on high-precision mapping of unique parts of the person’s face

Constantly analyzes the other person during the conversation

PALRO constantly analyzes the other person during a conversation. For example, it analyzes the mood of the other person based on his/her facial expression, voice quality, and response speed to a question on a specific theme. Based on this, PALRO obtains data such as the user's preference regarding the theme and its degree of popularity.

Executive Operating Officer Shibuya states, “It is programmed to talk about a certain theme at the same time every day.”

It gathers the voice (response) of many users from there, and it accumulates information on the theme as data. Mr. Shibuya says that, even if a certain theme is very interesting to elderly users, the user’s mood regarding that theme always tends to turn negative due to a decline in physical functions. The company has gathered and accumulated such information on mood from many people through conversations with PALRO, and in the future it believes that analyzing this data using a cloud-based AI will lead to plans to improve people’s mood on specific themes. Even under extremely busy situations at elder care facilities, the people working there properly understand the information of each user. By further using AI to analyze certain words or moods found in daily conversations with PALRO on specific themes, the company may be able to propose ways of switching a negative mood into an improvement in motivation (i.e. provide new value), and PALRO would be able to significantly demonstrate its role in an IoT environment.

Executive Operating Officer Shibuya states, “I believe in the future we will be able to provide proposals for improvement while gathering and accumulating detailed information that even workers are unable to directly hear from users.”

Toward this end, the company has currently begun considering to provide feedback to facilities that have installed PALRO of the analysis outcomes using cloud-based AI of big data collected through front-end AI.

Mr. Shibuya further states, “We hope to continue solving issues faced at elder care facilities through AI and IT. We also believe there needs to be improvements, including systemic ones, such as the applicability of PALRO under long-term care insurance. Furthermore, we hope to provide [PALRO] in new fields, too, because AI technology is currently needed in various fields.”

In addition to elder care facilities, FUJISOFT plans on broadly deploying PALRO in other industrial fields including financial institutions.

  • Other Trends and Topics on Communication Robots
    Communication-Type Robot that Can Be Differentiated From Competitors By Customization Through Original Scenarios
    -NTT Docomo/S&B Foods-

NTT Docomo started offering Oshaberi Robot for Biz, a communication-type robot service for companies that has NTT Docomo's base technology for conversation mounted on Tomy Company’s communication robot, OHaNAS. This service allows companies to differentiate from competitors by programming original scenarios, and the major food manufacturer S&B Foods has launched sales promotion using OHaNAS at supermarkets and shopping malls nationwide.

  • Robot that Supports Teller Service at Bank Branches
    -NTT DATA-

NTT DATA has used Vstone’s communication robot, Sota, to carry out demonstration experiments of assisting customer service at branches of financial institutions since November 2015. When customers approach the service desk of financial institutions, the robot gives a guide to the service content and listens to the their questions in place of clerks. While continuing to conduct such demonstration experiments, NTT DATA is starting to find possibilities in providing new services based on the data of conversations with customers gathered by the service robots.

  • Service Robots & New Businesses
    Released One After Another by Major Electronics Manufacturers
    -NEC Platforms/Fujitsu-

Major electronics manufacturers are releasing service robots and related services one after another in anticipation of an age of IoT. In May 2016, Fujitsu announced its communication robot Robopin and, in July, NEC Platforms launched a new business with open platform PaPeRo i, which uses a robot as the user interface.

Fujitsu’s Robopin (left) and NEC Platforms’ PaPeRo i (right)

<Other>

Vol.1 Vol.2

share by SNS

related article