Research

Reactive User-Guided Path Planning for Visual-Servoing Adaptation

2023-2024, TUM, Munich

Imitation learning promises to allow humans to teach robots tasks simply through demonstration. Traditional methods, relying on visual or kinesthetic teaching, have been limited to guiding robots along a single, static trajectory. This often results in the robot's end-effector colliding with objects, especially when navigating between start and goal poses. Our research introduces an innovative approach: visual servoing with online adaptation of key points and end-goals. This method enhances the robot's capability to handle dynamic tasks, such as moving objects, by allowing for multiple trajectories and supporting iterative learning through on-the-fly demonstrations. This adaptability is crucial in real-world applications where conditions and tasks can vary significantly. The main goal of our project is to develop an adaptive system for Learning by Demonstration (LbD) that is both user-friendly and capable of real-time, reactive planning based on user demonstrations. Our approach extends traditional LbD by incorporating real-time visual servoing, overcoming its limitations in adaptability and responsiveness. By enabling robots to learn from multiple demonstrations and adapt to changes in their environment, we aim to create a system that can handle a wide range of tasks, from moving cups to grasping objects on the move. This system will be equipped to deal with the unpredictability of real-world scenarios, including object movement and human intervention, making robots more flexible and accessible for everyday applications.
Project: Maintain mobility and independence with Geriatronics robotic assistance systems. [Video] [Video]

Flowbot: Robot navigation in crowded human flows

2021, ETH Zürich, Switzerland

Human-aware navigation is essential for achieving harmonious coexistence between humans and robots. However, to generate a feasible path in highly dynamic crowded environments is a huge challenge for human-conscious robot navigation. During the fellowship, my research aims to develop a path planner that can efficiently navigate a robot to its goal through a crowded flow of pedestrians. Traditional planners struggle to find collision-free paths in such scenarios since the amount of free space is limited and always changing. In contrast to traditional planners, we are developing a flow map method that considers the velocity and density of the crowd using shear force and viscosity from fluid mechanics to build the constraints used by our novel sampling-based planning method. Our preliminary experiments show that the proposed method can generate efficient paths in crowded environments by taking into account the flow of pedestrians.
Project: Robotics Student Fellowship Program. [Video]

Dual-arm Mobile Robot with Human-Robot Interaction in Uneven Terrain Environment

2020, Southern University of Science and Technology, China

As salary costs in China increase, it is becoming expensive to recruit workers to handle repetitive tasks such as transporting goods. To address this challenge, we are developing a robot capable of human-robot interaction with the ability to navigate uneven terrain. Our aim is for the robot to lift boxes in response to human instruction and navigate autonomously over environments with uneven terrains, such as sloping ground. Our robot consists of a mobile platform, manipulator arms, and sensors. The core technologies include simultaneous localization and mapping (SLAM), path planning in uneven terrain, dynamic obstacle avoidance, object recognition, grasp detection, grasping control, and automatic speech recognition.
Project Number: JCYJ20170413161616163. [Video]

An Intelligent Robotic System for Autonomous Airport Passenger Trolley Deployment

2019, The Chinese University of Hong Kong, Hong Kong, China

In international airports, it is common to see airport trolleys strewn haphazardly throughout the terminals and surrounding areas. A large amount of manpower has to be hired just to retrieve these trolleys and return them to distribution points. This project proposed an intelligent multi-robot system that can detect the distribution of unoccupied trolleys in different regions of the airport in real-time. The system consequently allocates robots using an optimized strategy to determine the most efficient method and optimal paths to collect the trolleys and bring them to distribution points. The robot was developed using core technologies such as precise localization, active obstacle avoidance, and intelligent motion planning. It completes the following tasks: 1) detecting the trolleys and determining their status, 2) navigating the robot to work safely and smoothly, 3) collecting the target trolley and stacking multiple trolleys together, and 4) cooperating and coordinating with the other robots to transport trolleys to the designated location.
Project Number: Hong Kong ITC ITSP Tier 2 grant # ITS/105/18FP. [Video]

Mobile Sorting Robot for Supermarket Environments

2018, Harbin Institute of Technology, China

In recent years, the automation of logistical processes has captured widespread attention. The JD Robot Challenge focused on the problem of picking and placing objects on supermarket shelves using a fully automated solution. We designed a sorting robot to replace traditional manual picking methods with core technologies, including simultaneous localization and mapping, path planning and dynamic obstacle avoidance, object recognition, attitude estimation, grasp detection, and grasping control. In our solution, the robot, given a sorting task, uses its arm to find and pick up the specific goods that meet the requirements, places the goods in a built-in box to transport to the designated sorting area using automatic navigation, and picks up the goods from the box and place them in the designated spot. In this way, the entire sorting process is completed by the robot autonomously.
Achievement: Second Prize & Golden Egg Prize of International Finals, JD Robot Challenge, Top 2 / 300, Bonus: $50,000. [Video]

Autonomous navigation robot for pest recognition and environmental monitoring

2017, Hainan University, China

Diseases and pests are detrimental to crop yield. At present, the identification of crop diseases and insect pests in China is done manually, which requires significant human resources to carry out. Moreover, it takes experience and knowledge to properly identify the specific issue. To resolve these problems, we designed and implemented an autonomous navigation robot that could identify pests and monitor the environment. Its main functions include pest identification, pest trapping, environmental mapping, and automatic navigation. The following steps comprise the robot’s workflow: 1) navigating autonomously, 2) using the camera on the robot arm to collect plant data and recognize and identify the types of pests on the plant, 3) recording the location of the infected plant and the name of the pests, and 4) transmitting this information to the farmers.
Achievement: 1. Provincial First Prize, ”Challenge Cup” National College Student Curricular Academic Science and Technology Works Competition 2. National Special Prize, The Silk Road Robotics innovations competition. [Video]

An Intelligent Device for Assisting Rubber Tapping

2016, Hainan University, China

The technical level of rubber work is a crucial factor affecting rubber production. Laborers who have a low degree of skill harvest approximately 20–30 percent less rubber than those who are more experienced. Unskilled laborers can even damage rubber trees. For this reason, we put forward a rubber workers’ technical level evaluation method based on the cloud model so as to accurately evaluate technical skills. We also designed an intelligent device with the above method to provide real-time feedback on laborers’ skills; knowing where they have made errors will allow them to subsequently improve. Our results showed a 90 percent accuracy rate in evaluating the workers’ skills, and it provided satisfactory real-time performance benchmarks. Our instrument is beneficial for workers to improve their tapping level and increase their rubber production.
Achievement: 1. Provincial First Prize, ”Challenge Cup” National College Student Curricular Academic Science and Technology Works Competition 2. National Special Prize, The Silk Road Robotics innovations competition.