Manipulation and Vision
Real-world manipulation remains one of the biggest challenges in robotics. Robots are still poorer than humans in using hand-eye-coordination to grasp objects and place them. The Centre is working on improving the manipulation capability of robots through the Manipulation and Vision project.
Team Members
Peter Corke
Queensland University of Technology
Peter Corke is a Distinguished Professor at the Queensland University of Technology (QUT), director of the ARC Centre of Excellence for Robotic Vision, and director of the QUT Centre for Robotics. He is a Fellow of the IEEE, the Australian Academy of Science and the Australian Academy of Technology and Engineering. He is a co-founder of the Journal of Field Robotics, member of the editorial board of the Springer STAR series, former Editor-in-Chief of the IEEE Robotics and Automation magazine and former member of the editorial board of the International Journal of Robotics Research. He has over 500 publications in the field, an h-index of 77 and over 30,000 citations. He has held visiting positions at the University of Pennsylvania, University of Illinois at Urbana-Champaign, Carnegie-Mellon University Robotics Institute, and Oxford University. Peter created the MATLAB and Python Toolboxes for Robotics and Machine Vision, is the author of the popular textbook “Robotics, Vision & Control”, created the Robot Academy repository of open online lessons, and was named the 2017 Australian University Teacher of the Year by the Australian Government’s Department of Education and Training. In 2020 he was awarded the prestigious IEEE Robotics and Automation Society George Saridis Leadership Award in Robotics and Automation. He is Chief Scientist for Dorabot and advisor to LYRO Robotics. His interests include visual-control of robots and the application of robots to problems such as large-scale environmental monitoring and agriculture, internet-based approaches to teaching at scale, open-source software development and writing.
Jack Collins
Queensland University of Technology
Jack Collins is an associated PhD researcher at the Centre and the Commonwealth Scientific and Industrial Research Organisation (CSIRO). His research is focused on the simulation of robots and how we can traverse the ‘Reality Gap’ using sim-to-real methods. The ‘Reality Gap’ describes the discrepancy we see when trying to transfer a controller developed, learnt or evolved in simulation to a real-world platform.
Jesse Haviland
Queensland University of Technology
Jesse graduated from QUT in 2018 with First Class Honours in a Bachelor of Electrical Engineering. He was welcomed to the world of research by his supervisor, Peter Corke when he completed his honours project titled “Interactive Voice Interface for Robotic Manipulation Demonstrator” in 2018/2019. Jesse is currently pursuing his PhD with the Centre supervised by Peter Corke. His research seeks to develop novel reactive control strategies which improve the robustness, exploit redundancy, and maximise the efficiency of mobile manipulators while being adaptable to changes in the environment.
Robert Lee
Queensland University of Technology
Robert graduated from QUT in 2017 with first class honours in a Bachelor of Mechatronics Engineering. During his degree he worked with the Centre on the LunaRoo hopping lunar payload robot, as well as a CSIRO collaboration project involving evolving spiking neural networks for quadrotor control. Robert started his PhD in 2018 and is supervised by Research Fellow Juxi Leitner, Research Fellow Valerio Ortenzi and Centre Director Peter Corke. His interests lie in deep reinforcement learning, control and vision, and he is currently working on applying these tools to improve robotic grasping.
Robert Mahony
Australian National University
Rob Mahony is a Professor in the Research School of Engineering at the ANU and has been a Chief Investigator with the Centre since its inauguration in 2014. His research interests are in non-linear systems theory with applications in robotics and computer vision.
He wrote the seminal paper providing a clear exposition of non-linear complementary filters on the special orthogonal group for attitude estimation; an enabling technology in the early development of quadrotor aerial robotic vehicles. He was the first to provide a principled analysis for using optical flow of control of aerial robotic vehicles and was a coauthor on the first experimental paper that demonstrated landing of a quadrotor vehicle on a textured but featureless moving surface. In 2016, Rob was named a Fellow of the IEEE, recognising his contribution to the control aspects of aerial robotics.
Douglas Morrison
Queensland University of Technology
Doug completed his PhD at QUT in 2020, supervised by Dr Juxi Leitner and Professor Peter Corke. His research is developing new strategies for robotic grasping in the unstructured and dynamic environments of the real world, that is, strategies which are general, reactive and knowledgeable about their environments. Hi goal is to create robots that can grasp objects anywhere at anytime. Doug was also the lead developer of Cartman, the ACRV’s winning entry into the 2017 Amazon Robotics Challenge!
Zheyu Zhuang
Australian National University
Zheyu Zhuang is a PhD student at ANU, supervised by Robert Mahony, Nick Barnes and Richard Hartley. His research is involved in developing visual reaching and grasping strategies that combine traditional control theories and deep learning. Zheyu graduated from ANU with first class honours, majoring in Electronics Engineering and Mechatronics. He was also a developer of Cartman, the Centre’s grasping robot that won Amazon Robotics Challenge 2017.
Project Aim
Real-world manipulation remains one of the biggest challenges in robotics. Compared to humans, robots are still very poor at ‘understanding’ the world they see around them, and using hand-eye-coordination to manipulate objects. The Centre is working hard to address both of these shortcomings through the Manipulation and Vision Research Project (and Manipulation Demonstrator project). Almost all robot manipulators in operation today are used in the manufacturing industry. They are generally separated from humans in tightly-controlled environments. The aim of this project is to develop methods that enable robot manipulators to interact with people and everyday objects. The project team is focussed on three key challenges: vision-guided reaching, vision-guided grasping and task-focused manipulation.
Key Results
The Manipulation & Vision Team continued to work together on solving a number of the challenging Manipulation problems including kinematics, reaching, grasping, placement, non-rigid objects and human-robot interaction.
PhD Researcher Douglas Morrison proposed an object dataset that is useful for training machine learning algorithms for grasping objects. Doug’s dataset was designed to procedurally generate objects that have greater geometric diversity than the sets of “everyday” objects normally used for training. Researchers in other institutions can 3D-print these objects of interest. Douglas successfully completed his PhD at QUT mid-2020.
PhD Researcher Zheyu Zhang worked on utilising the control theory on learning-based visual reaching tasks. With Zheyu’s work, the robot can robustly reach towards different object classes amid multiple identical object instances and distractors. He also collaborated with the Centre’s Robots, Humans and Action project team led by Professor Stephen Gould, on developing a grasping algorithm that allows a robot to grasp and place objects such as IKEA table legs.
PhD Researcher Jesse Haviland has been researching novel reactive control strategies which improve the robustness, exploit redundancy, and maximise the efficiency of manipulators while being adaptable to changes in the environment. A closed-loop manipulator motion controller was developed, which provides obstacle avoidance even when the obstacles are moving. This allows for applications such as closed-loop visual grasping to operate in more complex and dynamic environments.
PhD Researcher Robert Lee focused on the challenging task of deformable object manipulation, such as folding fabric. He developed a novel reinforcement learning method, that teaches a robot folding behaviours without simulation or human demonstrations. The approach is being extended to a variety of materials.
Visiting Ph.D. student Patrick Rosenberger (TU Wien, Austria) spent three months at the Centre’s QUT node, working on human-to-robot object handovers. He collaborated with Centre Director Peter Corke and Group Lead Akan Cosgun to investigate how robots can safely and reliably take an object from a person’s hand. On the same topic, Group Lead Akan Cosgun and an ex-member of the Manipulation Group, Valerio Ortenzi, collaborated on a journal publication called “Object handovers: A review for robotics” which provides a overview of the state of the art in the field.
Undergraduate Researcher Rhys Newbury developed a method for reliably placing handheld objects on a surface in an upright orientations. This work focuses on how a robot should place an object on a table or a shelf, so that the object does not fall and stays in the preferred orientation. For instance, mugs should not be placed upside down. The robot learned to place previously unseen objects in a way that a human would put them down.
The Manipulation & Vision Project team participated in the Alibaba Open Cloud Robot Table Organisation Challenge with 11 members from QUT, Monash and ANU students, and including 4 undergraduate students. The aim of this challenge was to have a robot rearrange objects on a table so that a goal scene is achieved. The team qualified for the second stage and used the challenge as an opportunity to develop essential Manipulation and Vision capabilities for robots.
