Advancing Robotic Grasping, Dexterous Manipulation & Soft Robotics

Jeff Trinkle says he’s not one to get too worked up about things. Still, he has always had a keen interest in robot hands. And, though it may be a long way off, Trinkle, whose research in robotic grasping, dexterous manipulation and related simulation problems has been funded continuously by the National Science Foundation since 1989, says he’s most compelled by the prospect of robots performing “dexterous manipulation” at the level of a human “or beyond.” 

“I’ve always felt that for robots to be really useful they have to pick stuff up, they have to be able to manipulate it and put things together and fix things, to help you off the floor and all that,” he says, adding: “It takes so many technical areas together to look at a problem like that that a lot of people just don’t bother with it.”

But Trinkle, who was the NSF Program Officer in charge of the National Robotics Initiative before arriving at Lehigh's department of computer science and engineering, does more than bother with it. Some of the technical challenges involved in grasping are at the heart of one of his current projects: a collaboration to develop a new approach to the design and construction of soft robots inspired by the movement of natural muscles in soft animal structures—think: giraffe tongues, octopus tentacles and elephant trunks. 

Soft robots, which are pliable and can be deformed and reformed, are the future of human-machine collaboration. Trinkle calls them Robots 2.0.

“Robotics 1.0 was putting robots that are big and heavy—that you would never want to get in the way of—in a warehouse and having them build cars and paint them and all that,” he says. “Robots 2.0 is ‘OK, well, instead of forcing robots and humans to be separated for safety, why don’t we build robots that are intrinsically safe?’”

Robots that move the way tongues, tentacles and trunks move could be useful for sending into areas that are too dangerous for humans, such as disaster zones, the deep ocean or outer space. They may also be “intrinsically safe,” able to work side by side someday with humans in a warehouse or operating room.

Trinkle’s excitement for the potential of such robots led him to work with colleagues from Yale University, the University of Washington and Brown University on the soft robots project, funded by a National Science Foundation Emerging Frontiers in Research and Innovation grant. His role is to use mathematical models, along with computer science techniques like search algorithms, to develop the computer system that “tells” the robot how to move. He offers the Roomba, the robotic vacuum that moves across the floor autonomously, as an accessible example.

“Roombas use the same kind of technologies we are using, just in a very simplified setting,” he says. “Roombas know that there’s a wall. They have an internal map that ensures they can move around without bumping into things too much.”

Trinkle says this map functions similarly to the human brain.

“Your brain sends signals to your muscles and they change their length, contracting or expanding, depending on what you’re trying to execute,” he says. “We are doing something similar here. So imagine taking hundreds or thousands of modular units and using them to build your own elephant trunk.”

To do this, the researchers are closely examining the biology of a number of soft animal structures to better understand how muscle cells work in concert with tendons and other tissue to articulate movements. Trinkle and his students apply this biological data to construct a computer simulation of, in one example, an elephant trunk. The structure’s components and their connections to each other are represented with crisscrossing lines—red lines stand in for active muscle cells and gray for fat and connective tissue, for example. 

Using a simulation tool, Trinkle applies mathematical models to instruct the simulated appendage to curl around a simulated object such as a circle, representing what would be a disk in three dimensions.

To design the “brain” or “map” that will ultimately instruct a three-dimensional robot how to move, Trinkle uses techniques employed in building artificial neural networks, a type of machine learning that is modeled on the human brain. These neural networks are trained through data in a process similar to human learning. The data “trains” the system through a process akin to “trial and error.” In this case, the network is trained with data generated by the computer simulation of an abstracted elephant trunk.

To get the abstracted trunk to curl around the circle and, ultimately, move the circle to another part of the screen, involves multiple steps and a lot of trial and error as the system gets trained.

“[The system] doesn’t know anything, so when it tightens up the fibers on one side or the other, it doesn’t know in advance how it’s going to move, but the neural network is going to figure out that if it does certain things to its muscle fibers, it’s going to move a certain way,” says Trinkle. “Over a long time horizon, the simulation will figure some things out. It’s going to try to end up pushing the disk in the right location.”

He likens the process to an infant learning to crawl. 

“If a baby is trying to learn how to crawl, it’s going to do some things that won’t work, and eventually the infant figures it out,” says Trinkle. “At some point, all of a sudden, the baby solves the problems and now it’s crawling because its neural network has been trained from its experience.” 

In this research, computer simulation is the training ground for robot systems the teams will build.

More intelligent, autonomous & adaptive

Trinkle and his students conduct experiments to extract “ground truth,” or directly observed information, to feed into their computer program. They are working with a next-generation lightweight robotic arm designed specifically for academic and industrial research. A key feature of the arm, according to Jinda Cui, Ph.D. student and research assistant in Trinkle’s lab, is that it has “seven degrees of freedom,” or seven independently controlled joints that allow for superior reachability in the three-dimensional workspace.

“This robot has one redundant joint that can move without impacting the end-effector,” says Cui.

This is significant because it means, for example, that the robot’s gripping mechanism, or “hand,” can remain in the same pose even while the other “arm” joints are moving. This is especially useful when the environment is cluttered and the robot needs to make a lot of adjustments, explains Cui.

Still, the more joints it has, the more challenging a robot is to control. Another benefit of the advanced nature of the robot is that the team can access the low-level system that directly controls the joint movements.

“Through the internet or other communication protocols, we can communicate with the controller directly and tell the joints what to do,” says Cui. “We can modify the joint speed individually, or even how much current, or how much torque we want to give a particular joint.”

Trinkle and his team have also set up a motion capture camera system, similar to what is used to map the movement of actors in computer-generated imagery in film. Their system is designed to gather data about object positioning that would eventually inform their computer program.

“If this robot is trying to manipulate something, like pick your phone up, the vision system should be able to track where your phone is and then the robot can adjust if the phone is not where it expected it to be,” says Trinkle.

In the future, instead of an external motion capture camera system, robots will have built-in sensors that track object motion.

The team’s robot comes with a three-dimensional camera already installed. For their experiments, Trinkle and Cui plan to add other detectors, such as tactile sensors which would be important for programming the robot to note changing contact conditions, like an object’s slipperiness, and then be able to adjust for it.

“Our work,” says Cui, “is to make it more intelligent, more autonomous, more adaptive.”

Closing the ‘reality gap’

Trinkle notes that training a robot’s control system with simulations has its shortcomings. When attempted in three-dimensional space, there is often a “reality gap.” The artificial neural network, trained from simulation data, once applied to a physical robot may try to perform the same task as in the simulation and fail. He says that occurs because the model that was learned from the simulation data was biased toward the way the simulation works.

The challenge for robotics researchers like Trinkle is to develop a solid simulation and then do some testing in the physical world, knowing that some retraining will be required. Hopefully, says Trinkle, researchers are just making small adjustments so that it works in the physical world without having to start the whole process over.

In other words, in developing this new approach to teaching soft robots, Trinkle will try to train the “baby” 90 percent of the way to crawling in simulation, and then get it the rest of the way there by experimenting on a physical robot.

The team plans to build three full robot systems as testing prototypes, from modular motor units including a soft robotic hand that can grasp a wide range of object sizes and shapes; a trunk-like structure with a static base that can grasp and manipulate forward and backward, up and down and left and right; and a worm-like robot that can move freely over terrain with large obstacles. These robots will likely involve some sort of silicone skin to create a more continuous surface for contact.

As for robots someday being able to grasp, Trinkle notes: “It was a hot topic while I was in grad school and then it got cold for 15 years and then it got hot again. Perhaps because so many other problems were solved, and the remaining ones were just as hard as the grasping problem, so now there are a lot more people working on grasping again. And now that AI and neural networks have gotten so big, people are trying to apply those techniques in all sorts of different ways to grasping because it’s still the hard problem that it was 30 years ago.”

Though he believes there is a long way to go before scientists resolve the technical challenges of getting robots to grasp as well as humans or better, Trinkle acknowledges that those are not the only challenges that need to be overcome.

“There are social issues such as: Will people want to be close to robots? Will they become friends?” he ponders. There are ethical challenges to wrestle with as well. He points to autonomous cars as an example. In a difficult circumstance, he asks, should the car be programmed to save the driver and passengers, or save a passing pedestrian who might be impacted?

“There are so many different kinds of problems that people could study,” Trinkle adds. “When it comes to robots, who knows if society will be able to accept them and how far we will be able to advance the technology.”

The possibilities, it seems, might be of interest even to those who don’t tend to get too worked up over things.

A Roadmap to Transforming AI Research

When Dan Lopresti and his colleagues talk about the future of artificial intelligence, be prepared to imagine a better world.

In this world, the full potential of AI is unleashed to benefit society: Health care is personalized and accessible through a friendly robot companion; education is customized to offer individualized plans for retraining and skills-building; and businesses large and small operate with previously unheard-of efficiency and provide a level of customer service that can only be dreamed of today.

“The question is what are we going to see over the next 10 or 20 years break loose as a result of the research, which is assuming the research gets done because of investments made,” says Lopresti, a professor of computer science and engineering. Lopresti is the incoming vice chair of the Computing Community Consortium (CCC) Council which, along with the Association for the Advancement of Artificial Intelligence (AAAI), spearheaded the creation of “A Twenty-Year Community Roadmap for Artificial Intelligence Research in the U.S.”

The Roadmap lays out a case for the best use of resources to fulfill the promise of AI to benefit society. The report also recognizes the tremendous social change that will result, says Lopresti, and that this must be addressed as well. Ethics is also an important consideration across the board.

The goal was to identify challenges, opportunities and pitfalls in the AI landscape, and to create a compelling report to inform future decisions, policies and investments in this area.

“We marshalled the community,” says Lopresti. “This was an amazing effort. In a period of about a year, we got info from hundreds of computing researchers around the country. We ran a series of workshops that were very well attended and produced this Roadmap for AI Research, which is this quite hefty document that looks out twenty years.”

It paints a compelling vision of a future made better through the unleashing of AI’s full potential, with an understanding that attention must also be paid to the possible negative repercussions of this revolution. It’s a future, Lopresti and his colleagues say, that can only be realized through strategic, substantial and sustained investment and a reimagining of how AI research is done.