Neural networks have made a seismic impact on how engineers design controllers for robots, catalyzing more adaptive and efficient machines. Still, these brain-like machine-learning systems are a double-edged sword: Their complexity makes them powerful, but it also makes it difficult to guarantee that a robot powered by a neural network will safely accomplish its task.

The traditional way to verify safety and stability is through techniques called Lyapunov functions. If you can find a Lyapunov function whose value consistently decreases, then you can know that unsafe or unstable situations associated with higher values will never happen. For robots controlled by neural networks, though, prior approaches for verifying Lyapunov conditions didn’t scale well to complex machines.

Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and elsewhere have now developed new techniques that rigorously certify Lyapunov calculations in more elaborate systems. Their algorithm efficiently searches for and verifies a Lyapunov function, providing a stability guarantee for the system. This approach could potentially enable safer deployment of robots and autonomous vehicles, including aircraft and spacecraft.

To outperform previous algorithms, the researchers found a frugal shortcut to the training and verification process. They generated cheaper counterexamples — for example, adversarial data from sensors that could’ve thrown off the controller — and then optimized the robotic system to account for them. Understanding these edge cases helped machines learn how to handle challenging circumstances, which enabled them to operate safely in a wider range of conditions than previously possible. Then, they developed a novel verification formulation that enables the use of a scalable neural network verifier, α,β-CROWN, to provide rigorous worst-case scenario guarantees beyond the counterexamples.

“We’ve seen some impressive empirical performances in AI-controlled machines like humanoids and robotic dogs, but these AI controllers lack the formal guarantees that are crucial for safety-critical systems,” says Lujie Yang, MIT electrical engineering and computer science (EECS) PhD student and CSAIL affiliate who is a co-lead author of a new paper on the project alongside Toyota Research Institute researcher Hongkai Dai SM ’12, PhD ’16. “Our work bridges the gap between that level of performance from neural network controllers and the safety guarantees needed to deploy more complex neural network controllers in the real world,” notes Yang.

For a digital demonstration, the team simulated how a quadrotor drone with lidar sensors would stabilize in a two-dimensional environment. Their algorithm successfully guided the drone to a stable hover position, using only the limited environmental information provided by the lidar sensors. In two other experiments, their approach enabled the stable operation of two simulated robotic systems over a wider range of conditions: an inverted pendulum and a path-tracking vehicle. These experiments, though modest, are relatively more complex than what the neural network verification community could have done before, especially because they included sensor models.

“Unlike common machine learning problems, the rigorous use of neural networks as Lyapunov functions requires solving hard global optimization problems, and thus scalability is the key bottleneck,” says Sicun Gao, associate professor of computer science and engineering at the University of California at San Diego, who wasn’t involved in this work. “The current work makes an important contribution by developing algorithmic approaches that are much better tailored to the particular use of neural networks as Lyapunov functions in control problems. It achieves impressive improvement in scalability and the quality of solutions over existing approaches. The work opens up exciting directions for further development of optimization algorithms for neural Lyapunov methods and the rigorous use of deep learning in control and robotics in general.”

Yang and her colleagues’ stability approach has potential wide-ranging applications where guaranteeing safety is crucial. It could help ensure a smoother ride for autonomous vehicles, like aircraft and spacecraft. Likewise, a drone delivering items or mapping out different terrains could benefit from such safety guarantees.

The techniques developed here are very general and aren’t just specific to robotics; the same techniques could potentially assist with other applications, such as biomedicine and industrial processing, in the future.

While the technique is an upgrade from prior works in terms of scalability, the researchers are exploring how it can perform better in systems with higher dimensions. They’d also like to account for data beyond lidar readings, like images and point clouds.

As a future research direction, the team would like to provide the same stability guarantees for systems that are in uncertain environments and subject to disturbances. For instance, if a drone faces a strong gust of wind, Yang and her colleagues want to ensure it’ll still fly steadily and complete the desired task. 

Also, they intend to apply their method to optimization problems, where the goal would be to minimize the time and distance a robot needs to complete a task while remaining steady. They plan to extend their technique to humanoids and other real-world machines, where a robot needs to stay stable while making contact with its surroundings.

Russ Tedrake, the Toyota Professor of EECS, Aeronautics and Astronautics, and Mechanical Engineering at MIT, vice president of robotics research at TRI, and CSAIL member, is a senior author of this research. The paper also credits University of California at Los Angeles PhD student Zhouxing Shi and associate professor Cho-Jui Hsieh, as well as University of Illinois Urbana-Champaign assistant professor Huan Zhang. Their work was supported, in part, by Amazon, the National Science Foundation, the Office of Naval Research, and the AI2050 program at Schmidt Sciences. The researchers’ paper will be presented at the 2024 International Conference on Machine Learning.

At SmartThings, we’re committed to providing developers with the tools they need to create innovative and seamless smart home experiences. We are excited to introduce our newest addition to the developer toolkit: Works with SmartThings (WWST) Test Suite.

The Test Suite is designed to assist our device company partners in certifying their devices as part of the WWST certification process. With Test Suite, developers can easily test SmartThings integrations, get certified faster, and showcase their devices to tens of millions of SmartThings users.  

What is the WWST Test Suite?

The Test Suite is a comprehensive tool designed specifically for device developers integrating with the SmartThings platform. It provides a unified environment where developers can thoroughly test their products’ capabilities across various scenarios, helping to identify and address potential issues before deployment.

Why Should Developers Use the Test Suite?

• Centralized Testing Environment: With the Test Suite, developers can leverage a centralized testing environment hosted on the SmartThings platform. This eliminates the need to set up complex testing configurations locally, saving time and effort in the testing process. Recently, there has been expanded coverage for 90+ of the most popular “Capabilities” with more planned by the end of the year. 

• User-Friendly Interface: With an intuitive and user-friendly interface, the Self-Testing Tool ensures a seamless experience for product owners and developers of all levels. 

• Real-Time Feedback: Receive instant and detailed feedback on your product’s compliance status, allowing for quick identification and resolution of any issues.

• Reduce your Testing Costs: Save money and time on testing with our free and easy-to-use Test Suite for your certification process. 

Hear from some of our partners using Test Suite to certify easier and faster than ever.

“I think comparing all the processes, SmartThings was probably one of the fastest in terms of total time spent from start to finish.” Mark Hollands, LIFX

“The SmartThings new console is very easy to use. Compared to others, SmartThings Developer Center is a much more user-friendly interface.” James Shurte, Leviton

How to get started with the WWST Test Suite

Ready to use the Test Suite for your device(s)? Here’s a quick guide to help you get started: 

Access the Test Suite

Visit the SmartThings Developer Console and navigate to the Test Suite section, which is accessible at SmartThings Developer Console.

Set Up Your Testing Environment

Test Suite will display a list of available devices connected across your locations. If you do not have any devices connected to the location associated with your developer account, use the SmartThings app to add them. 

It’s easy to collaborate with colleagues using Test Suite. You can add team members in the development console and they can check the status of device tests which makes it easy to work together.  

Test Scenarios

Test Scenarios are automatically generated to test the Capabilities of your device. This reduces the time needed to set up tests and makes your development testing easy. 

We know some Capabilities may require more complicated setups; you can select which tests to perform in each session.

Tip: Make sure you include them all before submitting for certification to ensure a faster WWST process. 

Run Tests

Once you have set up your Capabilities, you can start your tests.   

Many tests will automatically execute, but certain tests require manual action to be taken. For example, when testing a motion sensor, a motion detection event is required to pass successfully. 

These requirements are indicated before and during an active test session. Note that some tests must be completed within a set time or otherwise will be considered a failed test. 

Submit your Test Results

Once the test is complete, you can use any feedback or insights to iterate on your product and retest if necessary.  

Once you’re ready, attach your test results to your WWST certification submission. We will review your test results and assign a Partner Success Manager to help you complete your Works with SmartThings certification.

Our new Test Suite tool represents a significant step forward in our commitment to supporting developers and enhancing their experience on the SmartThings platform. By providing a robust testing environment with simulation, automation, and integration capabilities, we aim to empower developers to build high-quality SmartThings integrations with confidence.

Are you ready to elevate your SmartThings development experience? Dive into the Test Suite today and unlock new possibilities for your smart home innovations.

For support with bulk registrations of your portfolio of products, contact partners@smartthings.com