Neural Control

The Sensors Group develops neuromorphic sensors and hardware neural accelerators for agile nonlinear robotic control.

Inspired by our event driven perception and quick sparsity-aware neural hardware and observations like seeing a bat catching an insect and an octopus manoeuvring its eight arms, we have been working to close the gap between the control we observe in nature and robotic applications. We use machine learning tools to find the optimal control quickly and with limited power budget. We are interested in learning system models and studying how to incorporate continual learning for robust adaptive control strategies. The algorithms are tested on systems in simulation and on physical robots. We perform physical experiments on our high-performance “INIvincible” F1TENTH race car (in collaboration with the ForzaETH team in PBL at ETH) and our physical cartpole robots. We aim in proving the usefulness of our ideas in the competitive setting of F1TENTH international autonomous race.  Since 2018, we organized a series of neural control topic areas at the Telluride Neuromorphic Workshops.

Questions we are working to answer include:

1. Can we teach a recurrent neural network controller to automatically adjust to changes in the environment (e.g. friction change of the racetrack)? 

2. Can we deploy our controller to be executed by neural accelerator, e.g. Spartus, or EdgeDRNN, our FPGA RNN hardware accelerators? 

3. Sensor fusion: Can we use a neural network to reduce the computation required for state estimation of our car?

Results

Neural Control Publications

1. Heetmeyer, Frederik, Marcin Paluch, Diego Bolliger, Florian Bolli, Xiang Deng, Ennio Filicicchia, and Tobi Delbruck. 2023. “RPGD: A Small-Batch Parallel Gradient Descent Optimizer with Explorative Resampling for Nonlinear Model Predictive Control.” In *2023 IEEE International Conference on Robotics and Automation (ICRA)*, 3218–3224. doi:10.1109/ICRA48891.2023.10161233. http://dx.doi.org/10.1109/ICRA48891.2023.10161233 . 

2. Delbruck, Tobi, Rui Graca, and Marcin Paluch. 2021. “Feedback Control of Event Cameras.” In *2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)*, 1324–1332. IEEE. doi:10.1109/cvprw53098.2021.00146. https://openaccess.thecvf.com/content/CVPR2021W/EventVision/html/Delbruck_Feedback_Control_of_Event_Cameras_CVPRW_2021_paper.html . 

3. Gao, Chang, Rachel Gehlhar, Aaron D. Ames, Shih-Chii Liu, and Tobi Delbruck. 2020. “Recurrent Neural Network Control of a Hybrid Dynamical Transfemoral Prosthesis with EdgeDRNN Accelerator.” In *2020 IEEE International Conference on Robotics and Automation (ICRA)*, 5460–5466. ieeexplore.ieee.org. doi:10.1109/ICRA40945.2020.9196984. http://dx.doi.org/10.1109/ICRA40945.2020.9196984 . 

Supervised undergraduate projects in Neural Control

To gain experience in neural control we supervised a large number of ETH student projects, starting in 2021