Ruta Desai

I am a Staff Research Scientist in Fundamental AI Research (FAIR) at Meta. At FAIR, I am exploring reasoning and long-horizon planning problems towards agents that can partner with humans. Prior to that, I was a Tech Lead Manager at Meta Reality Labs Research. At Reality Labs, I led an embodied AI team focused on solving perception and planning problems for Augmented Reality (AR) systems that can extend their users' capabilities. I am broadly interested in Vision and Language and Human-AI collaboration.

I obtained my PhD at the Robotics Institute, Carnegie Mellon University in 2018 working with Stelian Coros and Jim McCann. For my thesis, I devised human-AI systems that enabled casual users to build and program robots — towards increasing accessibility of robotics. In the past, I have also had not-so-brief stints in the area of human motor control and biomechanical simulations with Hartmut Geyer and Jessica K. Hodgins.



Email: rutadesai[at]meta.com

CV  /  Meta AI  /  Scholar

Publications

Embodied AI

PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks
Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Undersander Tsung-Yen Yang

[Under Review] International Conference on Learning Representations (ICLR), Singapore (2025).                         

PDF | Code | Webpage

Learning Human Preferences Over Robot Behavior as Soft Planning Constraints
Austin Narcomey, Nathan Tsoi, Ruta Desai, and Marynel Vazquez

[Under review] 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), Michigan, USA (2025).

arXiv

Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
Xavi Puig*, Eric Undersander*, Andrew Szot*, Mikael Cote*, Ruslan Partsey*, Jimmy Yang*, Ruta Desai*, Alexander Clegg*, Michal Hlavac, Tiffany Min, Theo Gervet, Vladimir Vondrus, Vincent-Pierre Berges, John Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai**, Roozbeh Mottaghi**

International Conference on Learning Representations (ICLR), Vienna, Austria (2024).                         
PDF | Code | Webpage

Adaptive Coordination in Social Embodied Rearrangement
Andrew Szot, Unnat Jain, Dhruv Batra, Zsolt Kira, Ruta Desai, and Akshara Rai

International Conference on Machine Learning (ICML), Hawaii, USA (2023).                         

PDF | Code

Effective Baselines for Multiple Object Rearrangement Planning in Partially Observable Mapped Environments
Engin Tekin, Elaheh Barati, Nitin Kamra, and Ruta Desai

AAAI Workshop on User-Centric AI for Assistance in At-Home Tasks, Washington DC, USA (2023).                         

arXiv | bib

Cross-Domain Imitation Learning via Semantic Skills
Karl Pertsch, Ruta Desai, Franziska Meier, Vikash Kumar, Dhruv Batra, and Akshara Rai

Conference on Robot Learning(CoRL), Aukland, New Zealand (2022).                         

PDF | bib | Video | Webpage

Vision and Language

User-in-the-loop Evaluation of Multimodal LLMs for Activity Assistance
Mrinal Verghese*, Brian Chen*, Hamid Eghbalzadeh, Tushar Nagarajan, and Ruta Desai

IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Arizona, USA. (2025).                         
PDF | Video

Pretrained Language Models as Visual Planners for Human Assistance
Dhruvesh Patel, Hamid Engbalzadeh, Nitin Kamra, Michael Louis Iuzzolino, Unnat Jain, and Ruta Desai

International Conference on Computer Vision (ICCV), Paris, France (2023).                         

PDF | Code

EgoTV: Egocentric Task Verification from Natural Language Task Descriptions
Rishi Hazra, Brian Chen, Akshara Rai, Nitin Kamra, and Ruta Desai

International Conference on Computer Vision (ICCV), Paris, France (2023).                         

PDF | Code | Webpage

Action Dynamics Task Graphs for Learning Plannable Representations of Procedural Tasks
Weichao Mao, Ruta Desai, Michael Louis Iuzzolino, and Nitin Kamra

AAAI Workshop on User-Centric AI for Assistance in At-Home Tasks, Washington DC, USA (2023).                         

arXiv | bib

Egocentric Scene Context for Human-centric Environment Understanding from Video
Tushar Nagarajan, Santhosh Kumar Ramakrishnan, Ruta Desai, James Hillis, and Kristen Grauman

Advances in Neural Information Processing Systems (Neurips), New Orleans, USA (2023).                         
PDF | bib | Webpage

Episodic Memory Question Answering
Samyak Datta, Sameer Dharur, Vincent Cartellier, Ruta Desai, Mukul Khanna, Dhruv Batra, and Devi Parikh

The Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA (2022).                         
PDF | bib | Video

How You Move Your Head Tells What You Do: Self-supervised Video Representation Learning with Egocentric Cameras and IMU Sensors
Satoshi Tsutsui, Ruta Desai, and Karl Ridgeway

International Conference on Computer Vision (ICCV), EPIC Workshop (2021).                         

PDF | bib | Video

Human-AI Interaction

A Meta-Bayesian Approach for Rapid Online Parametric Optimization for Wrist-based Interactions
Yi-Chi Liao, Ruta Desai, Alec M Pierce, Krista E Taylor, Hrvoje Benko, Tanya R Jonker, Aakar Gupta

ACM Conference on Human Factors in Computing Systems (CHI), Hawaii, USA (2024).                         
PDF | Video

Optimizing the Timing of Intelligent Suggestion in Virtual Reality
Difeng Yu, Ruta Desai, Ting Zhang, Hrvoje Benko, Tanya R. Jonker, and Aakar Gupta

ACM User Interface Software and Technology Symposium (UIST), Bend, USA (2022).                         
PDF | bib | Video

Optimal Assistance for Object-Rearrangement Tasks in Augmented Reality
Benjamin Newman, Kevin Carlberg, and Ruta Desai

arXiv (2020).                         

PDF | bib | Video

Towards Inferring Cognitive State Changes from Pupil Size Variations in Real World Conditions
Naga Venkata Kartheek Medathati, Ruta Desai, and James Hillis

ACM Symposium on Eye Tracking Research and Applications (ETRA), Stuttgart, Germany (2020).                         

PDF | bib

Geppetto: Enabling Semantic Design of Expressive Robot Behaviors
Ruta Desai, Fraser Anderson, Justin Matejka, Stelian Coros, James McCann, George Fitzmaurice, and Tovi Grossman

ACM Conference on Human Factors in Computing Systems (CHI), Glasgow, UK (2019).                         
Best paper award (top 1%) [Details]
PDF | bib | Video | Fastforward | Supplementary (PDF)

Assembly-aware Design of Printable Electromechanical Devices
Ruta Desai, James McCann, and Stelian Coros

ACM User Interface Software and Technology Symposium (UIST), Berlin, Germany (2018).                           
PDF | bib | Video | Fastforward

Skaterbots: Optimization-based Design and Motion Synthesis for Robotic Creatures with Legs and Wheels
Moritz Geilinger, Roi Poranne, Ruta Desai, Bernhard Thomaszewski, and Stelian Coros

ACM Transactions on Graphics (Proc. ACM SIGGRAPH), Vancouver, Canada (2018).                           
PDF | bib | Video

Automatic Design of Task-specific Robotic Arms
Ruta Desai, Margarita Safonova, Katharina Muelling, and Stelian Coros

ICRA Workshop on Autonomous Robot Design, Brisbane, Australia (2018).                           

arXiv | bib | Slides | Video

Interactive Co-Design of Form and Function for Legged Robots using the Adjoint Method
Ruta Desai, Beichen Li, Ye Yuan, and Stelian Coros

International Conference on Climbing and Walking Robots (CLAWAR), Panama city, Panama (2018).  

Best technical paper (second place)  
arXiv Preprint | Video

Computational Abstractions for Interactive Design of Robotic Devices
Ruta Desai, Ye Yuan, and Stelian Coros

IEEE International Conference on Robotics and Automation (ICRA), Singapore (2017).                          
PDF | bib | Slides | Video | Code
Robot models for fabrication: car, walking robot


3D Printing Pneumatic Device Controls with Variable Activation Force Capabilities
Marynel Vazquez, Eric Brockmeyer, Ruta Desai, Chris Harrison and Scott E. Hudson

ACM Conference on Human Factors in Computing Systems (CHI), Seoul, Korea, (2015).

PDF | bib | Poster | Video


Human Motion Modeling, Human Motor Control, Biomechanics

A Simple Model of Skill Acquisition in a Dynamic Balance Task
Ruta Desai and Jessica K. Hodgins

Dynamic Walking, Columbus, USA (2015).

PDF | bib | Poster | Video
Virtual Model Control for Dynamic Lateral Balance
Ruta Desai, Hartmut Geyer, and Jessica K. Hodgins

IEEE International Conference on Humanoid Robots (Humanoids), Madrid, Spain (2014).

PDF | bib
Integration of an Adaptive Swing Control into a Neuromuscular Human Walking Model
Seungmoon Song, Ruta Desai, and Hartmut Geyer

IEEE Engineering in Medicine and Biology Society (EMBC), Osaka, Japan (2013).

PDF | bib | Video

Muscle-Reflex Control of Robust Swing Leg Placement
Ruta Desai and Hartmut Geyer

IEEE International Conference on Robotics and Automation (ICRA), Karlsruhe, Germany (2013).

PDF | bib

Robust Swing Leg Placement under Large Disturbances
Ruta Desai and Hartmut Geyer

IEEE International Conference on Robotics and Biomimetics (ROBIO), Guangzhou, China (2012).

PDF | bib | Video

Selected Patents and Preprints

Human Action Anticipation: A Survey
Bolin Lai*, Sam Toyer*, Tushar Nagarajan, Rohit Girdhar, Shengxin Zha, James M. Rehg, Kris Kitani, Kristen Grauman, Ruta Desai, and Miao Liu

arXiv Preprint (2024).                         

Human-Centered Planning
Yuliang Li, Nitin Kamra, Ruta Desai, and Alon Halevy

arXiv Preprint (2023).                         

Optimal Assistance for Object-Rearrangement Tasks in Augmented Reality
Benjamin A. Newman, Kevin T. Carlberg, Ruta Desai, James Hillis

US Patent US20220114366 A1 (2022).

Generative design techniques for robot behavior
Fraser Anderson, Stelian Coros, Ruta Desai, Tovi Grossman, Justin Matejka, George Fitzmaurice

US Patent 20200030988 A1 (2020).                                                                                                                 


Robust swing leg controller under large disturbances
Ruta Desai and Hartmut Geyer

US Patent 20150066156 A1 (2014).                                                                                                                 


A Brief Overview of Human and Robot Motor Learning
Ruta Desai

RI CMU Technical Report                                                                                                                 

Academic Service

Conference Program Committee Chair and Reviewer

  • Program Committee, AAAI (2025).
  • Organizer, CVPR Workshop on Causal and Object-centric Representations for Robotics (2024).
  • Reveiwer for IEEE CVPR, IEEE ICCV, ICLR, Neurips, ICML (2023-2024).
  • Program committee, ACM UIST (2019-2020).
  • Associate Chair (AC), ACM CHI (2020-2021).
  • Reveiwer for IEEE IROS, IEEE ICRA, ACM TOCHI, ACM UIST, ACM CHI, ACM GI, IEEE WHC (2015-2022).

Talks

Outreach

Women in Technology

I am always looking to support and contribute towards encouraging women in science and technology. In the past:

  • I have led roadshows and volunteered in teaching middle school girls at Technights at CMU. We have smashed computers, fought combats in code, and have even gazed stars through apps!
  • I helped organize OurCS 2015, a conference to encourage undergraduate women in research. I was also a part of Google Anita Borg Scholars Alumni Planning committee.

Selected Press

  • Silicon Angle, Meta's Habitat 3.0 simulates real-world environments for intelligent AI robot training, 2023.
  • Techcrunch, Embodied AI spins a pen and helps clean the living room in new research, 2023.
  • Techcrunch, New toolkit makes it easy to drag and drop your own robot, 2017.
  • ACM Communications, Robot Design For Dummies, 2017.
  • EurekAlert, CMU's interactive tool helps novices and experts make custom robots, 2017.
  • NSF, Graduate Student Earns Prestigious Scholarships for Women - Ruta Desai, 2012.
  • CMU SCS, Five SCS Students Named Siebel Scholars, 2012.

Last updated: January 2025

Webpage adapted from Jon Barron.