Seungjae (Jay) Lee

이승재

I am a Ph.D. student in the Department of Computer Science at UMD, where I am fortunate to be co-advised by Dr. Furong Huang and Dr. Jia-Bin Huang. I am also a Research Intern at NVIDIA GEAR Lab, where I have the pleasure of working with Dr. Jim Fan, Dr. Danfei Xu, and Dr. Ruijie Zheng.

Prior to joining UMD, I earned my Master's degree in the Department of Aerospace Engineering at SNU, where I was fortunate to be advised by Dr. H. Jin Kim. I also had the wonderful opportunity to work on robot learning with Dr. Lerrel Pinto of NYU.

Before that, I received Bachelor's degrees in Mechanical and Aerospace Engineering at SNU.

Seungjae Lee, holding teleoperation grippers beside a robot arm

News

Education & Experiences

Education

Ph.D. in Computer Science
University of Maryland

Advised by Professor Furong Huang and Professor Jia-Bin Huang.

Aug 2024 - Present | College Park, MD

M.S. in Aerospace Engineering
Seoul National University

Advised by Professor H. Jin Kim.

Mar 2021 - Feb 2024 | Seoul, Korea

B.S. in Mechanical & Aerospace Engineering
Seoul National University

Mar 2015 - Feb 2021 | Seoul, Korea

Experiences

NVIDIA

GEAR Team Research Intern

May 2026 - | Santa Clara, CA

Toyota Research Institute

Large Behavior Model Team Research Intern

May 2025 - Aug 2025 | Cambridge, MA

New York University

Research collaborator (Remote), advised by Professor Lerrel Pinto.

Jul 2023 - Jun 2024 | Seoul, Korea (Remote)

Samsung Electronics

Deep Learning Algorithm Team Intern

Jul 2020 - Sep 2020 | Gyunggi-do, Korea

Research

My research interest is understanding the interaction between agents and environments, and devising data-efficient decision-making (or robot learning) algorithms. Selected publications are marked with ★ below.

2026

arXiv2026

HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos
Zhi Wang, Botao He, Kelin Yu, Seungjae Lee, Ruohan Gao, Furong Huang, Yiannis Aloimonos
Under review
project website / arXiv / github

NeurIPS2026

DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
Jusuk Lee, Seungjae Lee, Jonghun Shin, Hoseong Jung, Sungha Kim, Daesol Cho, H. Jin Kim, Jia-Bin Huang†, Furong Huang† (†equal advising)
project website / arXiv

CoRL2026

μ0: A Scalable 3D Interaction-Trace World Model
Seungjae Lee*, Yoonkyo Jung*, Jusuk Lee, Jonghun Shin, Amir Hossein Shahidzadeh, Yao-Chih Lee, H. Jin Kim, Jia-Bin Huang†, Furong Huang† (*equal contribution, †equal advising)
project website / arXiv
IROS 2026 Workshop RoBoWoMo (🏆 Spotlight)

CoRL2026

TAPESTRY: A Single Backbone for Video Generation and Robot Control
Jeremy A. Collins, Seungjae Lee, Rachanon Wachakorn, Krishnan Srinivasan, Animesh Garg, Vitor Campagnolo Guizilini, Paarth Shah
project website

IROS2026

Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering
Zikui Cai*, Kaushal Janga*, Tan Dat Dao*, Seungjae Lee*, Shivin Dass, Mingyo Seo, Kaiyu Yue, Mintong Kang, Nandhu Pillai, Monte Hoover, Aadi Palnitkar, Ruchit Rawal, Ruijie Zheng, Bo Li, Yuke Zhu, Roberto Martín-Martín, Tom Goldstein, Furong Huang
arXiv

CVPR2026

TraceGen: World Modeling in 3D Trace-Space Enables Learning from Cross-Embodiment Videos
Seungjae Lee*, Yoonkyo Jung*, Inkook Chun*, Yao-Chih Lee, Zikui Cai, Hongjia Huang, Aayush Talreja, Tan Dat Dao, Yongyuan Liang, Jia-Bin Huang, Furong Huang (*equal contribution)
project website / arXiv / github

ICLR2026 🏆 Oral

MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planning
Yuanchen Ju, Yongyuan Liang, Yen-Jen Wang, Nandiraju Gireesh, Yuanliang Ju, Seungjae Lee, Qiao Gu, Elvis Hsieh, Furong Huang, Koushil Sreenath
project website / arXiv

2025

NeurIPS2025

Dynamic Test-Time Compute Scaling in Control Policy: Difficulty-Aware Stochastic Interpolant Policy
Inkook Chun, Seungjae Lee, Michael Samuel Albergo, Saining Xie, Eric Vanden-Eijnden
arXiv / github

CoRL2025

Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models
Seungjae Lee*, Daniel Ekpo*, Haowen Liu, Furong Huang†, Abhinav Shrivastava†, Jia-Bin Huang† (*equal contribution, †equal advising)
project website / arXiv

ICLRw2025

Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysis
Jeffrey Yang Fan Chiang*, Seungjae Lee*, Jia-Bin Huang, Furong Huang, Yizheng Chen (*equal contribution)
project website / arXiv

ICRA2025

Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments
Haritheja Etukuru, Norihito Naka, Zijin Hu, Seungjae Lee, Julian Mehu, Aaron Edsinger, Chris Paxton, Soumith Chintala, Lerrel Pinto, Nur Muhammad Mahi Shafiullah
project website/ paper/ github
CoRL 2024 Workshop on Language and Robot Learning (🏆 Oral)

2024

ICML2024 🏆 Spotlight, Top 3.5%

Behavior Generation with Latent Actions
Seungjae Lee, Yibin Wang, Haritheja Etukuru, H. Jin Kim, Nur Muhammad Mahi Shafiullah, Lerrel Pinto
project website / arXiv / github / 🤗 Lerobot Library
RSS 2024 Workshop SemRob (🏆 Oral spotlights) ICML 2024 Workshop MFM-EAI (🏆️ Outstanding Paper Award - Winner)

2023

NeurIPS2023

CQM: Curriculum Reinforcement Learning with a Quantized World Model
Seungjae Lee, Daesol Cho, Jonghae Park, H Jin Kim
arXiv

NeurIPS2023

Diversify & Conquer: Outcome-directed Curriculum RL via Out-of-Distribution Disagreement
Daesol Cho, Seungjae Lee, H Jin Kim
arXiv

ICML2023

SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learning
Dongseok Shim*, Seungjae Lee*, H Jin Kim (*equal contribution)
arXiv / github

ICLR2023 🏆 Spotlight, Top 5.65%

Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generation
Daesol Cho*, Seungjae Lee*, H Jin Kim (*equal contribution)
arXiv / github

IJCAS2023

Deep End-to-End Imitation Learning for Missile Guidance with Infrared Images
Seungjae Lee, Jongho Shin, Hyeong-Geun Kim, Daesol Cho, H. Jin Kim

2022

NeurIPS2022 🏆 Oral, Top 1.76%

DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning
Seungjae Lee, Jigang Kim, Inkyu Jang, H. Jin Kim
arXiv / github

2021

IROS2021

Robust and Recursively Feasible Real-Time Trajectory Planning in Unknown Environments
Inkyu Jang, Dongjae Lee, Seungjae Lee, H Jin Kim
arXiv

Code

huggingface/lerobot 27.8k Making AI for robotics more accessible with end-to-end learning vq_bet_official 219 Behavior Generation with Latent Actions — ICML 2024 Spotlight TraceGen 63 World modeling in 3D trace-space from cross-embodiment videos — CVPR 2026 Yoonkyo/mu0 53 A scalable 3D interaction-trace world model — CoRL 2026 dhrl_official 34 Graph-based long-horizon, sparse hierarchical RL — NeurIPS 2022 Oral snerl_official 32 Semantic-aware neural radiance fields for RL — ICML 2023 outpace_official 12 Curriculum goal generation for outcome-directed RL — ICLR 2023 Spotlight imagine_verify_execute 3 Memory-guided agentic exploration with VLMs — CoRL 2025

Awards and Achievements

Academic Services

Teaching