Seungjae (Jay) Lee

I am a Ph.D. student in the Department of Computer Science 💻 at UMD, where I am fortunate to be co-advised by Prof. Furong Huang and Prof. Jia-Bin Huang.

Prior to joining UMD, I earned my Master's degree in the Department of Aerospace Engineering ✈️ at SNU, where I was fortunate to be advised by Prof. H. Jin Kim. I also had the privilege of working at the Generalizable Robotics and AI Lab (GRAIL) 🤖 at NYU under the guidance of Prof. Lerrel Pinto.

Before that, I received Bachelor's degrees in Mechanical and Aerospace Engineering at SNU ⚙️.

"💻 + ✈️ + 🤖 + ⚙️ = Me"

profile photo

Education & Affiliations

Ph.D. in Computer Science

Advised by Professor Furong Huang and Professor Jia-Bin Huang.

Aug 2024 - Present | College Park, MD

Visiting Research

Advised by Professor Lerrel Pinto.

Jul 2023 - Jun 2024 | New York, NY

M.S. in Aerospace Engineering

Advised by Professor H. Jin Kim.

Mar 2021 - Feb 2024 | Seoul, Korea

B.S. in Mechanical & Aerospace Engineering

Mar 2015 - Feb 2021 | Seoul, Korea

Research

My research interest is understanding the interaction between agents and environments, and devising data-efficient decision-making (or robot learning) algorithms, especially in the field of reinforcement learning (RL).

TraceGen: World Modeling in 3D Trace-Space Enables Learning from Cross-Embodiment Videos
Seungjae Lee*, Yoonkyo Jung*, Inkook Chun*, Yao-Chih Lee, Zikui Cai, Hongjia Huang, Aayush Talreja, Tan Dat Dao, Yongyuan Liang, Jia-Bin Huang, Furong Huang
Under review
(*equal contribution)
project website / arXiv

Sequential-EQA: A Memory-Centric Benchmark for Embodied VQA
Zikui Cai, Shivin Dass, Seungjae Lee, Mingyo Seo, Kaushal Janga, Aadi Palnitkar, Tan Dat Dao, Ruchit Rawal, Mintong Kang, Ruijie Zheng, Kaiyu Yue, Bo Li, Yuke Zhu, Roberto Martín-Martín, Tom Goldstein, Furong Huang
Under review
Dynamic Test-Time Compute Scaling in Control Policy: Difficulty-Aware Stochastic Interpolant Policy
Inkook Chun, Seungjae Lee, Michael Samuel Albergo, Saining Xie, Eric Vanden-Eijnden
NeurIPS, 2025 (Acceptance Rate: 24.52%)
Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models
Seungjae Lee*, Daniel Ekpo*, Haowen Liu, Furong Huang†, Abhinav Shrivastava†, Jia-Bin Huang†
CoRL, 2025 (Acceptance Rate: 35.77%)
(*equal contribution, †equal advising)
project website / arXiv

Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysis
Jeffrey Yang Fan Chiang*, Seungjae Lee*, Jia-Bin Huang, Furong Huang, Yizheng Chen
ICLRw, 2025
(*equal contribution)
+ ICLR 2025 Workshop Building Trust in Language Models and Applications project website / arXiv

Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments
Haritheja Etukuru, Norihito Naka, Zijin Hu, Seungjae Lee, Julian Mehu, Aaron Edsinger, Chris Paxton, Soumith Chintala, Lerrel Pinto, Nur Muhammad Mahi Shafiullah
ICRA, 2025
+ CoRL 2024 Workshop on Language and Robot Learning, "Oral"
project website/ paper/ github

Behavior Generation with Latent Actions
Seungjae Lee, Yibin Wang, Haritheja Etukuru, H. Jin Kim, Nur Muhammad Mahi Shafiullah, Lerrel Pinto
ICML, 2024 (Spotlight) (Top: 3.5%)
+ RSS 2024 Workshop SemRob, "Oral spotlights"
+ ICML 2024 Workshop MFM-EAI, "Outstanding Paper Award - Winner"

project website/ arXiv/ github/ 🤗 Lerobot Library

CQM: Curriculum Reinforcement Learning with a Quantized World Model
Seungjae Lee, Daesol Cho, Jonghae Park, H Jin Kim
NeurIPS, 2023 (Acceptance Rate: 26.07%)
arXiv

Diversify & Conquer: Outcome-directed Curriculum RL via Out-of-Distribution Disagreement
Daesol Cho, Seungjae Lee, H Jin Kim
NeurIPS, 2023 (Acceptance Rate: 26.07%)
arXiv

SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learning
Dongseok Shim*, Seungjae Lee*, H Jin Kim
(*equal contribution)
ICML, 2023 (Acceptance Rate: 27.96%)
arXiv / github

Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generation
Daesol Cho*, Seungjae Lee*, H Jin Kim
(*equal contribution)
ICLR, 2023 (Spotlight) (Top: 5.65%)
arXiv / github

Deep End-to-End Imitation Learning for Missile Guidance with Infrared Images
Seungjae Lee, Jongho Shin, Hyeong-Geun Kim, Daesol Cho, H. Jin Kim
IJCAS, 2023

DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning
Seungjae Lee, Jigang Kim, Inkyu Jang, H. Jin Kim
NeurIPS, 2022 (Oral) (Top: 1.76%)
arXiv / github

Robust and Recursively Feasible Real-Time Trajectory Planning in Unknown Environments
Inkyu Jang, Dongjae Lee, Seungjae Lee, H Jin Kim
IROS, 2021
arXiv

Projects

Training Excavator Virtual Driver based on Inverse RL

with HD Hyundai Heavy Industries Co., Ltd.
Apr. 2023 - Mar. 2024

End-to-End Machine Learning Based Guidance Research
with Korean Agency for Defense Development (ADD)
May. 2021 - Apr. 2023

Experiences

Toyota Research Institute

Large Behavior Model Team Intern

May 2025 - Aug 2025 | Boston, MA

Samsung Electronics

Deep Learning Algorithm Team Intern

Jul 2020 - Sep 2020 | Gyunggi-do, Korea

Deepest

Sep 2020 - Feb 2022 | Seoul, Korea

Awards and Achievements

Academic Services