Seungjae (Jay) Lee

I am a Ph.D. student in the Department of Computer Science 💻 at UMD, where I am fortunate to be co-advised by Prof. Furong Huang and Prof. Jia-Bin Huang.

Prior to joining UMD, I earned my Master's degree in the Department of Aerospace Engineering ✈️ at SNU, where I was fortunate to be advised by Prof. H. Jin Kim. I also had the privilege of working at the Generalizable Robotics and AI Lab (GRAIL) 🤖 at NYU under the guidance of Prof. Lerrel Pinto.

Before that, I received Bachelor's degrees in Mechanical and Aerospace Engineering at SNU ⚙️.

"💻 + ✈️ + 🤖 + ⚙️ = Me"

profile photo

Education & Affiliations

Ph.D. in Computer Science

Advised by Professor Furong Huang and Professor Jia-Bin Huang.

Aug 2024 - Present | College Park, MD

Visiting Research

Advised by Professor Lerrel Pinto.

Jul 2023 - Jun 2024 | New York, NY

M.S. in Aerospace Engineering

Advised by Professor H. Jin Kim.

Mar 2021 - Feb 2024 | Seoul, Korea

B.S. in Mechanical & Aerospace Engineering

Mar 2015 - Feb 2021 | Seoul, Korea

Research

My research interest is understanding the interaction between agents and environments, and devising data-efficient decision-making (or robot learning) algorithms, especially in the field of reinforcement learning (RL). Selected publications are marked with ★ below.

TraceGen: World Modeling in 3D Trace-Space Enables Learning from Cross-Embodiment Videos
Seungjae Lee*, Yoonkyo Jung*, Inkook Chun*, Yao-Chih Lee, Zikui Cai, Hongjia Huang, Aayush Talreja, Tan Dat Dao, Yongyuan Liang, Jia-Bin Huang, Furong Huang (*equal contribution)
Under review

project website / arXiv

Sequential-EQA: A Memory-Centric Benchmark for Embodied VQA
Zikui Cai, Shivin Dass, Seungjae Lee, Mingyo Seo, Kaushal Janga, Aadi Palnitkar, Tan Dat Dao, Ruchit Rawal, Mintong Kang, Ruijie Zheng, Kaiyu Yue, Bo Li, Yuke Zhu, Roberto Martín-Martín, Tom Goldstein, Furong Huang

Under review
MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planning
Yuanchen Ju, Yongyuan Liang, Yen-Jen Wang, Nandiraju Gireesh, Yuanliang Ju, Seungjae Lee, Qiao Gu, Elvis Hsieh, Furong Huang, Koushil Sreenath
Under review

project website / arXiv

Dynamic Test-Time Compute Scaling in Control Policy: Difficulty-Aware Stochastic Interpolant Policy
Inkook Chun, Seungjae Lee, Michael Samuel Albergo, Saining Xie, Eric Vanden-Eijnden
NeurIPS, 2025
arXiv

Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models
Seungjae Lee*, Daniel Ekpo*, Haowen Liu, Furong Huang†, Abhinav Shrivastava†, Jia-Bin Huang† (*equal contribution, †equal advising)
CoRL, 2025
project website / arXiv

Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysis
Jeffrey Yang Fan Chiang*, Seungjae Lee*, Jia-Bin Huang, Furong Huang, Yizheng Chen (*equal contribution)
ICLRw, 2025
project website / arXiv

Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments
Haritheja Etukuru, Norihito Naka, Zijin Hu, Seungjae Lee, Julian Mehu, Aaron Edsinger, Chris Paxton, Soumith Chintala, Lerrel Pinto, Nur Muhammad Mahi Shafiullah
ICRA, 2025
project website/ paper/ github
CoRL 2024 Workshop on Language and Robot Learning (🏆 Oral)

Behavior Generation with Latent Actions
Seungjae Lee, Yibin Wang, Haritheja Etukuru, H. Jin Kim, Nur Muhammad Mahi Shafiullah, Lerrel Pinto
ICML, 2024 (🏆 Spotlight, Top 3.5%)
project website / arXiv / github / 🤗 Lerobot Library
RSS 2024 Workshop SemRob (🏆 Oral spotlights) ICML 2024 Workshop MFM-EAI (🏆️ Outstanding Paper Award - Winner)

CQM: Curriculum Reinforcement Learning with a Quantized World Model
Seungjae Lee, Daesol Cho, Jonghae Park, H Jin Kim
NeurIPS, 2023
arXiv

Diversify & Conquer: Outcome-directed Curriculum RL via Out-of-Distribution Disagreement
Daesol Cho, Seungjae Lee, H Jin Kim
NeurIPS, 2023
arXiv

SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learning
Dongseok Shim*, Seungjae Lee*, H Jin Kim (*equal contribution)
ICML, 2023
arXiv / github

Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generation
Daesol Cho*, Seungjae Lee*, H Jin Kim (*equal contribution)
ICLR, 2023 (🏆 Spotlight, Top 5.65%)
arXiv / github

Deep End-to-End Imitation Learning for Missile Guidance with Infrared Images
Seungjae Lee, Jongho Shin, Hyeong-Geun Kim, Daesol Cho, H. Jin Kim
IJCAS, 2023

DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning
Seungjae Lee, Jigang Kim, Inkyu Jang, H. Jin Kim
NeurIPS, 2022 (🏆 Oral, Top 1.76%)
arXiv / github

Robust and Recursively Feasible Real-Time Trajectory Planning in Unknown Environments
Inkyu Jang, Dongjae Lee, Seungjae Lee, H Jin Kim

IROS, 2021 arXiv

Experiences

Toyota Research Institute

Large Behavior Model Team Intern

May 2025 - Aug 2025 | Boston, MA

Samsung Electronics

Deep Learning Algorithm Team Intern

Jul 2020 - Sep 2020 | Gyunggi-do, Korea

Deepest

Sep 2020 - Feb 2022 | Seoul, Korea

Awards and Achievements

Academic Services