Seungjae (Jay) Lee

I am first-year Ph.D. student at UMD Department of Computer Science 💻, co-advised by professors Furong Huang and Jia-Bin Huang.

Prior to UMD, I had my Masters Degree in Department of Aerospace Engineering ✈️ at SNU advised by Prof. H. Jin Kim. I also spent time at Generalizable Robotics and AI Lab (GRAIL) 🤖 at NYU, advised by Prof. Lerrel Pinto. I worked on enhancing the data efficiency of Reinforcement Learning (RL) and Imitation Learning (IL) systems and applied them to various decision-making scenarios, including real-world robots.

Before that, I received Bachelor's degrees in Mechanical and Aerospace Engineering at SNU ⚙️.

"💻 + ✈️ + 🤖 + ⚙️ = Me"

profile photo

Education & Affiliations

Ph.D. in Computer Science

Advised by Professor Furong Huang and Professor Jia-Bin Huang.

Aug 2024 - Present | College Park, MD

Visiting Research

Advised by Professor Lerrel Pinto.

Jul 2023 - Jun 2024 | New York, NY

M.S. in Aerospace Engineering

Advised by Professor H. Jin Kim.

Mar 2021 - Feb 2024 | Seoul, Korea

B.S. in Mechanical & Aerospace Engineering

Mar 2015 - Feb 2021 | Seoul, Korea

Research

My research interest is understanding the interaction between agents and environments, and devising data-efficient decision-making (or robot learning) algorithms, especially in the field of reinforcement learning (RL).

Sequential-EQA: A Memory-Centric Benchmark for Embodied VQA
Zikui Cai, Shivin Dass, Seungjae Lee, Mingyo Seo, Kaushal Janga, Aadi Palnitkar, Tan Dat Dao, Ruchit Rawal, Mintong Kang, Ruijie Zheng, Kaiyu Yue, Bo Li, Yuke Zhu, Roberto Martín-Martín, Tom Goldstein, Furong Huang
Under review
Dynamic Test-Time Compute Scaling in Control Policy: Difficulty-Aware Stochastic Interpolant Policy
Inkook Chun, Seungjae Lee, Michael Samuel Albergo, Saining Xie, Eric Vanden-Eijnden
NeurIPS, 2025 (Acceptance Rate: 24.52%)
Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models
Seungjae Lee*, Daniel Ekpo*, Haowen Liu, Furong Huang†, Abhinav Shrivastava†, Jia-Bin Huang†
CoRL, 2025 (Acceptance Rate: 35.77%)
(*equal contribution, †equal advising)
project website / arXiv

Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysis
Jeffrey Yang Fan Chiang*, Seungjae Lee*, Jia-Bin Huang, Furong Huang, Yizheng Chen
ICLRw, 2025
(*equal contribution)
+ ICLR 2025 Workshop Building Trust in Language Models and Applications project website / arXiv

Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments
Haritheja Etukuru, Norihito Naka, Zijin Hu, Seungjae Lee, Julian Mehu, Aaron Edsinger, Chris Paxton, Soumith Chintala, Lerrel Pinto, Nur Muhammad Mahi Shafiullah
ICRA, 2025
+ CoRL 2024 Workshop on Language and Robot Learning, "Oral"
project website/ paper/ github

Behavior Generation with Latent Actions
Seungjae Lee, Yibin Wang, Haritheja Etukuru, H. Jin Kim, Nur Muhammad Mahi Shafiullah, Lerrel Pinto
ICML, 2024 (Spotlight) (Top: 3.5%)
+ RSS 2024 Workshop SemRob, "Oral spotlights"
+ ICML 2024 Workshop MFM-EAI, "Outstanding Paper Award - Winner"

project website/ arXiv/ github/ 🤗 Lerobot Library

CQM: Curriculum Reinforcement Learning with a Quantized World Model
Seungjae Lee, Daesol Cho, Jonghae Park, H Jin Kim
NeurIPS, 2023 (Acceptance Rate: 26.07%)
arXiv

Diversify & Conquer: Outcome-directed Curriculum RL via Out-of-Distribution Disagreement
Daesol Cho, Seungjae Lee, H Jin Kim
NeurIPS, 2023 (Acceptance Rate: 26.07%)
arXiv

SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learning
Dongseok Shim*, Seungjae Lee*, H Jin Kim
(*equal contribution)
ICML, 2023 (Acceptance Rate: 27.96%)
arXiv / github

Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generation
Daesol Cho*, Seungjae Lee*, H Jin Kim
(*equal contribution)
ICLR, 2023 (Spotlight) (Top: 5.65%)
arXiv / github

Deep End-to-End Imitation Learning for Missile Guidance with Infrared Images
Seungjae Lee, Jongho Shin, Hyeong-Geun Kim, Daesol Cho, H. Jin Kim
IJCAS, 2023

DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning
Seungjae Lee, Jigang Kim, Inkyu Jang, H. Jin Kim
NeurIPS, 2022 (Oral) (Top: 1.76%)
arXiv / github

Robust and Recursively Feasible Real-Time Trajectory Planning in Unknown Environments
Inkyu Jang, Dongjae Lee, Seungjae Lee, H Jin Kim
IROS, 2021
arXiv

Projects

Training Excavator Virtual Driver based on Inverse RL

with HD Hyundai Heavy Industries Co., Ltd.
Apr. 2023 - Mar. 2024

End-to-End Machine Learning Based Guidance Research
with Korean Agency for Defense Development (ADD)
May. 2021 - Apr. 2023

Experiences

Toyota Research Institute

Large Behavior Model Team Intern

May 2025 - Aug 2025 | Boston, MA

Samsung Electronics

Deep Learning Algorithm Team Intern

Jul 2020 - Sep 2020 | Gyunggi-do, Korea

Deepest

Sep 2020 - Feb 2022 | Seoul, Korea

GitHub Projects

DHRL Repository VQ-BeT Repository
OUTPACE Repository SNeRL Repository
View All Projects on GitHub →

Awards and Achievements

Academic Services