Divyam Goel

Divyam Goel

Biography

Hi! I am a first year MS in Robotics student at Robotics Institute, Carnegie Mellon University. I am fortunate to be co-advised by Prof. Zackory Erickson and Prof. David Held.

Prior to coming to CMU, I spent two wonderful years working on text-to-video generation models at Adobe and Rephrase.ai (now part of Adobe). Concurrently, I was a research assistant at the 3D Language Group at SFU working with Prof. Angel Chang. Even earlier, I received a B.Tech in E&CE from IIT Roorkee.

My research interests are primarily in computer vision and robot learning, with forays in natural language processing.

Interests

  • Computer Vision
  • Robot Learning
  • Robotics
  • Probabilistic Graphical Models

Education

  • MS in Robotics, 2026

    Carnegie Mellon University

  • B.Tech. in Electronics and Communications Engineering, 2022

    Indian Institute of Technology Roorkee

News

  • Aug 2024 Headed to Carnegie Mellon University for my graduate studies in Robotics!
  • Mar 2024 Paper titled “Semi‑NMF Regularization‑Based Autoencoder Training for Hyperspectral Unmixing” accepted at NCC 2024.
  • Nov 2023 Rephrase AI gets acquired by Adobe! I will be joining Adobe - DVA as a Machine Learning Engineer following the acquisition.
  • June 2023 Joined Prof. Angel X. Chang’s lab at SFU as a Research Assistant.
  • Apr 2023 Promoted to Deep Learning Researcher at Rephrase AI!
  • June 2022 Joined Rephrase AI as a Deep Learning Junior Researcher!
  • June 2022 Paper titled “Language Guided Meta-Control for Embodied Instruction Following” accepted at the Embodied AI Workshop @ CVPR 2022! Preprint coming soon.
  • May 2022 Paper titled “Leveraging Dependency Grammar for Fine-Grained Offensive Language Detection using Graph Convolutional Networks” accepted at SocialNLP@NAACL 2022! Preprint coming soon.
  • Mar 2022: Paper “On the Cross-Modal Transfer from Natural Language to Code through Adapter Modules” accepted at ICPC 2022! Preprint coming soon.
  • Feb. 2022: Recipient of the IIT Roorkee Heritage Foundation Excellence Award for 2021!
  • July 2021: Joined Prof. Jonghyun Choi’s GIST Computer Vision Lab, South Korea (virtually) as a Research Assistant.
  • June 2021: Headed to Prof. Fatemeh H. Fard’s lab at UBC, Canada (virtually) to work as a Mitacs Globalink Research Intern.
  • Dec. 2020: Selected for the prestigious MITACS Globalink Research Internship 2021!!
  • July 2020: Will be working at PandoCorp as a research intern this fall in their machine learning research lab - Pando Labs.
  • Mar 2020: Recipient of the IIT Roorkee Heritage Foundation Excellence Award for 2019!
  • Apr 2019: Selected for the Quantum Computing summer training organized by QuLabs.

Recent Publications

Quickly discover relevant content by filtering publications.

Semi‑NMF Regularization‑Based Autoencoder Training for Hyperspectral Unmixing

30th National Conference on Communications (NCC'24)
PDF

Language Guided Meta Control for Embodied Instruction Following

Embodied AI Workshop @ CVPR 2022. The 2nd Embodied AI Workshop (EAI’ 22)
PDF Poster
Language Guided Meta Control for Embodied Instruction Following

Leveraging Dependency Grammar for Fine-Grained Offensive Language Detection using Graph Convolutional Networks

SocialNLP @ NAACL 2022. The 10th International Workshop on Natural Language Processing for Social Media (SocialNLP ‘22)
PDF Code
Leveraging Dependency Grammar for Fine-Grained Offensive Language Detection using Graph Convolutional Networks

On the Cross Modal Transfer From Natural Language to Code Through Adapter Modules

ICPC 2022. 30th International Conference on Program Comprehension (ICPC ‘22)
PDF Code Dataset Project
On the Cross Modal Transfer From Natural Language to Code Through Adapter Modules

Experience

 
 
 
 
 

Machine Learning Engineer - L2

Adobe

Nov 2023 – Jul 2024 Bangalore, India
  • Worked on the research and development of Adobe’s family of GenAI models - Firefly.
  • Designed sampling methods, using Sequential Monte Carlo methods and Langevin Dynamics to generate highly-accurate sampled from large pre-trained diffusion models for several downstream tasks.
 
 
 
 
 

Research Assistant

3D Language Group, Simon Fraser University

Jun 2023 – Jul 2024 Burnaby, Canada (virtually)
  • Worked on the research and development of generative models for the task of 3D scene generation.
  • Submitted a work proposing a 3D scene layout generation framework (SemLayoutDiff) using a denoising diffusion probabilistic model along with an attribute prediction network to generate semantic layouts using the 3D-FRONT dataset.
  • Designed novel cost functions to implicitly impose ergonomic design constraints within the learned space of a diffusion model for the task of 3D scene layout generation.
 
 
 
 
 

Deep Learning Researcher

Rephrase AI

Jun 2022 – Nov 2023 Bangalore, India
  • My research focused on (tractable) generative models for speech AI and computer vision.
  • Enriched Rephrase.ai’s proprietary video generation pipeline, expanding its global outreach by enhancing avatar capabilities for multilingual applications.
  • Collaborated closely with DevOps engineers to efficiently deploy models on AWS, optimizing the generation pipeline to enable the creation of over 100K personalized videos daily.
  • Spearheaded a Text‑to‑Speech initiative, developing a voice cloning system from limited speech data. Also devised a speech restoration pipeline to extract clean speech audios of the source speaker from extremely low‑quality / noisy inputs.
 
 
 
 
 

Research Assistant

GIST Computer Vision Lab

Jul 2021 – Dec 2022 Gwangju, South Korea (virtually)
  • Published 1 research paper in EmbodiedAI at CVPR 2022 and submitted 1 paper at a top tier conference.
  • Released a work proposing a language‑guided meta‑controller to learn robust task‑agnostic representations, and an auxiliary reasoning loss to improve the overall cross‑modal grounding capabilities for embodied instruction following.
  • Submitted a work proposing a story‑visualization framework (SMART) using a multi‑stage multi‑modal transformer with in‑memory spatio‑temporal context, and a discretized variational autoencoder.
 
 
 
 
 

MITACS Globalink Research Intern

University of British Columbia

Jun 2021 – Sep 2021 Kelowna, Canada (virtually)
  • Published and presented a research paper at ICPC 2022.
  • Achieved 140× better parameter budget and ∼95% efficient storage in adapting foundational LLMs to source code.
  • Built the Super Code Clone Detection-88 (SCD-88) dataset to evaluate the proposed approach on python-specific code clone detection.
 
 
 
 
 

Research and Development Intern

Pando Labs - PandoCorp PLC

Aug 2020 – Nov 2020 Chennai, India
  • I worked on developing a practical Deep Reinforcement Learning solution to optimize shipping costs in the Supply Chain industry.
  • Achieved ~85% packing efficiency in 3D bin packing and ~50% cost efficiency in capacitative vehicle routing over existing solutions using off-policy agents in simulated environments.
  • Re-architected a heuristic algorithm to improve efficiency and allow parallelised operations on GPUs.
 
 
 
 
 

Undergraduate Student Researcher

Machine Vision Lab

Dec 2019 – Mar 2020 IIT Roorkee
  • Worked under the supervision of Prof. Balasubramanian Raman on a project titled - Medical Image Segmentation and Classification Problems.

Achievements

IIT Roorkee Heritage Foundation Excellence Award 2021

IIT Roorkee Heritage Foundation Feb 2022
  • Awarded to 45 students from a pool of over 5000 undergraduates.
  • The Excellence Awards recognize students for outstanding academic, co-curricular and extra-curricular achievements.
See certificate

MITACS Globalink Research Internship

MITACS Apr 2021
  • Selected for the prestigious MITACS Globalink Research Internship 2021 program.
  • The Mitacs-SICI partnership awards this research grant to top international undergraduates for a research internship under the supervision of Canadian university faculty members.
See certificate

IIT Roorkee Heritage Foundation Excellence Award 2019

IIT Roorkee Heritage Foundation Mar 2020
  • Awarded to 45 students from a pool of over 5000 undergraduates.
  • The Excellence Awards recognize students for outstanding academic, co-curricular and extra-curricular achievements.
See certificate

Quantum Computing Summer Training

QuLabs Apr 2019
  • Among the 15 students selected for the Quantum Computing Summer Training - 2019.
  • The course comprised of intensive lectures on linear algebra, quantum mechanics and topics in quantum computation including entanglement, multi-qubit computation, quantum key distribution and quantum algorithms.

Featured Projects

*
All Computer Vision Robotics RL Datasets

SemiNMF-Autoencoders

PyTorch code for experiments with a semi-NMF based regularization objective over multiple autoencoder architectures for hyperspectral unmixing.
CODE

Super Code Clone Detection - 88

A dataset consisting of 11,400 python codes for a retrieval based code clone detection.
LINK

HyperspecAE

PyTorch reproduction of the paper titled ‘Hyperspectral Unmixing Using a Neural Network Autoencoder’ (Palsson et al. 2018).
CODE

Policy Gradient Methods

Implemented some Policy Gradient RL algorithms (using PyTorch) on the CartPole environment by OpenAI.
CODE

Self Driving Car

Designed an autonomous vehicle using a combination of image processing and IoT methods.
CODE DEMO