Projects
Introduction:
Here are my previous academic and professional projects, which has these focused areas:
VR/AR
Machine Learning
Game
Quantitative Analysis
Software Engineering
Robotics
★ XRT: Geo-based Mobile AR Museum Platform
Technical Producer, Lead Programmer
#AR #Geo #Unity #Google Firebase
Cooperated with Andy Warhol Museum and CMU-HCII
Spring 2023
The project originally aims at iteratively design, prototype, and evaluate a virtual immersive gallery with marginalized Pittsburgh artists, which will be physically and virtually centered at the Andy Warhol Museum Pop District. However after iterations, we see much more meaningfulness to promote the idea to the general public.
Supported by Google AR Core and Firebase features, the project allows the real-time, multi-user museum experience, both for content creators and viewers. Our product also serves as a platform to strengthen the connection in art community.
Final Demo (Credit to: Jiayu)
★ Branching Narrative Editing Tool in Roblox
Producer, Lead Programmer
CMU academic project cooperated with Roblox
Fall 2022
This is a Roblox Education sponsored project, aiming to discover the potential of Roblox in education area. In the project, we learnt from Twine - a widely used narrative building tool, and deliver a similar application that supports real-time editing and story building within Roblox, the leading content creation platform.
As the lead programmer and producer, I am responsible for the key function development: dialogue system, and implemented the frontend and backend including server-side programming with Lua. We also communicate with the Roblox, participating in weekly meetings and reporting on progress during the delivery timeline of 13 weeks.
Final Demo
★ Machine Learning Supportive UI Tools
Unity Technologies, Machine Learning Engineer Intern
ML Artistry Team, San Francisco
Summer 2022
Along with the tremendous growth of virtual worlds in games and movies, scene arranging has always been a time-consuming issue that takes a group of technical artists months to complete a high-quality scene.
Our goal is to implement optimized layout(set-dressing) tools that greatly boost this process, by providing real-time prototype solution, including the placement tool, the physic-based transform tool, and the intelligent assets palette. The project is also compatible with future XR devices interaction since it's in Unity platform. Meanwhile, I align with team regularly about the machine learning techniques in scene editing, including terrain generation and smart brushes in the natural scene playground.
As a Machine Learning Engineer Intern, I started by collecting critical needs and started by making system designs, then followed by prototyping UI features and implemented them. Also we experimented with different physics methods for one of the UI features as well.
Usage Demo (Part)
Asset Explorer
Physic tools
★ WebGL Self-driving Car Demonstrating AI Concepts
Lead Programmer
Collaborated with CMU Machine Learning & Computer Sciences Departments
Spring 2022
This is a project whose client is Boys and Girls Club of America and is currently pushed in use.
trAIn gAIn is an interactive WebGL game to assist 14-16-year-old AIPI (Artificial Intelligence Pathways Institute) students understand the concepts of neural networks through interactive visualization of the self-driving car.
Scene Art
Flyers
★ 16-726: Learning-Based Image Synthesis
This course taught by Jun-Yan Zhu start from Computer Vision basics and then go further into content generation with machine learning methods. The primary focus for this course is Generative Adversial Network(GAN), but it take advantage of deep learning methods and digital image processing as well.
See the course page and my project writeups: proj1 | proj2 | proj3 | proj4
★ Meta-human: Motion Capture and Emotion Model
Algorithm Engineer Intern
Lilith Games, Summer 2021
Our algorithm team researched making the automatic workflow from a normal video containing human dancing to an avatar with the same body movement.
Besides, we have implemented a virtual pet prototype with RESTful APIs backend by Python Flask.
★ Cotton Picking Robot
Research Assistant
ShanghaiTech School Lab & Xinjiang Esquel, 2020-2021
After college graduation, I worked as Research Assistant in the SMART Lab at ShanghaiTech, and worked in a research team with Xinjiang Esquel Group. We aim to develop a mobile cotton-picking platform, with the robot arm and precise cotton sense with the 3D camera and computer vision object detection algorithm. My main focus is the suction device on the end of the robot arm. We altogether iterated several versions of the suction device and I coded on the control part of the suction device. The device is a prototype based on Arduino and Raspberry Pi, and is connected to the ROS robot platform. Besides, I helped with developing and deploying detection methods on the robot.
★ Venture Capital Investment Data Analysis
During my senior year in ShanghaiTech, I became a student Research Assistant in School of Entrepreneurship and management. My contribution is to analysis venture capital investment data collected by pitchbook.com and run machine learning methods on them, aiming to find mechanism of how it is related with an important index: Economy Policy Uncertainty Index. Besides, we found this vary between investors depends on their education level, gender and some other personal attributes. This is also my graduation paper as well.
