Ryan Yen

HCI Researcher / Entrepreneur / Developer

Human-AI Interactions/ Community-Centered AI/ Computational Theory Systems

I am currently a Human-AI Researcher. In the past, I was a full-stack developer, and I aspire to become an entrepreneur in the future. My present research interest are human/community-AI interaction. Specifically, Designing interactive visualization systems that enhance AI's transparency and controllability for programmers; and Developing social computing systems that can derive insights to lay the groundwork for fostering the programmers-AI community.

I am pursuing my Thesis-based Master's in Computer Science at the University of Waterloo, mentored by Dr. Jian Zhao of the WatVis Lab at UWaterloo. In the past, I was guided by Dr. Zhicong Lu from DEER Lab and Dr. Can Liu of the ERFI Lab at CityU HK. After obtaining my Master's, I will pursue my PhD at MIT CSAIL under the supervision of Dr. Arvind Satyanarayan to continue my research in HAI.

Current State

I am going to present my LBW and Paper at CHI24 & I have accepted my offer from MIT CSAIL for my PhD.

Research Interest
HAI/ Community-Centred AI/ Cogntive Engineering
Hobbies Violin/ Swimming/ Board Game

Latest News

1 Paper + 1LBW at CHI24! 🌺🏄 2024-02-06

We explored the workflow of collaborative natural language programming and designed a system to support prompt sharing and referring.

Paper Accepted by CUI! 🔤🇳🇱2023-02-26

This work explores literature from Cognitive Science to synthesize relevant theories and findings for the HCI audience to reference. Thanks to wonderful supervisor Can Liu and co-authors, Brinda Mehra and Kejia Shen.

Paper Accepted by CHI! ⛵️🇩🇪2023-01-16

I'm coming to the CHI23 conference at Hamburg, German in 2023. I will present my work on the collaboration between human and code generation models. I will also present my work on moderating online social media and fostering the sense of community.

Paper Accepted by CSCW23! 👬2023-01-07

Wizundry is a WoZ platform that allows multiple Wizards to collaboratively operate a speech-to-text based system remotely. Our findings reveal the promises and challenges of the multi-Wizard approach and open up new research questions.

Publications

Upcomming

Reifying the Reuse of User-AI Conversational Memories [pdf] [doi] [video]

Ryan Yen, Jian Zhao

To support users in recalling and reusing relevant user-AI conversational memories, we introduce Memolet, an interactive object that reifies memory reuse. Users can directly manipulate Memolet to specify which memories to reuse and how to use them. We developed a system demonstrating Memolet's interaction across various memory reuse stages.

CHI24 LBW

To Search or To Gen? Exploring the Synergy between Generative AI and Web Search in Programming [pdf] [doi] [video]

Ryan Yen, Nicole Sultanum, Jian Zhao

We have identified three major challenges and proposed three decision-making stages, each with its own relevant factors. Additionally, we present a thorough process model that captures programmers' interaction patterns.

CHI 2024

CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programming [pdf] [doi] [video]

Ryan Yen*, My girlfriend 😚Felicia Feng*, Yuzhe You, Mingming Fan, Jian Zhao, Zhicong Lu ^{*These authors contributed equally to this work.}

A system to support collaborative prompt engineering by providing referring, requesting, sharing, and linking mechanisms. It assists programmers in comprehending collaborators' prompts and building on their collaborators' work, reducing repetitive updates and communication costs.

Arxiv Preprint

CoLadder: Supporting Programmers with Hierarchical Code Generation in Multi-Level Abstraction [pdf] [doi] [video]

Ryan Yen, Jiawen Zhu, Sangho Suh, Haijun Xia, Jian Zhao

A system that assists programmers by enabling hierarchical task decomposition, incremental code generation, and verification of results during prompt authoring. Bridging the abstraction gap between programmers and LLMs.

CHI 2023

StoryChat: Designing a Narrative-Based Viewer Participation Tool for Live Streaming Chatrooms [pdf] [doi] [video]

Ryan Yen, My girlfriend 😚Felicia Feng, Brinda Mehra, Ching Christie Pang, Siying Hu, Zhicong Lu

A narrative-based viewer participation tool that utilizes a dynamic graphical plot to reflect chatroom negativity. We discovered that StoryChat encouraged viewers to contribute prosocial comments, increased viewer engagement, and fostered viewers' sense of community.

CUI 2023

Gist and Verbatim: Understanding Speech to Inform New Interfaces for Verbal Text Composition [pdf] [doi] [video]

Brinda Mehra, Kejia Shen, Ryan Yen, Can Liu

Exploring speech input in HCI, we address editing challenges. Our study combines Cognitive Science with HCI, revealing memory patterns and proposing new interaction concepts for efficient speech editing.

CSCW 2023

Wizundry: A Cooperative Wizard of Oz Platform for Simulating Future Speech-based Interfaces with Multiple Wizards [pdf] [doi] [video]

Siying Hu, Ryan Yen, Ziwei Yu, Mingjian Zhao, Katie Seaborn, Can Liu

A real-time, web-based WoZ platform that allows multiple Wizards to collaboratively operate a speech-to-text based system remotely. Our findings reveal the promises and challenges of the multi-Wizard approach and open up new research questions.

Working Experience (full-time)

AI Research Intern

May 2021 - Jan 2022

Huawei Research Centre

Lung Cancer Detection, CV, AI Explainability in pathology diagnosis
MindSpore Pathology Diagnosis AI Challenge organizer
Online AI training, testing and evaluation platform
Speaker of Training AI Model on Huawei Cloud ModelArts AI Platform
Opening Speaker of Huawei AI challenge
Invited guest to Huawei Cloud Summit 2021
Computer Vision Deep Learning Explainable AI Full-Stack Development

CTO, Co-Founder

Oct 2021 - April 2021

Nérocean

We have developed a patented sensor for measuring dissolved oxygen in the ocean and used its fast and real-time nature to build AI network, which is dedicated to monitoring water quality in real time and predicting the health of the ocean within three months.

Ocean water sensor & Dissolved Oxygen prediction AI model

Deep-Learning based real-time classifier for ocean water quaity prediction

Funding from HK Government

Funding from Hong Kong Science Park

Funding from HKTech300, CityU

Full-Stack Intern

Sep 2020 - May 2021

Networld Tech Ltd.

I am part of a new team at Networld, nearD, a social networking site with a focus on privacy, multi-identity, and locality.

Cross-platform user interface, data visualization in nearD admin panel

Several Campaign Websites built with Leveral and React

Docker, K8s containerized and auto deployment to Google Cloud

Server side network and API management and scaling

CI/CD and two-step authorization on IP network restriction

Education

Research-based Master Degree

Jan 2023 - Present

University of Waterloo
David R. Cheriton School of Computer Science

UW Grad Scholarship
International Masters Award Excellence Scholarship
Graduate Research Studentship Scholarship
TA Funding
Research Assistant in WatVis Lab

Visting Student

June 2021 - Aug 2021

Stanford University
Department of Statistics

HKSAR Government Reaching Out Awards Funding
CGPA 4.0/4.0

Bachelor Degree

Sep 2018 - June 2022

City University of Hong Kong
Department of Computer Science

Vice President of Computer Science Student Chapter
4 years Full CityU International Student Scholarship
Awarded Dean's List
HKSAR Government Scholarship Fund
HK STEM Internship Scholarship
Department Overseas Learning Funding
English Proficiency Scholarship
Departmental Research Funding
Research Assistant in DEER Lab
Research Assistant in ERFI Lab
Research Assistant in MEI Lab

Communities & Events & Volunteers

Lead

Google Developers Student Club
CityU CS Student Chapter
Microsoft Learn Ambassador
AWS Educate Ambassador
IEEE Student Chapter
CityU IT Team

Organizer

AWS Cloud AI workshop 2021
CityU IT Team Tech seminars
CityHack 2018-21
MindSpore AI Challenge 2021
CityU AWSome Day 2020-21
Google Cloud Summit 2021
Google I/O Extend 2021
Python Workshop Series 2019

Volunteer

Stage actor - Mikado Children Theatre
Hospital Violinist (Taipei)
International Educational Volunteer (Thailand)

Papers that Inspired my Thinking

I love to read therotical papers about interface design and HCI design principles.

Reification, Polymorphism and Reuse

Michel Beaudouin-Lafon & Wendy E. Mackay - 2000

Reification turns concepts into first class objects, polymorphism permits commands to be applied to objects of different types, and reuse makes both user input and system output accessible for later use.

Demonstrational Interfaces

B.A. Myers - 1992

Demonstrational interfaces, interfaces that let the user perform actions on concrete example objects while constructing an abstract program, thus letting the user create parameterized procedures and objects without learning a programming language, are discussed.

Cognitive Engineering

Donald A. Norman - 1986

The seven-stage interaction model consists of (1) Establishing the Goal, (2) Forming the Intention, (3) Specifying the Action Sequence, (4) Executing the Action on the System's Interface, (5) Perceiving the System's State as a Response to the Action, (6) Interpreting the State, and (7) Evaluating the System State with respect to the Goals and Iterating until the goal is achieved.

Designing Interaction, not Interfaces

Michel Beaudouin-Lafon - 2004

The paper argues for a shift from interface design to interaction design as the means to significantly enhance user interfaces. It calls for the development of powerful interaction models, a better understanding of sensory-motor aspects, and novel interaction architectures addressing key challenges like reinterpretability, resilience, and scalability.

Instrumental Interaction

Michel Beaudouin-Lafon - 2000

Instrumental Interaction describes graphical user interfaces in terms of domain objects and interaction instruments. Interaction between users and domain objects is mediated by interaction instruments, similar to the tools and instruments we use in the real world to interact with physical objects.

Direct Manipulation Interfaces

Edwin L. Hutchins, James D. Hollan & Donald A. Norman - 1985

Direct manipulation has been lauded as a good form of interface design, and some interfaces that have this property have been well received by users. This article delves into the cognitive aspects of direct manipulation interfaces, examining both their advantages and disadvantages.

Thinking with External Representations

David Kirsh - 2010

Why do people create extra representations to help them make sense of situations, diagrams, illustrations, instructions and problems? The obvious explanation—external representations save internal memory and computation—is only part of the story.

Ultra-Lightweight Constraints

Scott E. Hudson & Ian Smith - 1996

Layout constraints in a user interface toolkit provide a declarative mechanism for controlling the size and position of objects in an interactive display, along with an efficient update mechanism for maintaining display layouts automatically in the face of dynamic changes.

What Is Interaction?

Kasper Hornbæk, Antti Oulasvirta - 2017

This paper explore the concept of "interaction" that lack of clear definitions in the field. It identifies various existing concepts, such as interaction as dialogue, transmission, optimal behavior, embodiment, and tool use. These concepts vary in scope and their understanding of the causal relationships between humans and computers.