Rafal Kocielnik

I am a Postdoctoral Researcher at Caltech where I am mentored by Anima Anandkumar, Mike Alvarez, and Andrew Hung. I received my PhD in Human Centered Design & Engineering from the University of Washington, where I was advised by Gary Hsieh.

My primary interests focus on Leveraging AI for Behavior Change as well as Empowering Users to Inspect AI Systems. I have unique experience in Human-Computer Interaction and Design in combination with Applied ML. I worked on projects in various areas including Explainable AI, Surgical AI, Health Behavior Change, Workspace Productivity and Social Media Moderation.

Over the summers, I've been lucky to intern with Jonathan Bragg and Doug Downey at Allen Institute for AI, Saleema Amershi and Andrés Monroy-Hernández at Microsoft Research Redmond, and Daniel Avrahami at Fuji-Xerox Palo Alto Research Lab.

rafalko [at] caltech.edu

News

Nov 2024 Two proceedings papers accepted at Machine Learning for Health 2024:
"Multi-Modal Self-Supervised Learning for Surgical Feedback Effectiveness Assessment" (spotlight)
"Automating Feedback Analysis in Surgical Training: Detection, Categorization, and Assessment"
May 2024 Invited talk at USC's Information Sciences Institute on "Human-AI Interaction: From Supporting Surgical Training to Inspecting Social Bias in LLMs".
Feb 2024 Invited talk at University of Oxford's Future of Tech & Society Discussion Group on "Testing Fairness in Generative AI".
Dec 2023 Our work on "Deep Multimodal Fusion for Surgical Feedback Classification" received a Best Paper Award at Machine Learning for Health 2023!
Dec 2023 Two workshop papers on "User-Centered Tools for Inspecting Social Bias in Generative AI" and on "Bias Quantification in Downstream Tasks of Image-Text Diffusion Models" accepted to NeurIPS 2023.
Nov 2023 Invited talk at NVIDIA's Conversational AI & Deep Learning Product Team on "BiasTestGPT: Using ChatGPT for Social Bias Testing of Language Models".

Research

Challenges in Moderating Disruptive Player Behavior in Online Competitive Action Games
Rafal Kocielnik, Zhuofang Li, Claudia Kann, Deshawn Sambrano, Jacob Morrier, Mitchell Linegar, Carly Taylor, Min Kim, Nabiha Naqvie, Feri Soltani, Arman Dehpanah, Grant Cahill, Animashree Anandkumar, R Michael Alvarez
Frontiers in Computer Science: Human-Media Interaction, 2024
Publisher

Artificial Intelligence-Based Video Feedback to Improve Novice Performance on Robotic Suturing Skills: A Pilot Study
Runzhuo Ma, Dani Kiyasseh, Jasper A Laca, Rafal Kocielnik, Elyssa Y Wong, Timothy N Chu, Steven Cen, Cherine H Yang, Istabraq S Dalieh, Taseen F Haque, Mitch G Goldenberg, Xiuzhen Huang, Anima Anandkumar, Andrew J Hung
Journal of Endourology, 2024
Publisher

Deep Multimodal Fusion for Surgical Feedback Classification
Rafal Kocielnik, Elyssa Y. Wong, Timothy N. Chu, Lydia Lin, De-An Huang, Jiayun Wang, Anima Anandkumar, Andrew J. Hung
Machine Learning for Health (ML4H), 2023 Best Proceedings Paper
Publisher

Exploring Social Bias in Downstream Applications of Text-to-Image Foundation Models
Adhithya Saravanan, Rafal Kocielnik, Roy Jiang, Pengrui Han, Anima Anandkumar
NeurIPS workshop on Failure Modes in the Age of Foundation Models, 2023
(accepted for special issue of PMLR)
arXiv OpenReview

BiasTestGPT: Using ChatGPT for Social Bias Testing of Language Models
Rafal Kocielnik, Shrimai Prabhumoye, Vivian Zhang, Roy Jiang, R. Alvarez Michael, Anima Anandkumar
ICML workshop on Deployment Challenges for Generative AI 2023
arXiv HuggingFace Tool Project Website Dataset

Large Language Models and Political Science
Mitchell Linegar, Rafal Kocielnik, R Michael Alvarez
Frontiers in Political Science, 2023
Publisher

Can You Label Less by Using Out-of-Domain Data? Active & Transfer Learning with Few-shot Instructions
Rafal Kocielnik, Sara Kangaslahti, Shrimai Prabhumoye, Meena Hari, Michael Alvarez, Anima Anandkumar
NeurIPS 2023 Transfer Learning for Natural Language Processing Workshop
Publisher Medium Blog

From Who You Know to What You Read: Augmenting Scientific Recommendations with Implicit Social Networks
Hyeonsu B Kang, Rafal Kocielnik, Andrew Head, Jiangjiang Yang, Matt Latzke, Aniket Kittur, Daniel S Weld, Doug Downey, Jonathan Bragg
CHI 2022
Publisher

Using real-time feedback to improve surgical performance on a robotic tissue dissection task
Jasper A Laca, Rafal Kocielnik, Jessica H Nguyen, Jonathan You, Ryan Tsang, Elyssa Y Wong, Andrew Shtulman, Anima Anandkumar, Andrew J Hung
European Urology Open Science, 2022
Publisher

Can I Talk to You about Your Social Needs? Understanding Preference for Conversational User Interface in Health
Rafal Kocielnik, Raina Langevin, James S George, Shota Akenaga, Amelia Wang, Darwin P Jones, Alexander Argyle, Callan Fockele, Layla Anderson, Dennis T Hsieh, Kabir Yadav, Herbert Duber, Gary Hsieh, Andrea L Hartzler
Conversational User Interfaces, 2021 Best Paper Honorable Mention
Publisher

Will You Accept an Imperfect AI? Exploring Designs for Adjusting End-user Expectations of AI Systems
Rafal Kocielnik, Saleema Amershi, Paul N Bennett
CHI 2019
High Impact: 240+ citations
Publisher MSR Blog Presentation Video

HarborBot: A Chatbot for Social Needs Screening
Rafal Kocielnik, Elena Agapie, Alexander Argyle, Dennis T Hsieh, Kabir Yadav, Breena Taira, Gary Hsieh
AMIA Annual Symposium Proceedings 2019
Publisher

Reflection Companion: A Conversational System for Engaging Users in Reflection on Physical Activity
Rafal Kocielnik, Lillian Xiao, Daniel Avrahami, Gary Hsieh
UbiComp / IMWUT 2018
High Impact: 150+ citations
Publisher

Designing for Workplace Reflection: A Chat and Voice-Based Conversational Agent
Rafal Kocielnik, Daniel Avrahami, Jennifer Marlow, Di Lu, Gary Hsieh
Design of Interactive Systems 2018
Publisher

Reciprocity and Donation: How Article Topic, Quality and Dwell Time Predict Banner Donation on Wikipedia
Rafal Kocielnik, Os Keyes, Jonathan T Morgan, Dario Taraborelli, David W McDonald, Gary Hsieh
CSCW 2018
Publisher

Send Me a Different Message: Utilizing Cognitive Space to Create Engaging Message Triggers
Rafal Kocielnik, Gary Hsieh
CSCW 2017
Publisher

Calendar. help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop Authors
Justin Cranshaw, Emad Elwany, Todd Newman, Rafal Kocielnik, Bowen Yu, Sandeep Soni, Jaime Teevan, Andrés Monroy-Hernández
CHI 2017
Publisher

You get who you pay for: The impact of incentives on participation bias
Gary Hsieh, Rafal Kocielnik
CSCW 2016 Best Paper Award
Publisher

Smart technologies for long-term stress monitoring at work
Rafal Kocielnik, Natalia Sidorova, Fabrizio Maria Maggi, Martin Ouwerkerk, Joyce HDM Westerink
International Symposium on Computer-based Medical Systems 2013
High Impact: 170+ citations
Publisher