Rafał Kocielnik

Exploring novel scenarios for chatbot use in Behavior Change

(Presented at "Talking With Conversational Agents in Colabortive Action" CSCW 2017 workshop
and at "Conversation UX design" CHI 2017 workshop)

Summary: We explore a number of innovative scenarios for chatbots in behavior change domain. These are scenarions where back-and-forth between user and the chatbot provides value not attainable otherwise. We explore scenarios in which bot guides the user to make sense of the self-tracking data patterns, helps the user formulate clear future goals and understand behavior change barriers. We describe each proposed scenario by a technical flow chart and a use case example scenario from users perspective.

Problem: Despite many recent advancements in conversational interaction and personal assistants, relatively little of that revolution has affected the behavior change domain. The commercial applications that do use conversational interaction, generally just repackage the already well supported functionalities in the conversational form (e.g. reminders, reports of daily activity, demographics questions). They do not take advantage of the new possibilities for novel forms of engaging the user that dialogue-based interaction enables.

Problem: Relapse takes place when the person stops following the agreed on actions and reverts back the previous patterns of behavior. Relapse is one of the hardest aspects to handle due to its, often unpredictable, appearance and causes, as well as the difficulty of reestablishing rapport with the user to get back on track.

Approach: In our approach, we use the dialogue-based capabilities to follow-up on user's non-adherence with a negotiation tactics. The system tries to prompt a user to understand the particular reason for non-adherence at the moment and adjust the next action in a way that would increase the chance of user doing at least part of the activity. A number of negotiation strategies can be employed. For example, when the reason for non-adherence is lack of time, the system could offer moving the exercise for a later time. If the reason is temporary physical inability the system could offer an alternative exercise or propose a less intense variant.

Challenges: Encouraging a user to spend additional time on typing non-adherence reasons might be challenging. Similarly building a well-performing back end for automatically recognizing flexible user non-adherence reasons is challenging as well. A hybrind machine-human intelligence system might be the most feasible approach.

Problem: Measurable and realistic goal formulation is one of the most important prerequisites for a successful behavior change. Yet many people are often unable to formulate their goals in such way due to lack of experience and proroper guidance.

Approach: An approach in behavior change called motivation interviewing uses concepts of reflection to help guide people to realize their own behavior change goals and better formulate their own action plans for achieving the desired behaviors. Dialogue based interaction lends itself well to supporting such reflection as arriving at measurable goals is oftentimes an iterative process.

Challenges: One of the main challenges in this scenario is helping users reflect and formulate their goals, but in such way as to not overwhelm them with long conversational sessions.

Problem: successfully identifying patterns in the data, does not necessarily lead to meaningful interpretation and actionable decisions. An important step of making sense of the data is needed.

Approach: We propose a dialogue-based interaction that can guide the user thorough forming understanding and explanations of behavior patterns. We first trigger the user to think about the explanations of observed patterns. Such trigger itself may lead the person to successful reflection. In case of difficulties the dialogue assists by presenting similar patterns from the past or offering guidance in retracing steps of anactivity.

Challenges: Successfuly guiding users to recall and understadn unobserved activity context is a particular challenge in this scenario.

Workshop publications:

1. New Opportunities for Dialogue-based Interaction in Behavior Change Domain
   R. Kocielnik, G. Hsieh
   CSCW 2017 workshop on Talking with Conversational Agents in Collaborative Action
   
   
2. Designing Reflective Dialogues on Self-Tracking Data
   R. Kocielnik, G. Hsieh
   CHI 2017 Workshop on Conversational UX Design
   
   

Personal conversational AI leveraging NLP and crowd-sourcing

Summary: Scheduling mult-paty meetings can be a tedious process. As part of my Microsoft Research internship at Fuse Labs I helped built a conversational AI system that acts as a personal scheduling assistant indistinguishable from an actual human in that role. A user can write an email to the AI assistant asking for scheduling a meeting and the assistant is able to take all the hardships to mabe vemt meeting done. You can try the system yourself as it has been released as a product. Behind the scenes every meeting request is decomposed, based on a structured workflow, into a number of parallel information retrieval tasks that extract information such as duration, location and attendee contact information. These tasks are first attempted to be completed using full-automation with ML and NLP techniques. If that fails, the automated tasks are turned into micro-tasks for crowd-workers. If that fails, e.g. due to lack of sufficient context, the so-called macro-task is exectuted in which a trained worker has access to larget context of the meeting. Such 3 tiered architecture allows for seamless end-user experience with the system despite the limitation of current strate-of-the-art ML.

My contribution: During my intership I analyzed the logs from past executions of the schueduling workflow to discover scenarios which most frequently led to automation failures. I redesigend part of the workflow to address one of the major inefficiencies - lack of responses from attendees. I then implemented this redesign in C# using Azure and NoSQL technologies. My implementation has been included into the official release of a Microsoft product that is now available on the market. The internship also resulted in a publication at a major HCI conference as well as 2 patent applications.

Publication:

1. Calendar. help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop
   J. Cranshaw, E. Elwany, T. Newman, R. Kocielnik, B. Yu, S. Soni, J. Teevan, A. Monroy-Hernández
   Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
   
   

Alleviating Annoyance and Boredom from Repeared Notification Exposure: Generating Engaging Behavior Change Triggers

Summary: Repeated exposure to behavior change reminders can lead to boredom, annoyance and decreased engagement. While existing research suggests that diversification of contents may mitigate these issues, no systematic strategy for contents diversification has been introduced and studied. We propose two strategies based on the use of cognitive space modeling: 1) target-diverse (B) – diversification through use of semantically-related concepts (e.g., for exercising: strength training, aerobics, fitness); and 2) self-diverse (C) – diversification through use of tailored value profile (e.g., exercising motivations: relaxation, stress reduction, physical appearance). Behavior change applications on the market mostly use multiple similar reminders (A) e.g. “exercising is good for you.” or “exercising can help you.” which are likely to result in decreased user engagement over time.

In this work we propose a systematic diverse notification generation process that leverages semantic-relatedness measure usign wikipedia links and categories structure and the crowd-sourced contents generation workflow. We evaluate our generated diverse notifications in a controlled lab study as well as 2-week long field deployment to find that the self-diverse strategy significantly reduces annoyance and boredom from repeated exposure and that both strategies increase perceived informativeness and helpfulness of the triggers. In the field the diversified notifications led to significantly higher-exercise completion.

My contribution: 1) Design of 2 innovative message diversification strategies based on use of cognitive space modeling theries, 2) Implementation of an automated mobile SMS delivery system for field deployment of the message generation service, 3) Preparation and execution of lab and fields studies.

Publication:

1. Send Me a Different Message: Utilizing Cognitive Space to Create Engaging Message Triggers
   R. Kocielnik, G. Hsieh
   Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing
   
   

Understanding Twitter Data with Lariat: Applying Human-Centered Process for Designing a Next Generation Social Science Data Exploration Tool

Summary: In this project I worked together with a group of researchers at HCDE. The problem we were trying to solve was the lack of easy to use text data exploration tools for social science researchers. Part of this problem was also the lack of clear understanding of the specific needs of social science researchers that could inform proper design of such tools. Following rigorous human-centered design process we analyzed the needs of social science researchers and designed a tool for visual exploration of text based datasets from Twitter called Lariat. Through multiple design, evaluation, and redesign cycles we gradually arrived at an easy-to-use yet powerful final design. Our tool facilitates exploratory analysis of Twitter data through a unique integration of grouping, filtering and visualization of tweets with context. We also provided a set of design implications for future analytics tools in this domain.

Team: Michael Brooks, Nan-Chen Chen, Ray Hong.

My contribution: 1) Preparation of mockups and wireframes of the interface, 2) Collection of user requirements through interviews and focus groups, 3) Research direction formulation, 4) Preparation, execution and analysis of qualitative results from the final contextual interview based evaluation.

Publication:

1. Lariat: A Visual Analytics Tool for Social Media Researchers to Explore Twitter Datasets
   N.C. Chen, M. Brooks, R. Kocielnik, S. R. Hong, J. Smith, S. Lin, Z. Qu, C. Aragon
   Hawaii International Conference on System Sciences HICSS-50, 2017
   
   

Stress@Work - Measuring physiology and context to visualize and understand stress

Summary: International project with many partners where the goal was to create a stress monitoring and management system at work. I started working on it during my internship and continued throughout my employment as a researcher. The focus of the project was on monitoring teacher's stress during work using wearable sensors provided by Philips Research. We created an interactive visualization that empowers people to be the detectives of their own stress patterns in various contexts – interacting with specific people, participating in meetings on particular topics, or traveling. We performed a number of studies with teachers at Dutch vocational schools, employees of Philips Research and professional stress coaches. We published work on interactive visual analytics of stress, signal processing algorithm for deriving stress levels from measurements of electrodermal activity and about the underlying software platform. You can find more on the project website.

My contribution: design of the visual analytics of stress, development a stress coaching application, preparation and execution out 6 user studies (3 long term field deployments), analysis of sensor measurements, preparation of more than 7 scientific publications, promotion of the project on tradeshows.

Publications:

1. Smart technologies for long-term stress monitoring at work
   R. Kocielnik, N. Sidorova, F.M. Maggi, J.Westerink, M.Ouwerkerk,
   Computer-Based Medical Systems, CBMS, Porto, Portugal, 2013.
   
   
2. Enabling self-reflection with LifelogExplorer: Generating simple views from complex data
   R. Kocielnik, F. M. Maggi, N. Sidorova,
   Pervasive Computing Technologies for Healthcare, Venice, Italy, 2013.
   
   
3. A Trust Evaluation Framework for Sensor Readings in Body Area Sensor Networks
   V. Bui, R. Verhoeven, J. Lukkien, R. Kocielnik,
   Body Area Networks, Boston, Massachusetts, USA, 2013.
   
   
4. A Practical Platform for Combining Sensor-Measurement from Body Sensor Networks with Flexible Human-Provided Tagging
   V. Bui, R. Kocielnik, N. Sidorova, R. Verhoeven, J. Lukkien,
   Mobile Services 2013, Santa Clara Marriott, CA, USA, 2013.
   
   
5. Wireless Multi Sensor Bracelet with Discreet Feedback
   M. Ouwerkerk, P. Dandine, D. Bolio, R. Kocielnik, J. Mercurio, H. Huijgen, J. Westerink,
   Wireless Health, Baltimore, MD, USA, 2013.
   
   
6. Stress@Work: From measuring stress to its understanding, prediction and handling with personalized coaching
   J. Bakker, L. Holenderski, R. Kocielnik, M. Pechenizkiy i N. Sidorova,
   International Health Informatics, 2012.
   
   
7. Stress Analytics in Education
   R. Kocielnik, M. Pechenizkiy, N. Sidorova,
   Educational Data Mining, Chania, Greece, 2012.
   
   

Bachelor's final project - 3D computer game

Summary: For my bachelors project I created a 3D computer game using C++, DirectX and Lua(scripting language). Aside from the game itself I also created a map editor that enables adding new maps and defining new scenarios without writing new code.

My contribution: DirectX based 3D engine architecture design and implementation. External tool for scenario building using Lua scripting.

Culture and Facial Expressions: A Case Study with a Speech Interface

Summary: In this project we explored culture-tailored speech interface design. Based on Hofstede's model of cultures we created a train booking speech interface in 2 versions following design guidelines about specific aspects of our selected cultures: Greeks and Dutch. We asked 8 participants from each culture to interact with both versions. We recorded their facial expression during interaction and evaluated differences in their expressiveness with 17 reviewers. Among main findings were: 1) significant main effect of interface type and culture (interacting with High UA interface made people less expressive and Greeks were in general more expressive, 2) a significant interaction effect of interface type vs. culture (Greeks were more expressive when interacting with Low UA version of the interface).

Responsibilities: Designing experiment setup, creating speech interface mock-up, preparting and performing user studies, writing scientific publication.

Publication:

1. Culture and facial expressions: a case study with a speech interface
   B. Dhillon, R. Kocielnik, I. Politis, M. Swerts, D. Szostak,
   Human-Computer Interaction–INTERACT 2011, Lisbon, Portugal, 2011.
   
   

Visual Fidelity of Video Prototypes and User Feedback: A Case Study

Summary: In this research we investigated whether prototyping the same concept using videos of different visual fidelity can affect the feedback we collect. We created a concept of a system encouraging people to use stairs instead of a lift and depicted it using the same video script in 2 versions of visual fidelity: (1) Lo-Fi – sketchy, (2) Hi-Fi – with actors. We collected qualitative and quantitative feedback through user studies. Contrary to our initial hypotheses, based on review of other forms of prototyping, we did not find expected differences in feedback quantity and quantity. The results suggested that Lo-Fi video yields equally good feedback on the prototype. However, since the attractiveness of the Lo-Fi video was actually rated higher we considered that, perhaps, this version has been too refined - drawn by a professional designer, compared to amateur actors playing in Hi-Fi video.

My contribution: Filming and editing video, preparting and performing user studies, developing on-line evaluation website, writing scientific publication.

Publication:

1. Visual fidelity of video prototypes and user feedback: a case study
   B. Dhillon, P. Banach, R. Kocielnik, J. Peregrín Emparanza, I. Politis, A. Raczewska i P. Markopoulos,
   BCS Conference on Human-Computer Interaction, Newcastle, UK, 2011.
   
   

Anthropometric Facial Emotion Recognition

Summary: In my master thesis we focused on improving existing algorithms for automated facial emotion recognition. We approached emotion recognition using 17 points describing facial elements and our main contribution was a hybrid approach to extraction of these feature points. We further used points' positions in training the Neural Network classifier. We improved the accuracy on the test database over the competing solutions from ~72% to ~78%. The automated point localization accuracy was ~96% measured against manual tagging. The algorithm was also fast enough to process live video. The remaining challenges involved improving robustness in various light conditions and recognizing natural emotions.

My contribution: Developing an image analysis part of the facial emotion recongition system in C++ using OpenCV library, performing evaluation, writing scientific publication and my master thesis ;)

Publications

1. Anthropometric Facial Emotion Recognition
   J. Jarkiewicz, R. Kocielnik, K. Marasek
   HCI International, San Diego, CA, USA, 2009.
   
   

Tangible Interaction in Medical Domain

Summary: We worked with the visualization of the brain's white matter created by the Biomedical Engineering department of TU/e and focused on: (1) selecting optimal settings of visualization parameters, and (2) creating and evaluating the prototype for tangible interaction with the visualization. The medical visualization had myriad of parameters that controlled its appearance and the challenge was to find their optimal settings. For the second part we used an optical 3D tracker to directly map movements of a physical object to a 3D graphical model of a brain. We evaluated the interface with medical professionals and learned that they prefer it mainly for collaboration. Please see the video showing the functionality of our prototype.

Team: Angeliki Angeletou, Miroslav Bojic, Rafal Kocielnik, Dalila Szostak, Flavio Signoreli.

My contribution: Designing visual analytics of stress, developing a stress coaching application, preparting and performing user studies, analyzing sensor measurements, writing scientific publications, tutoring students, presenting project on tradeshows - valorization