Janin Koch

PhD Candidate at Aalto University

May AI? Design Ideation with Cooperative Contextual Bandits

Design ideation is a prime creative activity in design. However, it is challenging to support computationally, because of its quickly evolving and exploratory nature. This paper studies cooperative contextual bandits (CCB) as a machine learning method for interactive ideation support. CCBs can learn to propose domain-relevant contributions and adapt their exploration/ exploitation strategy. We developed a CCB for an interactive design ideation tool that 1) suggests inspirational and situationally relevant materials (“may AI?”), 2) explores and exploits inspirational materials with the designer, and 3) explains suggestions to aid reflection. As an application case, we study digital mood board design, wherein visual inspirational materials are collected and curated in collages. In a controlled study, 14 of 16 professional designers preferred the CCB-augmented tool. CCBs are promising for ideation activities where adaptive and steerable support is welcome but designers need to retain full control of the outcome.

Citation: Koch, Janin, Andrés Lucero, Lena Hegemann and Antti Oulasvirta. 2019. “May AI? Design Ideation with Cooperative Contextual Bandits”, Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 2019.

Project Page with Code and Additional Material: https://userinterfaces.aalto.fi/ccb/


Paper Download : May AI? Design Ideation with Cooperative Contextual Bandits


Aalto Interface Metrics (AIM): A Service and Codebase for Computational GUI Evaluation

Aalto Interface Metrics (AIM) pools several empirically validated models and metrics of user perception and attention into an easy-to-use online service for the evaluation of graphical user interface (GUI) designs. Users input a GUI design via URL, and select from a list of 17 different metrics covering aspects ranging from visual clutter to visual learnability. AIM presents detailed breakdowns, visualizations, and statistical comparisons, enabling designers and practitioners to detect shortcomings and possible improvements. The web service and code repository are available at interfacemetrics.aalto.fi.

Citation: Antti Oulasvirta, Samuli De Pascale, Janin Koch, Thomas Langerak, Jussi Jokinen, Kashyap Todi, Markku Laine, Manoj Kristhombuge, Yuxi Zhu, Aliaksei Miniukovich, Gregorio Palmas, and Tino Weinkauf. 2018. Aalto Interface Metrics (AIM): A Service and Codebase for Computational GUI Evaluation. In The 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings (UIST '18 Adjunct). ACM, New York, NY, USA, 16-19. DOI: https://doi.org/10.1145/3266037.3266087.

You can downlad this paper here.

Contextual Bandits for Design: A Human-Computer Collaboration Approach

Human-Computer Collaboration offers great potential for explorative and creative problem-solving strategies. While previous work in HCI and ML mainly focuses on exploiting either human or machine capabilities, the concept of collaboration suggests work on equal terms to achieve synergy effects. The uncertain nature of creative problems raises new questions regarding the adaptability of systems to changing objectives in iterative processes. We present a collaborative system for mood board design based on a state-of-the-art contextual bandit structure that is able to iteratively adapt to changing behaviors, and moves autonomously through solution spaces to propose suitable contributions. Besides the technical implementation, we discuss the need for further research on collaborative interaction behaviors between humans and machines.

Citation: Koch, Janin. 2018. "Contextual Bandits for Design: A Human-Computer Collaboration Approach." Workshop paper CHI Conference Extended Abstracts on Human Factors in Computing Systems.

You can downlad this paper here.

Surfing for Inspiration: Digital Inspirational Material in Design Practice

Over the last decade, many new opportunities have emerged to support creativity and problem-solving in design by finding inspirational materials via the Internet. Online design communities such as those of Behance and Pinterest showcase portfolios and user-made artwork, and they offer support for designers’ day-to-day work to find and collect inspirational material. However, very little is known about how these communities affect inspiration-related practices of professional designers and how designers view them. This paper presents new data on the practices designers employ when seeking digital inspiration sources online and reflecting on, tracking, and managing them in today’s Web design. Current practice and views on sources of inspiration were described based on responses from 51 professional designers. The results suggest that the Internet has become a prevalent source for ideas in design, yet designers experience mounting issues of trust and relatedness with regard to online sources. Therefore, encouraging both should be considered a guiding principle for tools aimed at supporting designers within the realm of design practice.

Citation: Koch, Janin, Magda Laszlo, Andrés Lucero, and Antti Oulasvirta. 2018. “Surfing for Inspiration: Digital Inspirational Material in Design Practice.” In Proceedings of Design Research (DRS), 2018.

Paper Download : Surfing for Inspiration: Digital Inspirational Material in Design Practice

Group Cognition and Collaborative AI

Significant advances in artificial intelligence suggest that we will be using intelligent agents on a regular basis in the near future. This chapter discusses group cognition as a principle for designing collaborative AI. Group cognition is the ability to relate to other group members’ decisions, abilities, and beliefs. It thereby allows participants to adapt their understanding and actions to reach common objectives. Hence, it underpins collaboration. We review two concepts in the context of group cognition that could inform the development of AI and automation in pursuit of natural collaboration with humans: conversational grounding and theory of mind.
These concepts are somewhat different from those already discussed in AI research. We outline some new implications for collaborative AI, aimed at extending skills and solution spaces and at improving joint cognitive and creative capacity.

Citation: Koch, Janin, and Antti Oulasvirta. 2018. “Group Cognition and Collaborative AI.” In Human and Machine Learning: Visible, Explainable, Trustworthy and Transparent. Springer International Publishing. 

Book available here.
Download our chapter here.

Tools for Inspiration Seeking: Raising Open Questions

An increasing trend toward the digitalization of design practice and research on tools for augmenting creativity will encourage novel ways of designing in the future. Especially, tools for collecting and interacting with digital inspirational material present great opportunities for current design practice. However, the complex nature of such highly creative practice raises new challenges and questions in relation to such developments. We present three tools for inspiration seeking as a base for discussing open questions identified in our previous work. These tools vary in their agency within current practice for seeking and interacting with digital inspirational material to allow a wider scope of analysis. We intend to use these questions to discuss future guidelines for designing such tools and systems.

Published at DIS’18 Workshop: Designing interactive systems to support and augment creativity

 Citation: Koch, Janin and Lucero, Andrés. 2018. "Tools for Inspiration Seeking: Raising Open Questions." Workshop paper DIS'18 Workshop: Designing interactive systems to support and augment creativity. 

Paper Download: Tools for Inspiration Seeking

Design implications for Designing with a Collaborative AI

This paper proposes a framework for a collaborative designing system from an interaction design perspective. Using the agent-based model from the mixed-initiative interaction framework as a starting point, an ideal interaction scenario in a web design context is described and implications for designing collaborative systems are presented. Previous work on machine learning and artificial intelligence for interaction design has already looked at recognition of designers’ intent and combinatorial problem-solving in design. This paper, in contrast, focuses on the interaction design perspective of designing such a system, and introduces a framework that highlights requirements in this context. The framework uses the notion of task model and world model from agent-based models as a frame, and the resulting implications call for a stronger involvement of designers in the process. ”
Position paper at the AAAI Spring Symposium on Designing the User Experience of Machine Learning Systems, 2017.

Paper Download: Design implications for Designing with a Collaborative AI

Computational Layout Perception using Gestalt Laws

We present preliminary results on computational perception of interactive layouts. Our goal is to algorithmically estimate how users perceive a layout. Potential applications range from automated usability evaluation to computer-generated and adaptive interfaces. Layout perception is challenging, however, because of diverse features, combinatorial complexity, and absence of approaches. We have explored Gestalt laws as parsing heuristics. Our approach finds a parametrization that optimally resolves conflicts among competing interpretations of a layout. The output is a hierarchical grouping of main elements. The results are promising: an implementation of just four Gestalt laws enables hierarchical grouping that presents promising results in 90% of our (realistic) test cases

Citation: Koch, Janin, and Antti Oulasvirta. "Computational layout perception using gestalt laws." Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2016.

Paper Download: Computational Layout Perception using Gestalt Laws