Designers create inspirational mood boards to express their design ideas visually, through collages of images and text. They find appropriate images and reflect on them as they explore emergent design concepts. After presenting the results of a participatory design workshop and a survey of professional designers, we introduce SemanticCollage, a digital mood board tool that attaches semantic labels to images by applying a state-of-the-art semantic labeling algorithm. SemanticCollage helps designers to 1) translate vague, visual ideas into search terms; 2) make better sense of and communicate their designs; while 3) not disrupting their creative flow. A structured observation with 12 professional designers demonstrated how semantic labels help designers successfully guide image search and find relevant words that articulate their abstract, visual ideas. We conclude by discussing how SemanticCollage inspires new uses of semantic labels for supporting creative practice.

Citation: Janin Koch, Nicolas Taffin, Andrés Lucero, and Wendy Mackay. 2020.
SemanticCollage: Enriching Digital Mood Board Design with Semantic Labels.
In Proceedings of the Designing Interactive Systems Conference 2020 (DIS ’20). ACM

Project Page with Code and Additional Material: https://userinterfaces.aalto.fi/SemanticCollage/   Paper Download : SemanticCollage: Enriching Digital Mood Board Design with Semantic Labels

Professional designers create mood boards to explore, visualize, and communicate hard-to-express ideas. We present ImageSense, an intelligent, collaborative ideation tool that combines individual and shared work spaces, as well as collaboration with multiple forms of intelligent agents. In the collection phase, ImageSense offers fluid transitions between serendipitous discovery of curated images via ImageCascade, combined text- and image-based Semantic search, and intelligent AI suggestions for finding new images. For later composition and reflection, ImageSense provides semantic labels, generated color palettes, and multiple tag clouds to help communicate the intent of the mood board. A study of nine professional designers revealed nuances in designers’ preferences for designer-led, system-led, and mixed-initiative approaches that evolve throughout the design process. We discuss the challenges in creating effective human-computer partnerships for creative activities, and suggest directions for future research.

Citation: Janin Koch, Nicolas Taffin, Michel Beaudouin-Lafon, Markku Laine, Andrés Lucero, and Wendy E. Mackay. 2020.
ImageSense: An Intelligent Collaborative Ideation Tool to Support Diverse Human-Computer Partnerships.
Proc. ACM Hum.-Comput. Interact. 4, CSCW1, Article 045 (May 2020), 27 pages. DOI:https://doi.org/10.1145/3392850

Project Page with Additional Material: https://userinterfaces.aalto.fi/ImageSense/   Paper Download : ImageSense: An Intelligent Collaborative Ideation Tool to Support Diverse Human-Computer Partnerships

Art and design are essential aspects of our culture and how we interact with the world. Artists and designers use a wide selection of tools, whose impact is rapidly growing with the progression of digital technologies. This change has opened up new opportunities for the CHI community to build creative supportive tools. The digital switch has come with many benefits such as lowering barriers, mobile work environments and mass production for distribution of work. Along with these benefits we also see challenges for art and design work and its future perception in society. As technology takes a more significant role in supporting art and design what will this mean for the individual artist or designer? The focus of this workshop is to bring together researchers and practitioners to explore what the future of digital art and design will hold. The exploration will centre around synthesizing key challenges and questions, along with ideas for future interaction technologies that consider mobile and tangible aspects of digital art.

Citation: Janin Koch, Jennifer Pearson, Andrés Lucero, Miriam Sturdee, Wendy E. Mackay, Makayla Lewis, and Simon Robinson. 2020. Where Art Meets Technology: Integrating Tangible and Intelligent Tools in Creative Processes. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–7. DOI:https://doi.org/10.1145/3334480.3375172 

Paper download: Where Art Meets Technology.

Design ideation is a prime creative activity in design. However, it is challenging to support computationally, because of its quickly evolving and exploratory nature. This paper studies cooperative contextual bandits (CCB) as a machine learning method for interactive ideation support. CCBs can learn to propose domain-relevant contributions and adapt their exploration/ exploitation strategy. We developed a CCB for an interactive design ideation tool that 1) suggests inspirational and situationally relevant materials (“may AI?”), 2) explores and exploits inspirational materials with the designer, and 3) explains suggestions to aid reflection. As an application case, we study digital mood board design, wherein visual inspirational materials are collected and curated in collages. In a controlled study, 14 of 16 professional designers preferred the CCB-augmented tool. CCBs are promising for ideation activities where adaptive and steerable support is welcome but designers need to retain full control of the outcome.

Citation: Koch, Janin, Andrés Lucero, Lena Hegemann and Antti Oulasvirta. 2019. “May AI? Design Ideation with Cooperative Contextual Bandits”, Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 2019.

Project Page with Code and Additional Material: https://userinterfaces.aalto.fi/ccb/   Paper Download : May AI? Design Ideation with Cooperative Contextual Bandits

Collaborative AI’s offer great potential for explorative and creative methods. While previous work in HCI and ML mainly focuses on exploiting either human or machine capabilities, the concept of collaboration suggests work on equal terms to achieve synergy effects as seen in collaborative creativity. However, the uncertain nature of creative problems raises new questions regarding the definition and design of such systems. We present a collaborative system for mood board design based on a state-of-the-art contextual bandit structure that is able to iteratively adapt to changing objectives, and autonomously selects explorative/ exploitative strategies to propose suitable contributions to the current mood board. Besides the technical implementation, we discuss the need of design professionals in the design of such collaborative systems as well as open questions based on our results.

Aalto Interface Metrics (AIM) pools several empirically validated models and metrics of user perception and attention into an easy-to-use online service for the evaluation of graphical user interface (GUI) designs. Users input a GUI design via URL, and select from a list of 17 different metrics covering aspects ranging from visual clutter to visual learnability. AIM presents detailed breakdowns, visualizations, and statistical comparisons, enabling designers and practitioners to detect shortcomings and possible improvements. The web service and code repository are available at interfacemetrics.aalto.fi.

Citation: Antti Oulasvirta, Samuli De Pascale, Janin Koch, Thomas Langerak, Jussi Jokinen, Kashyap Todi, Markku Laine, Manoj Kristhombuge, Yuxi Zhu, Aliaksei Miniukovich, Gregorio Palmas, and Tino Weinkauf. 2018. Aalto Interface Metrics (AIM): A Service and Codebase for Computational GUI Evaluation. In The 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings (UIST '18 Adjunct). ACM, New York, NY, USA, 16-19. DOI: https://doi.org/10.1145/3266037.3266087.

You can downlad this paper here.

Significant advances in artificial intelligence suggest that we will be using intelligent agents on a regular basis in the near future. This chapter discusses group cognition as a principle for designing collaborative AI. Group cognition is the ability to relate to other group members’ decisions, abilities, and beliefs. It thereby allows participants to adapt their understanding and actions to reach common objectives. Hence, it underpins collaboration. We review two concepts in the context of group cognition that could inform the development of AI and automation in pursuit of natural collaboration with humans: conversational grounding and theory of mind. These concepts are somewhat different from those already discussed in AI research. We outline some new implications for collaborative AI, aimed at extending skills and solution spaces and at improving joint cognitive and creative capacity.

Citation: Koch, Janin, and Antti Oulasvirta. 2018. “Group Cognition and Collaborative AI.” In Human and Machine Learning: Visible, Explainable, Trustworthy and Transparent. Springer International Publishing. 

Book available here. Download our chapter here.

An increasing trend toward the digitalization of design practice and research on tools for augmenting creativity will encourage novel ways of designing in the future. Especially, tools for collecting and interacting with digital inspirational material present great opportunities for current design practice. However, the complex nature of such highly creative practice raises new challenges and questions in relation to such developments. We present three tools for inspiration seeking as a base for discussing open questions identified in our previous work. These tools vary in their agency within current practice for seeking and interacting with digital inspirational material to allow a wider scope of analysis. We intend to use these questions to discuss future guidelines for designing such tools and systems. Published at DIS’18 Workshop: Designing interactive systems to support and augment creativity

 Citation: Koch, Janin and Lucero, Andrés. 2018. "Tools for Inspiration Seeking: Raising Open Questions." Workshop paper DIS'18 Workshop: Designing interactive systems to support and augment creativity. 

Paper Download: Tools for Inspiration Seeking

Over the last decade, many new opportunities have emerged to support creativity and problem-solving in design by finding inspirational materials via the Internet. Online design communities such as those of Behance and Pinterest showcase portfolios and user-made artwork, and they offer support for designers’ day-to-day work to find and collect inspirational material. However, very little is known about how these communities affect inspiration-related practices of professional designers and how designers view them. This paper presents new data on the practices designers employ when seeking digital inspiration sources online and reflecting on, tracking, and managing them in today’s Web design. Current practice and views on sources of inspiration were described based on responses from 51 professional designers. The results suggest that the Internet has become a prevalent source for ideas in design, yet designers experience mounting issues of trust and relatedness with regard to online sources. Therefore, encouraging both should be considered a guiding principle for tools aimed at supporting designers within the realm of design practice.

Citation: Koch, Janin, Magda Laszlo, Andrés Lucero, and Antti Oulasvirta. 2018. “Surfing for Inspiration: Digital Inspirational Material in Design Practice.” In Proceedings of Design Research (DRS), 2018.

Paper Download : Surfing for Inspiration: Digital Inspirational Material in Design Practice

Human-Computer Collaboration offers great potential for explorative and creative problem-solving strategies. While previous work in HCI and ML mainly focuses on exploiting either human or machine capabilities, the concept of collaboration suggests work on equal terms to achieve synergy effects. The uncertain nature of creative problems raises new questions regarding the adaptability of systems to changing objectives in iterative processes. We present a collaborative system for mood board design based on a state-of-the-art contextual bandit structure that is able to iteratively adapt to changing behaviors, and moves autonomously through solution spaces to propose suitable contributions. Besides the technical implementation, we discuss the need for further research on collaborative interaction behaviors between humans and machines.

Citation: Koch, Janin. 2018. "Contextual Bandits for Design: A Human-Computer Collaboration Approach." Workshop paper CHI Conference Extended Abstracts on Human Factors in Computing Systems.

You can downlad this paper here.

This paper proposes a framework for a collaborative designing system from an interaction design perspective. Using the agent-based model from the mixed-initiative interaction framework as a starting point, an ideal interaction scenario in a web design context is described and implications for designing collaborative systems are presented. Previous work on machine learning and artificial intelligence for interaction design has already looked at recognition of designers’ intent and combinatorial problem-solving in design. This paper, in contrast, focuses on the interaction design perspective of designing such a system, and introduces a framework that highlights requirements in this context. The framework uses the notion of task model and world model from agent-based models as a frame, and the resulting implications call for a stronger involvement of designers in the process. Position paper at the AAAI Spring Symposium on Designing the User Experience of Machine Learning Systems, 2017.

Citation: Koch, Janin. "Design implications for Designing with a Collaborative AI." 2017 AAAI Spring Symposium Series. 2017.

Paper Download: Design implications for Designing with a Collaborative AI

We present preliminary results on computational perception of interactive layouts. Our goal is to algorithmically estimate how users perceive a layout. Potential applications range from automated usability evaluation to computer-generated and adaptive interfaces. Layout perception is challenging, however, because of diverse features, combinatorial complexity, and absence of approaches. We have explored Gestalt laws as parsing heuristics. Our approach finds a parametrization that optimally resolves conflicts among competing interpretations of a layout. The output is a hierarchical grouping of main elements. The results are promising: an implementation of just four Gestalt laws enables hierarchical grouping that presents promising results in 90% of our (realistic) test cases.

Citation: Koch, Janin, and Antti Oulasvirta. "Computational layout perception using gestalt laws." Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2016.

Paper Download: Computational Layout Perception using Gestalt Laws

This position paper discusses the use of machine learning methods in layout design. Interactive layouts are pervasive and a central part of e.g. GUIs, Web interfaces, menus and forms. They have been hard to design algorithmically because search spaces are large and multiple factors contributing to design choices. We argue that in order to touch base with real design practices, machine learning approaches should take into account the requirements posed by user-centered design. We have identified four touch points to user-centered design. For each touch point we discuss both opportunities and challenges and show results from our on-going work.

Citation: Koch, Janin, Weir, Daryl and Antti Oulasvirta. "Learning Layout Design: Challenges and Opportunities." Workshop paper CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2016.

Paper Download: Learning Layout Design: Challenges and Opportunities

In this paper, we present a design exploration of the domain of stress therapy, involving stress researchers and professional therapists. We address the problem of using bio-data, collected during the course of everyday live, in stress therapy. The challenge is centered on how to visualize data in a way that can be useful for therapists and patients. We present our results and reflect on our design exploration, arguing that research through design can provide useful insights in domains that are hard to access using other methods of inquiry, as well as support imagination of future scenarios. Paper Download: Imagining the future of stress therapy through a design exploration