The presentation of portable digital tools such as mobile phones, tablets is changing the landscape of cooking experience in most families. People start using digital cookbooks in kitchen for many reasons. These reasons include but not limited to the following: creating enough space for food and cooking, being more accessible than traditional cookbooks, etc. However, there is still a number of problems involved in the digital cooking age due to the limits of current mobile technologies, such as screen interaction/recognition and the information architecture design for mobile apps in HCI. In these years, emerging technologies such as gesture recognition and motion analysis are offering a lot of room for explorations and possibilities as cooks find themselves struggling with the current experience in following cooking instructions in many ways when their hands are not available.

Project Goal

In this project, we are going to explore opportunities in a non-touch based cooking experience, aiming to leverage the current experience with emerging gesture control technologies and refine the user interfaces and interaction patterns. Check out the project with a Leap Motion.

Problem Statement

Design Criteria

  • It has to be including different types of users, and cover different levels of knowledge in cooking experience.

  • It uses simple gestures to control elements and navigate through the application.

  • It is self-explanatory-image-oriented which allows users to quickly understand instructions.

  • It uses different types of media/content to contextualize the instructional experience.

Grimy Fingerprints and Smudges

One of the major problems of available digital media design for cooking is, even though people take advantages of the mobile device portability and content accessibility, they still find it hard to interact with screens when they are cooking because their hands are typically busy with either preparing ingredients or cooking. It is worth thinking about how to enables cooks to interact with a digital device surface without messing it up. By taking advantages of gesture recognition technologies, I tried to solve this problem by allowing cooks avoid physically touching the device surface, leaving finger smudges on it.

Users with Different Levels of Expertise

The current design of most digital cookbooks assume the users have the basic knowledges in cooking so they use generic directions sometimes in the description and ignore supportive information which the designers think users should have been quite familiar with before. However, it turns out that there is a portion of novice users who know nothing about cooking and do not understand the generic directions listed in the recipe. These people could easily get confused especially when the directions are weigh too generic and there is no explanation to them or they could not continue with the directions due to the missing “common sense” steps. Therefore, how to arrange the content in a way which neither confuses novice users nor annoys experienced cooks deserves more attentions and considerations.

Design Process

User Research

2 0 1 6 / 0 8 ~ 2 0 1 6 / 1 0

I did contextual inquiries to five participants, asked them to cook a dish that they never cooked before, following recipes from whatever sources/platforms. Then I interviewed about their feelings, confusions and expectations in order to understand their problems and needs.

Competitive Analysis

2 0 1 6 / 1 1 ~ 2 0 1 6 / 1 2

According to my initial user interviews, I identified three main types of recipe use cases and projects -- traditional recipe book, e.g. Allrecipes, video recipes, e.g. Delish and Youtube video, and augmented reality cooking experience. I analyzed these three main use cases, compared their pros and cons, and figured out how the features could be applied to helping with leveraging a non-touch based cooking experience.


2 0 1 7 / 0 1 ~ 2 0 1 7 / 0 2

Taking those user needs and features in mind, I came up with two possible solutions. The first solution curates the instructional experience based on user’s expertise level. An filter functionality is added to allow the users to adjust the information. The second solution combines high-level steps with details such as breakdowns, tools and cooking tips. User with the second solution could go along the main path/high-level steps and pull up additional supportive information based on their need.

User Evaluation

2 0 1 7 / 0 2 ~ 2 0 1 7 / 0 3

I brought both solutions back to the five participants involved in my initial user study, showed them around and asked them to vote for the better one. Most of them agrees that second solution is more aligned with their expectation. Although the first solution offers the opportunities curate the content and enough flexibility, participants argue that it is still too complex and confusing for them to accept the new gesture and achieve the simple tasks. The prefer to see gestures that are more traditional and simple. So I decide to continue with the second solution.

Information Architecture Design

2 0 1 7 / 0 2 ~ 2 0 1 7 / 0 3

I designed a user workflow to conceptualize how cooks use the recipe book. Cooks get started with the an overview of the dish, introducing general information and ingredients. Then they go to the first step. If they feel confused about the generic direction in step one and need supportive information about it, they could go to the detail page for a foolproof explanation or watch a video tutorial. They could listen to an audio cooking tutorial as well if they would like to keep the same pace with cooking.

Interaction Design

2 0 1 7 / 0 2 ~ 2 0 1 7 / 0 3

Since most participants enjoy the simple and traditional interaction patterns, I decided to make gestures easy to understand. Compared to web interactions, Leap Motion does not support precise click on tiny items. In this case, I decided to use swiping as the most basic interaction unit. By swiping up and down, users navigate through different steps. When they swipe left and right, they got switched between different media types. There are some other gestures involved in this project as well, such as pinch to go back and start over and circle to play audio tutorials.


2 0 1 7 / 0 3 ~ 2 0 1 7 / 0 4

There are two advanced JavaScript libraries involved in the project implementation. I use Leap.js to recognize gestures and then control the navigation command. I am also using Fullpage.js to simulate a full page sliding effects.


Introductory Video

Come and quickly get the idea of the project by watching a 2 minutes video.

Hi-fi Flowchart

Come and check out how cookbooks navigate users through the process step by step.


The working prototype demonstrate the key features and addresses the main problems that I defined in my problem statement. Enjoy!


Link to the project proposal with detailed problem definition and some research I did in the first place. Feel free to check it out.


I created a documentation book which records the full description of the project, ranging from problem definition, design criteria as well as my explorations including both design and technical feasibility.