CONTROLLING A GAME CHARACTER USING HAND GESTURES AND COMPARING IT TO A KEYBOARD

Done by:

BOULMANE Oumayma
GUERWANE Houda

Supervised by:

COUTRIX Céline
GOGUEY Alix

The Context:

Interest in tangible user interfaces has grown constantly since the 90s, and with every year, more tangible systems are showing up. This has sparked a debate about whether these interfaces are better than their graphical counterparts. Tangible interfaces have a distinguishable feature that is providing the user with physical interaction that connects the digital with the physical world. Out of all the possible tangible interfaces, using hands to play games has piqued our interest. In fact, historically and in the modern world most games are played using hands and for good reason; hand games exist in a variety of cultures internationally. For instance, East Asia has a long history of hand games, which are known as ken games in Japan, such as Kitsune-ken which is a category of East Asian hand games played by using three hand gestures. A distinct feature of kitsune-ken is that the game is played by making gestures with both hands [1]. Therefore, it is interesting to test out an implementation of tangible interfaces using hand gestures to animate a character in a game and compare it to the graphical version of the same game.

The problem:

The thumb is one of the most flexible fingers because it has independent muscles serving it. Moreover, the other fingers are flexed by two muscles in the forearm called “long flexors” and one set of muscles in the hand called lumbricals. However, the index finger has the best movement because one of the forearm muscles has a separate tendon connected to it. This tendon has a separate extensor muscle that allows the index to independently point even when the hand is in a tight fist [3]. Taking advantage of this physical property will lead to creating a novel interface that provides a good user experience even for non-expert users. In this context, this project aims to compare the user experience provided by a tangible interface that benefits from the range of movements and the independence of the index fingers to move a character in a game, to a traditional graphical interface that is the keyboard arrows. The proprioception in the human body makes animating using fingers more spontaneous than using a typical graphical interface especially for beginner users. As for the thumb finger, although it is very flexible, its range of movements is quite restricted. This might negatively affect the intended user experience for the particular movement involved in this game, therefore the thumb will not be used in this case. However, it can later be used for other actions in the game for scalability purposes.

Our research question is: Will using the flexing movement of index fingers as a tangible interface provide a better user experience than using a keyboard to control a game character?

State of the art:

In the literature, many papers have come up with new tangible interfaces to control 3D characters especially for games:

A Tangible Interface for 3D Character Animation Using Augmented Reality Technology [4]:
- Overview: This paper suggests a tangible interaction technique for 3D character animation. Instead of a graphical interface, users interact with a cube that has several markers to animate an AR character. Each side of the cube is responsible for an action: jump, walk, idle, and run. The camera detects the side that is facing it, and the AR character moves accordingly.
- Pros: users have tried animating using both the Sims and the cube and have found that the tangible interface was more satisfying, stimulating, and easier to use than the graphical interfaces.
- Cons: This tangible interface allows a restricted range of motions, because the cube has only four active faces. This approach is not scalable.
Hand Gesture Interaction with a 3D Virtual Environment [5]:
- Overview: This paper proposes a new vision-based way to control a game with hand gesture recognition in a 3D virtual environment.
- Pros: Compared with traditional HCI devices such as keyboards and mouses, this vision-based hand gesture interface provides more natural and quick movement and entertainment for the user.
- Cons: The system contains cameras that can infringe upon users’ private life.
Spatial Motion Doodles: Sketching Animation in VR Using Hand Gestures and Laban Motion Analysis [6]:
- Overview: This paper suggests another tangible interaction technique to animate a VR 3D character in a game using an HTC Vive controller. The users execute some predefined actions with this controller to create animations.
- Pros: This technique provides a fast-drafting tool for animators, and it is easy to manipulate even for non-expert players.
- Cons: Some gestures may be ambiguous since the actions are meticulously predefined. This can lead to multiple equivalent action sequences.
Real Time Gesture Recognition System for Interaction in Dynamic Environment [7]:
- Overview: This paper develops a hand gesture recognition system for interacting with different applications like image browser; games etc.
- Pros: it provides a fruitful solution towards a user-friendly interface between human and computer.
- Cons: The proposed hand gestures look uncomfortable and long-sleeved clothes may affect the process of recognition.

The hypothesis:

Our hypothesis is that the tangible interaction will provide a better user experience than the keyboard.

The approach to solve the problem:

In order to test out this theory we have followed a tutorial to develop a game in which a character has to avoid obstacles while continuously running [7]. The game is over if the character runs into an obstacle or falls off one of the sides of the road. A score is kept at the top of the screen to indicate the number of coins collected off the road while running. This game can be played using a keyboard’s left and right arrows. As for the novel tangible version of this game, we have replaced the controls with the left and right index fingers of the users.

In order to detect the movement of the user’s fingers, on our first try, we used two piezo sensors: one attached to the right index finger and the second to the left index finger of the user. Every time one of the sensors is pressed, a discrete analog signal is sent to the game through the serial port of the Arduino. Depending on whether this signal is sent from the right or the left index, the character moves correspondingly to the right or to the left. However, pressing the sensor did not generate the desired behavior consistently which interfered with the quality of the user experience.

We then decided to approach the problem differently by using the flex sensor. This sensor generates continuous values upon being bent. To ensure a better user experience, we attached the sensors to a pair of gloves. Each sensor is linked to one of the index fingers of the user. Flexing a finger sends a continuous signal to the game through Arduino. As opposed to the piezo sensors, the flex sensors provide the desired results. The sequence of movements ‘bend-unbend’ applied to the right index finger moves the character one step to the right, the same sequence applied to left index finger moves the character one step to the left.

The implementation:

For the purposes of this project, we used Unity to create our 3D game. This game engine provides a primary scripting API in C# that helped us retrieve and process the data coming from the serial port of the Arduino. This data is a continuous signal that measures the amount of bend generated by flex sensors which are the sensing element of our prototype.
Click here to acces our project repository.

The experimental protocol:

Five participants will take part of the user study. The experimental protocol starts with three minutes of familiarization of the users with the novel tangible interface. This is not going to be the case for the graphical interface with the assumption that they are familiar with it. Afterwards, we will explain the ‘bend-unbend’ technique necessary to play the game using the tangible interface before asking the users to proceed to play the game using both the keyboard and our interface. Some of the users will start the experiment with the keyboard arrows and the others will start with the gloves to avoid any bias. At the end of the experiment, the users must fill out a questionnaire.

In order to evaluate the user experience, we have picked the System Usability Scale (S.U.S.) questionnaire as a qualitative metric assessment system. This scale gives a global view of the usability of a system based on certain aspects:

Effectiveness: can users successfully achieve their objectives?
Efficiency: how much effort and resource are expended in achieving those objectives?
Satisfaction: was the experience satisfactory?

As for the quantitative metric, the score will be recorded. The score indicates how many coins are collected during a game. Depending on the chosen strategy, a player can either focus on collecting coins which could result in them taking more risks, or they can focus on both avoiding obstacles and falling to survive for longer. We decided to instruct the players to follow the score-oriented strategy since it is more reflective of the challenges of the game.

These metrics will help compare between the two interfaces in terms of user experience which will answer our research question.

Below is a demonstration of our work:

The results of the experiment:

For the quantitative metric, we recorded the scores of each player into an Excel sheet. We then put the results into illustrative graphs. For the first graph, we plotted a box plot that represents the scores by interface type.

The plot above shows that users achieved higher scores using the keyboard than using the gloves. 75% of the keyboard scores are below 40 although the highest score is 79. The glove scores have a similar distribution where most results are below 20 and the highest score is 63. To understand the reason behind the better results achieved by the keyboard, we broke down the results by individual players.

The figure above shows boxplots of the scores by player. Player 1showed visible and verbal discomfort towards both interfaces and had issues synchronizing the movements of both index fingers using the tangible interface. Eventually, their keyboard scores were better. As for player 2 and 3, they were more comfortable with the glove and even claimed that the keyboard was harder to control because of finger placement. They expressed that using the keyboard they had to look down from time to time to properly manipulate the character, while that wasn’t the case with the fingers. Both these players achieved similar results using both interfaces and one of them got higher scores with the glove interface. Player 4 and 5 showed a great performance with the keyboard. They informed us that they were professional keyboard gamers.

This plot shows the number of games in the time allotted to each player using both interfaces. The first player has the highest number of games during the five minutes, since they failed many times so the time spent in each game is very short. Player 2 and 3 played a similar number of games using both interfaces. Player 4 and 5 failed more times when using the tangible interface. They played less games using the keyboard because these games were longer.

For the qualitative metric, we calculated the scores resulting from the S.U.S. questionnaire and illustrated them through this figure:

Players gave a S.U.S. Score to the glove that is higher than the keyboard’s which means that the glove provided them with a better user experience. Although the average score of the keyboard is a little higher than the glove’s average, three out of five users preferred the user experience provided by the tangible index interface. Both interfaces fall into the excellent range of the System Usability Scale.

Conclusion and general discussion:

Overall, the keyboard showed higher scores in the game, but the S.U.S. average scores were similar for both interfaces. In fact, the comments in the form filled by the users showed that using the gloves made them very eager to play more in order to set higher best scores. Some users reported that the keyboard was a more familiar interface to them and that more training with the glove will surely make it the better interface. Moreover, all the users emphasized on the fact that the gloves provide a more fun experience than the keyboard.

The players who were beginners in playing computer games achieved similar or better results with our novel interface as opposed to the professional gamers. This means that this tangible interface is fit for beginner players.

Using the keyboard arrows to play has proved to be easier than the glove interface. However, most users gave better S.U.S. scores to the glove and all the users agreed that it was a more fun experience. Thus, we conclude that the new tangible interface provides an overall better user experience than the keyboard.