Augmented and virtual reality: innovative interaction techniques

Eye-Gazing interactions with Ring

Ivan DERYABIN

Thomas FAUCHÉ

Nathan REBISCOUL

Context

Interactions with the eye and gaze are very promising for the future of human/machine communication.

Most interactions with a computer are done with the mouse/keyboard or with a touchscreen. But those interactions involve the hand. And there are many cases where people would like to have free hands and still be able to use a computer.

Eye-tracking and eye-gazing would solve this problem, since it allows to point to a precise location, and it is intuitive to use.

It would be very helpful for factory workers, teachers, people that are working with their hands and still would like to use a computer, or are standing far from their computer. It would also be helpful for handicapped people, and also for daily-life usage.

Explanation of the problem

Eye-tracking is the process of measuring the position of the eye, whereas gaze-tracking is the measuring of the location the gaze is pointing to.

To be able to have an interaction with a computer, there are basically two tasks that need to be done : pointing to a precise location and doing an interaction (sending a "click" message). There are also the tasks of scrolling, different kinds of clicking, drag and drop ...

Eye-gazing and eye-tracking allows to point to a precise location, but there is now the problem to "click" on this location.

For example, we could click by blinking, but this falls into the Midas problem.

People have a tendancy to blink naturally and often, so doing an interaction when someone blinks poses a real problem : did the person blink because he wanted to interact with the system, or did he blink naturally ?

Related work

The concept of eye-tracking started during the 18th and early 19th century, with for example the works of Edmund Huey. People thought really early of using the eyes as a mean of interaction. A lot of researches took place more recently on eye-tracking, starting in 1982, with the article "Eyes as the interface", and continuing during the 21th century (some examples like "Eye Tracking in Human-Computer Interaction and Usability Research" in 2003 and "Eye Gaze Tracking for Human Computer Interaction" in 2010). This was for the researchs on eye-tracking and eye-gazing.

Now, for some commercial applications, there is for example a company called Tobii, that designs and produces eye-tracking glasses (mostly for academic research, but their final goal is to make computers more productive and intuitive). There are some other companies like Bitbrain (neurologic aspect of eye-tracking) and Eyegaze (tools for handicapped people), but eye-tracking glasses are still not mainstream.

Other companies focuses on the association eye-tracking / virtual reality, some sponsored by the GAFA, like Eyefluence (by Google) and Eye Tribe (by Facebook).

On a general scale, eye-tracking and eye-gazing still don't have a widespread use, but a lot of developpement is being done in a lot of different fields

Approach to the problem

Our goal was to replace the mouse with eye-tracking interactions. In order to do that, we had to find an accurate eye-tracking implementation, find a way to emulate the clicks without falling into the Midas problem, find a way to emulate the scrolling too, and test our new interaction technique against the mouse to be able to conclude on it's efficiency.

We decided to create a ring that will allow most of the mouse interactions : left and right click, scrolling. That way, the user only clicks by pressing a button on the ring, so this fixes the Midas problem.

Also, the ring is not like a mouse that needs to be on a table. You can carry the ring wherever you want (no need to be in front of a computer). Of course, in order for handicapped people to use our invention, they would need working eyes and at least a hand, so it won't work for people who lost both their hands (but there are other solutions for them, like tools from the Eyegaze company, which focuses only on helping disabled people, whereas we are trying to make a general tool for workers, daily life uses, ...

An other limitation we found for the ring is the fact that making a scrolling wheel is an extremly complex problem, some people have spend an entire year making one that works correctly. For this reason, we decided to use two buttons on our ring, scroll up and scroll down. It doesn't change the results of our experiments.

Then, there is the problem of the eye-tracking / eye-gazing implementation. We will go more into the details in the "Implementation" part, but to be brief about it, we couldn't use eye-tracking glasses, so we decided to switch to eye-gazing with a webcamp (so there is a need to be in front of a computer for it to work). It isn't what we were trying to do at the start, we wanted something that could work even when you are not in front of a screen, but for the prototype, we needed to change that a little bit.

What's more, we found several limitations to the eye-gazing tool we used, so we decided to focus more on the ring for our tests. Eye-gazing is a very complex problematic, and the existing tools are still not perfect and ready for commercial applications.

Implementation

For the eye-gazing implementation, we chose an eye-gazing javascript library called webgazer. Since it is a web application, it can easily be used in our project, and it shows the best results by far, although there are still a lot of problems (the accuracy is still far from perfect, there is a lot of noise, ...).

In order to fix those limitations, and to further upgrade the eye-gazing, we are doing some calibration on the user, and we are using a filter, the One-euro filter, to remove the noise.

For the one-euro filter, we are using a javascript function, and we choosed the hyperparameters manually to have the best noise-removal for our experiments (we didn't use the official way to choose the hyperparameters).

We are also using a Fitts's Law framework to test the pointing and clicking interactions.

For the scrolling interactions, we created our own testing framework, a pdf document of more that 300 pages, with buttons that need to be clicked on specific pages to measure time.

The ring was made at the FabLab, using four press buttons attached to a ring shape, and connected to an Arduino. The Arduino then emulates the mouse clicking and scrolling withe the "Mouse" library.

Experimental protocol

We have two testing frameworks :

One for pointing and clicking, in four different cases : mouse pointing and clicking, touchpad pointing and clicking, eye-gazing pointing and clicking with the ring, mouse pointing and clicking with the ring.

We are doing two tests with the mouse, one by clicking with the mouse, the other by clicking with the ring, because eye-gazing is far from perfect, and so we want to know if the use of the ring is as good (or even better) than just using the mouse.

We are then calculating the results by using the width and distance of the Fitts's Law's circles, which allows us to calculate the corresponding ID's and time taken.

The other testing framework is on the scrolling part, with two cases : either without using the scroll bar, or by using the scroll bar.

In the two cases, we test on the ring, the mouse, and the touchpad. As was stated in the Implementation part, we have buttons on different pages of a pdf document, and the user has to scroll to them and click the faster possible.

Results and discussion

For the experiments, we asked 5-6 people to test the scrolling part, here are the corresponding results :

Those are the results for the Fitts's Law part of the experiment. We couldn't test it with many users, because the eye-gazing didn't work very well, so we were the only one that understood how it works (since we spent a lot of time on it) :

Thanks to those results, we understand many things :

The eye-gazing always shows worst results than mouse or touchpad, due to it's innacuracy. But it also shows the biggest differences between results, which could mean that it could be better than the mouse for some complex operations.

For the scrolling test, the ring can compete with the touchpad and the mouse (it's even better than the mouse in some cases).

Basically, eye-gazing is not accurate enough yet, but very promising.

Conclusion

To conclude, eye-gazing needs to be more accurate, we probably would have had better results by using eye-tracking glasses.

But nevertheless, the ring is useful for clicking and scrolling, and if the eye-tracking / eye-gazing was better, it could be as easy of use as the mouse, and probably even faster for some tasks.

We saw during this project why there are still no big commercial applications of eye-tracking and eye-gazing. But we also saw why it is so promising. In the near future, eye based interactions could very well replace the mouse for human/machine communications.