Other technologies
Here are some of the other fields in which interface and interaction work is being done:
Gestures. Although the idea of issuing commands by making gestures dates back to the 1970s, it was popularised by PDAs such as the Apple Newton MessagePad and to a lesser extent the far more successful range from Palm. To delete some text on a Newton, you scribble over it with an up-and-down zig-zag action. The caps lock command when entering text with the Graffiti alphabet on a Palm is a pair of up movements with the stylus. With the new Tablet PC, wiggling the stylus pen above the screen brings up the virtual keyboard.
More recently, they have shown up in Web browsers. Since browsing is largely mouse driven, the idea of being able to issue commands without tracking all the way to the tool-bar or menu bar (or using keyboard shortcuts) is attractive.
The lead in this are was set by Opera. In that program, most gestures are performed while holding down the right mouse button. The gesture for -previous page" is to move the mouse left; for -next page", move the mouse right. Slightly more complex gestures include moving up then right for -maximise window" and down then left for -minimise window".
The Mozilla community has also picked up this idea: the Optimoz add-on provides gesture support for the open-source browser. Some of the gestures match those used by Opera (such as those for previous and next pages), while others have been added, for example up-down-up to refresh the current page without using the local cache. While moving the mouse left matches the left-arrow on the back button, the link between up-down-up and refresh isn't so obvious.
The concept has also been implemented at a system-wide level. Sensiva's Symbol Commander maps gestures (including letter-shapes) onto commands such as running a particular program or saving a document.
Symbol Commander is available for Windows and Pocket PC, which is especially convenient for people who divide their time between full- size and handheld devices. Sensiva is part owned by Toshiba, and the latter will be shipping the gesture recognition software with its Portege 3500 Tablet PC.
Handwriting. Like speech recognition systems that are not dedicated to a particular task, handwriting recognition seems to work well for some people but not others. The primary answer to this problem has been the development of special alphabets for stylus-oriented systems, notably Palm's Graffiti system. For example, the letter M is drawn in the -golden arches" style, while A is drawn without a crossbar. The recent announcement of Tablet PCs is likely to renew interest in handwriting recognition for everyday applications, but it is unlikely to be a significant feature of desktop or notebook computing.
The Inkwell technology in Mac OS X 10.2 supports handwriting recognition as well as command gestures via a graphics tablet.
Avatars. One approach to humanising the user interface is to give the program a human face. Animated agents or avatars appear to be a positive feature for many users, though a lot of people were glad to see the back of Microsoft Office's Clippy. The Ananova newsreader was a relatively early example of a fully animated avatar, but some organisations are working with static images bearing expressions appropriate to the program's response. An example of the latter is Eve, the virtual service agent on eGain's Web site.
Animated faces make people more comfortable, according to Zac Jacobs, director of business development at Famous3D. Apart from Web-based avatars, this Australian company's technology is also applied to animated characters for film, TV and computer games.
-We can put a lot of personality into a character," says Jacobs. One way of applying facial animation is to provide a virtual assistant to help people fill in forms. -When you make a mistake in a field, the character can spot the mistake and tell you how to correct it," he says. Characters might adopt different demeanours to suit the situation and the user demographics: contrast -Mate! You've left some digits out of your mobile number" with -That mobile number doesn't look quite right, would you check it, please?"
The technology need not be limited to canned responses. It is quite feasible for contact centre agents to conduct a conversation with someone via an avatar. Furthermore, Famous3D is working with another company to back avatars with AI.
Jacobs expects the Famous3D technology to be in use on over 10 Web sites by the end of the year, including a major company in the financial sector. Avatars can also be used on intranets, and one very practical example is a -virtual receptionist" in the lift lobby of each floor in a building, ready to provide directions to particular offices or other facilities, or to summon a real person when necessary.
There are over 100 licensees of the company's software. Famous3D is also working with a Japanese company to put avatars in mobile phones to deliver information such as weather reports or the result of Internet queries. -The phone is such as visual medium these days in Japan," Jacobs says.
Research into the science behind avatars is ongoing. For example, an avatar that fails to smile naturally has a negative impact on users, according to Eduardo Chavez, associate lecturer and Web designer and developer at the University of Technology Sydney's Faculty of Information Technology.
Chavez' analysis of the eye expressions that accompany a genuine smile could be used to make avatars more realistic in a way that could have a significant affect on their acceptability to users. However, the relationship between the eye and mouth movements in true smiles is not simple.
Mood recognition. As if it wasn't enough to have software pulling your strings by displaying an appropriate facial expression, work is underway that will result in programs responding to your mood as displayed on your face.
NCR's Teradata division and the University of Southern California (USC) are collaborating in a project to explore ways computers can store, interpret and use human emotions.
-The idea is to capture the face, annotate it with dots and regions, so we can then process and match it with our database against a catalogue of emotions and have the system react the way a good salesperson might," says Dave Schraeder, technical expert at Teradata.
USC and Teradata see two particular applications for this technology. The first is to improve customer service by having systems react to expressions. For example, if the user is squinting at the screen, the device might switch to displaying text at a larger size.
NCR's interest in the project is understandable, since it is a major manufacturer of ATMs, automated checkouts and related devices.
The other area is health, where the technology might eventually be used to help diagnose various emotional and medical problems. USC clinical psychologist Dr Skip Rizzo suggests it might augment a therapist's skills when dealing with geographically remote clients via the Internet. -We still have a long way to go with this, but I believe that by tracking facial expressions it gives us added information that the therapist could use to get better insight into the patient," he says.
Teradata suggests it will be three to five years before these -e-motional" systems reach the market.
Just because a computer can't see you, don't assume it can't tell how you are feeling. Mitel has patented a system for detecting a person's mood from their tone of voice, choice of language and other cues available via a phone line, such as the speed at which numbers are pressed when responding to menus.
The idea is to allow contact centre systems to direct incoming calls to operators best able to handle people in a particular frame of mind, but the concept could have broader application. For example, voice-based applications might be able to detect increasing levels of frustration when someone is struggling with a particular facility, and deliver a brief, carefully focused tutorial.
But why not do a better job of designing the primary user interface in the first place (or reduce queuing times), so users don't get frustrated with your system?
