(Original article November 2001; updated and expanded in February of 2009)

The story goes that the current QWERTY keyboard layout was designed some 150 years ago specifically to slow down typists enough to keep the mechanical keys of 19th century typewriters from jamming. Whether this is true or not, the fact is that the keyboard represents the bottleneck between the human mind and the computer. With computers gaining more and more processing power and becoming ubiquitous in our society, the keyboard bottleneck becomes ever less acceptable. You could argue that this ancient method of communicating with machines is one of the greatest detriments to productivity we are facing today.

As a result, there have been many efforts to develop alternate ways of communicating with computers. The mouse was invented to provide an easier way to navigate a new generation of graphical user interfaces. Handwriting recognition was pursued to provide a more natural way to enter data for people not familiar with computers and keyboards. Speech recognition was seen as a way to simply speak to a computer instead of typing or writing.

Well, the keyboard is still here. And that is because making handwriting and speech recognition work turned out to be more complex than anticipated. However, both handwriting and speech recognition technologies have come of age and are now ready to be used in place of the keyboard.

The primary problem is that people working on these new technologies have spent most of their time getting the basics to work. With handwriting recognition, that is a set of algorithms that result in reasonably accurate recognition under a variety of circumstances. The same applies to speech recognition. The latest speech and handwriting recognition products running on the latest hardware will yield good results to a person willing to learn how to use the product and play by its rules.

Where the picture breaks down is in how these new technologies are used and applied. Recognizing handwriting alone is not enough. The software must also work in the environments we generally use, and it must provide easy and natural ways to edit and correct our work. The same applies to speech recognition. Reasonable accuracy in interpreting commands or even dictation is of little value when corrections and editing still require a mouse and a keyboard.

In addition, these alternate input/editing technologies have generally been treated as separate projects. Handwriting recognition tries to do it all: recognition, editing, and correction. Voice recognition likewise tries to do it all with voice. Thus, in addition to a lack of truly functional editing tools in each technology, each also seeks to be the be-all-end-all in replacing the keyboard.

As far as I am concerned each of these alternative input technologies has some advantages and some limitations. Dictating to a computer can result in faster data entry than typing or writing. But using verbal commands to edit, tap, point, and drag is not optimal. In contrast, the pen represents a clearly superior mobile pointing solution for performing editing commands by drawing gestures or symbols. What we have here are two technologies with complementary strengths and weaknesses that have the potential to be combined to form a revolutionary way of interacting with computers.

One such approach I have seen comes from Natural Input Solutions Inc. of Canada. They recognized this dilemma and developed the prototype of a hybrid solution. After analyzing each aspect of existing handwriting and speech recognition technologies they isolated the respective advantages and disadvantages of each recognition technology and created their Natural Input Software project, a solution that combines the best of pen and speech recognition technologies into a system that is more than the sum of its parts.

It is not easy to describe Natural Input in a sentence or two, or even a full review as what Natural Input does is employ and merge the respective strengths of two alternate input technologies, augmented by a number of pen-based editing and correction features that are more intuitive than anything I have seen to date.

The common "active modal shift" approach