No one has ever said they thought the keyboard and monitor was a great way to work with the computer.

20 years ago I was at conferences where impish GUI iconoclasts (pardon the expression) from PARC were advocating hyperbolic displays, motion sensing responsive screen surfaces, etc. The history of interface novelty at SIGGRAPH began with the first conference. MIT had 3-d surround, gesture sensing displays in the 80s.

Wired musicians like Tina Blaine http://www.jamodrum.net/ have built spatial interfaces to midi controllers that whole audiences could play with at once, like beach balls with pressure sensors that used rf connections to synths...churning out interactive music with unique and creative interfaces.

Nintendo commercialized a vr glove, for Engelbart's sake.

But none of the solutions have been, as they say, "sticky." Nothing comes close to the universality of the keyboard and the mouse.

And nothing will anytime soon.

The short reason is simply that every non-keyboard interface requires learning as much novel muscle/nerve coordination as it takes to learn to type. If you learn to type, you can type on a million keyboards. If you learn the multi touch interface of the apple iphone, however, you only can redial your grocery delivery service on your iphone. If you have to borrow a friends Blackberry, you are in an entirely different, multidimensional haptic universe. That requires typing.

Its one thing to create the intricate, robust and efficient mapping of intent to gesture to result that a satisfying interface design requires. Its another problem altogether to get the 100th monkey to learn it. Or even the second one.

Maybe Apple or Microsoft can impose a standard by brute force. Apple sort of succeeded in nudging Microsoft, and thence the globe, into the WIMP standard created at PARC in the 70s. But Microsoft intentionally scrambled some of Englebart, and Brenda Laurel's original brilliant clarity, for product differentiation. In so doing, they dumbed down the beauty of the original interface concept.

Making an interface intentionally inconsistent, as you can find in any version of Word, for example, is like introducing diseases into DNA. You can't replicate and generalize from success. Every encounter with a new task requires rote memory. These issues are simple, and at root, political, not technological.

Its like the 19th century Spanish generals who convinced the Parliament to require all railroads in Spain to run on a different guage track than the French, to make it harder to be invaded.

So, Surface makes great Utube video. A lot of other cool combinations of projection and touch sensitivity do too. But until Ebay puts out a Surface application that allows you to stack and sort your collector Barbies and Speak and Spells with haptic versimilitude, the Surface is just going to be one more fancy place to set your Metropolitan vodkatini while you size up the hot singles in the lobby.

Wow, that is a really thoughtful and interesting post! I learned a lot from it - I didn't know or think of interface that way. Thank you.

You are welcome.

The other critical U/I piece of the keyboard/mouse which makes it sticky, is the ability to multi-task.

Our fingers are extremely sensitive, and touchtypists don't need to spend a lot of time thinging about what they are typing (or looking at it). Thus an average typist can type much much faster than they can handwrite.

This means that if you want to have a conversation with someone, or look at what they are showing you, WHILE interacting with the computer, the keyboard is the best way to do this. The "smart-table" lacks this ability. It demands tight focus from you. And that means it has to provide enough additional functional payoff that it overcomes the extra energy required to interact with it.

But unfortunately it fails in this. For example, all the "objects" that it knows how to "recognize" require a barcode or RFiD tag on the back. That means that you have to take the time BEFORE hand, to tag and encode each object into the system. Either you end up with a very limited set of objects (enough so its a cool demo) or you end up doing a lot of tedious work. In either case the payoff in functionality isn't there.

As with most everything, the hardware won't be of much use without the software. However, it may not necessarily require people like eBay or Google to come up with it. Third-party applications that can act as the gopher between the application and the hardware would make it useful. You don't necessarily need eBay to come up with a visual list of your auction items. So long as you can have, essentially, more than one browser open for your account, there's no reason why you can't have an application that would be your broker and have it interact with eBay to produce a mashup of your catalog. Of course, that means that eBay will have to "play nice" and let you do that.

And while it would be nice if everything were barcoded, what could help as a stop-gap measure is easy access to a USB port. If you don't have a barcode on your iPod (which doesn't have wireless capability anyway), no problem. Just plug it into hub and the system should recognize what it is. Put it down on the table and the system should know that what you just placed on the table is the thing you just plugged in (and if it's confused, it can ask...highlight an object: "Is this your iPod?") Now you can not only sync up your playlist but also recharge it.

The next step is to have the Surface be a central dataport for the home network. If you're storing your data on a computer somewhere else in the house, there should be an easy way to have Surface hook up to that networked storage.

And if it is going to be a coffee table type appliance, then it needs to be a way to interact with the television. That is, have the TV guide come up on the table, scroll through the listings to find what you want, point to it, and the TV tunes to that channel. Advance into the guide to have the DVR record shows. Universal remote capability so you never have to wonder where the remote is: It's the coffee table. Integrate AvantGo-like features into it where it will download things that you find appropriate to acquire (like the local movie listings) so you can get a list of what you need quickly.

But all of this tends to assume that the table is always on. If that's the case, Surface needs to learn a trick from PowerBooks and be able to go from a sleep state to on and fully ready to use in one second or less (the goal of the first OS-X PowerBooks was that the computer would awaken in the amount of time it took you to raise the lid from shut to open.)

I don't think the problem with the adoption of Surface is going to be the interface. The problem is going to be the applications that take advantage of it.

Way too much work in almost everything you describe.

1. Mashups are a pain to do - they require actual work to mash stuff together and there isn't anything "in it" for eBay et. al. to play nice. Sure Mashups are easy for us tech geeks, but they are too much hassle even for someone like my wife who was one of the early pro desk-top publishers and who brought computer managment to the animation studio. She just isn't that interested in working WITH a computer
2. And if its confused it can ask - Sorry, that's too high a hassle threshold. Think about the reaction to the LUA Escalation in Windows Vista
3. Applications - TV Guide, central data clearinghouse etc. _ this is all going in the WRONG direction. Cheap CPUs are the Gutenberg press, and you are suggesting that instead of ubiquitous computing, we instead go back to the one-book/household bolted in one place.

That last I suggest is the biggest problem with this. It is NOT a step in the direction of ubiquitous computing and hence I would argue it is a step in the WRONG direction

“No one has ever said they thought the keyboard and monitor was a great way to work with the computer.”

Oh, really? Let me be the first, then. The keyboard and monitor IS a great way to work with a computer. It is certainly the best way we have developed so far.

‘Course, I’m slightly, well, old. The first computer I worked with was programmed using patch cords, and the output was blinking lights. Next I graduated to punch cards – that really sucked. After that I finally got a keyboard, a teletypewriter. What an improvement! I could type instructions directly into the computer, and the computer would reply by typing responses directly onto a roll of paper.

Finally, someone figured out how to use a TV-type monitor as a substitute for paper, and the world of writing and number manipulation was changed forever.

Not great? You gotta be kidding. 99% of the worlds computing involves writing words and doing arithmetic. Word processing and spreadsheets, in other words. While voice recognition software has improved immensely, if you want to write a novel, or a resume, or figure out what next quarter's earnings will be, you will be best served by a keyboard. Learning to touch type beats the heck out of any hardware upgrade in terms of productivity, as I always tell my students.

Of course you can find niche applications that work well for touchy-feelly gizmos like Surface and the like, but I’m sure keyboards and monitors will dominate for decades to come.

And get off of my lawn.

Bogolov.

Or maybe just a preference.....

In Truth I believe that all of the interface models requiring physical interaction will simply become obsolete in few short years when neural sensory interface takes over. There are models in the works right now that can respond to thought process..it's just a matter of time before we can think a word onto the screen and soon thereafter an entire paragraph.

Congrats though on a great run of excellent contributions to this new Fray.

jo

Did you not read "third-party application" in my post? I'm not suggesting you make the mashup. I'm suggesting that a third party will create an application that will do it for you, much the way Zillow does the mashup between Google Maps and real estate listings.

And are you really suggesting that it is "too high a hassle" to have the computer ask if, after having a device hooked up to it and it noticing that two things were put down on the table at the same time, "Is this your iPod"? Hell, if the camera recognition is as good as it claims to be, chances are it won't need to ask unless you put two identical things down. An iPod doesn't look like a PSP, after all. Since it knows you just connected an iPod to the machine, it should look for something that has the outline of an iPod. Only if it sees two things that look like an iPod would it need to ask.

And your whine against applications makes absolutely no sense at all. Did you not read the part in my post about the need for easy networking of the various computers in the household? Don't just identify the object as an iPod. Identify it as Billy's iPod as opposed to Janey's. That way, it will find Billy's repository of music on the network and allow him to sync up his playlist downstairs even though the computer holding the information is upstairs.

You seem to be completely ignoring the environment in which the computer exists. If we're going to have a coffee table in front of the TV, what sort of fool would claim that it shouldn't be able to bring up the TV guide and control the TV and DVR? If we're going to put one in the kitchen, it should be able to bring up a list of groceries that you're going to shop for. But that one is most likely to be mounted on a wall which will make putting your camera on the surface a bit difficult to do. The applications built for a wall-mounted Surface will be much different than the one for a table-mount, even though the interface will be the same.

And again, there's no reason for those two not to be networked together. Download the local movie guide once and display it everywhere. And with the ability to recognize your credit card, you can simply place your card on the table or press it to the screen (there's that need to comprehend the physical environment in which the computer exists, again) to have it buy the tickets for you.

Nothing of what you said makes any sense.

Hint, the surface doesn't use visual recognition. Thus it can't physically recognize the shape of an IPod (I've played with this thing - though admittedly about 2.5 years ago).

And yes, having the computer ask IS too high a hassle factor in my experience. Not for everyone of course - not for techgeeks nor for gadget hounds. But for tech-savy but non-computer focussed folks like my wife - yeah it is (I say this from watching her and other willing users of other technologies).

The problem that has faced "easy networking" has invariably been that of namespace auto-creation. This was a significant part of what doomed CEBus andmakes X-10 such a pain to work with, and why corporate IT pays Network admins $50k/yr. Network topology is a pain to create and automanage. And the more devices you throw onto the net, the more of an active management task it becomes.

Where it really becomes a barrier to the average consumer comes about with the mismatch between their "Mental Model" and how the system actually is configured to work. So either you take the time to make that mapping acccurate (ie name the devices in your son's bedroom Billy'sX,Y and Z rather than Bedroom#1X..Z) or you allow that cognitive disconnect to cause irritation everytime the behaviour isn't intuitive. The former has too high an upfront hassle factor and the latter a too high runtime hassle factor.

The same applies to this surface.

The problem with putting a TV guide and DVR control onto this surface - even if it is a coffee table in front of the TV is twofold

1. Its in the interests of the Consumer electronics mfgs to NOT provide closed-loop inputs into their devices (this was the other major nail in CEBus's coffin) - which means that you will have as much success in providing predictable DVR control across a broad set of vendors as Crestron has and your corporate AV support folks have in conference rooms (every conf room has its own set of AV instructions none of which are intuitive)
2. A flat horizontal surface requires that I be looking down at it - yet my TV is up in front of me. What I really want it to do is to project my U/I onto the TV display surface, and that's exactly what solutions like MediaCenter, Slingbox do more effectively - obviating the need for this surface. Same applies in the kitchen - unless the Refer can scan the barcodes of ALL the contents (including the leftover tubs - which don't have barcodes), a paper list of grocery items needed is more effective (My cousin is VP at a major Consumer Appliance co and we've had these discussions in depth).

Surface just is too single-minded a device that doesn't provide enough benefit over its negatives

Now have you played with this? Are you a Microsoftee? My guess is Yes to the latter given your responses, but I'm open to being wrong.

Hint: What do you think the five cameras under the table top are doing?

Hint: How do you think it reads the barcode sticker on the device just by having you place it on the table top?

Hint: How do you think it is capable of knowing where on the surface you have put the object and thus be capable of putting an outline around the object? And then track it as you slide it around the surface?

You're trying to extrapolate an experience you had two-and-a-half years ago to today. If you truly are someone who understands technology, you should know that that is a lifetime ago. Things are not as you remember them. Pay attention to what the demo told you it can do. It reads a barcode off the object. If it can read a barcode, it can read a shape. And if you just hooked up an object to it via cable, then it knows that it should expect to find an object of the same shape as what was just connected to it on the table.

And if it can't do that, no problem there: If you connect the object via wire rather than wirelessly, have it put up a picture of the object on screen rather than having you put the object on the table. It will act as your surrogate.

And you seemingly don't actually own a DVR of any sort if you think there is a problem with having a TV Guide in the Surface. You seem to have completely forgotten about the Media Center edition of Windows. It already has the TV guide. C'mon, degsme, you aren't this naive.

And yes, the TV can show you the guide, but we all know that it sucks big time to try and navigate through a TV-based interface since the only control you have is that rocker switch on the remote control. If the highlight is at the top of the screen and you want to get to the bottom, you have to traverse every single button in between. And forget about typing. Everybody is reduced to hunt-and-peck because nobody is going to use a remote keyboard...which is yet another remote to add to the pile...and a huge one at that.

And you clearly haven't had any discussions about a wall-mounted screen in the kitchen. What on earth do you think a printer is for? You use the one in the kitchen to make up your grocery list and then tell it to print it out. Hell, if you're shopping at a place like Stater Bros., you can even print out a list that will tell you which aisle to go to in order to find your items. Why drag the laptop into the kitchen?

You haven't given this any thought at all.

• The problem with the guide on the surface is the same problem digital cameras with no viewfinder have: it requires a shift of focus from the point of interest (TV in this case) to the surface and back again. That's why TV remotes do everything they can to give ergo feedback on what button you are pushing - so you don't have to look. Using the surface is a step backwards.
  So while I hate the U/I on our PVR (Dish, not Tivo), I would hate having to use a Surface on the coffee table more
• In the kitchen, I have a running grocery list on the side of the fridge. That way anyone going to the store can tear it off or add items as needed. No logging in/launching an app etc. When an acquaintance blogged about putting a flat panel in his kitchen, I politely nodded, but really, other than allowing his wife to watch TV during the day I don't see the ROI in utility. Even from a cookbook perspective, unless its a ruggedized tablet, a cookbook is more portable and durable during the prep of a complex meal, and bookmarks or multiple cookbooks work better than a single U/I.
• Barcodes can be recognized with laser scanners and don't need a camera - but I stand corrected on the 5 cameras built into the underside. Even so, those cameras are gonna have a tough time distinguishing an Ipod from a Zune from a Zen from my Pocket PC from my Pocket PC Phone. And again, the human tolerance for error by machines is very very low. The moment it has to regularly prompt is the moment it gets ignored.
• the only objects I am willing to cable these days are cameras, simply because of the bandwidth required is such that blue-tooth is useless and unless you go to really high end cameras, you don't find WiFi as an option (although that is changing). Otherwise cables are a pain no matter how bad the bluetooth stack is.

As for thinking about it I actually have given this thought for a very very long time, in fact I got paid to do it for a number of years. And my take on this is the same as my take was on CEBus, and your response is similar to what I got about CEBus.

First point: Huh? Have you not actually worked with an interactive TV Guide on screen? That feedback you get is put right in front of you. What on earth makes you think that you're not going to get that feedback with the TV guide in the coffee table? And since the guide on screen from current methods either shrink the TV signal to a postage stamp or cover up the TV signal completely, your complaint about a "focus shift" is meaningless: You weren't watching the TV to begin with. The only "focus shift" is the trivial one from the TV to the Surface. You do it once and when you're done, you go back to the TV. You're making it seem like you'd have to continually shift back and forth between the two.

In fact, I'd say that splitting the guide away from the TV is an improvement precisely because it means you don't have to shrink or cover up the TV screen. You can work on programming that special that'll be on next week without having to compromise the program you're currently watching.

Note, this doesn't mean that everything involving interacting with the TV should be done using the Surface. Having a handheld remote for things like pressing Channel Up or 10-keying a channel and the like is much more elegant than having to use the Surface but that's because you can memorize the physical placement of the keys on the remote so that you don't have to look at anything. But for things that require using the whole TV screen, why not use an interactive screen at your fingertips that leaves the TV screen untouched?

Second point: Huh? What is this "logging in" and "launching an app" thing you're talking about? Why on earth would you ever make someone log into the refrigerator? Why on earth would you make getting to the grocery list anything other than punching the grocery list button? Don't confuse your inability to design a decent app with a universal incompetence. They already make refrigerators that do this sort of thing...LG, if I recall my brand name correctly. You don't log in. Moving through the different functions is a single button press. Why on earth would anybody try to make it more complicated?

And you again seem to have forgotten the point of a printer. You're quite right that a cookbook is better than a multi-tasker, but that's pretty much for one reason: It's paper and paper is something no computer can match for its ability to be light, easy to carry, don't worry if you drop it properties. But that simply means that the Surface in the kitchen is your organizer...you go through your recipe book, find what you want, have it print out your shopping list (and again, with some THIRD-PARTY mashups to the store's web site, it'll even tell you where to find what you need) and cooking instructions, and go from there.

Third point: Huh? How many times do I have to say, "You just plugged it in" before you remember it? The system will be quite capable of telling a Zune from an iPod...you have to plug the iPod in. And since you just announced to the system that there's an iPod in play, what do you think the system ought to do when it sees an iPod-shaped device put on the table? The Zune has wireless capabilities and thus it identifies itself.

And since you ignored it the last time, let me say it again: If the cameras aren't sophisticated enough to do this, no problem: The peripheral has just announced itself to the system either through the WiFi connection or from having been PLUGGED IN to the system but it has no bar code: Rather than having it try to detect it when you put it on the table, put up an icon that represents it. And we could even design a process by which you could put the thing on the table, and associate the new object with the icon. Those who find that "too complicated" are not left out: They have the icon they can use.

Fourth point: Huh? When did Apple release a Bluetooth iPod? When did cameras start having Bluetooth? When did wireless power become a consumer item? If you want to power up and sync your iPod, you use a cable.

In a cradle.

Hmmm...seems like you wouldn't really put it on the Surface at all. Well, that gets rid of the need to barcode it. And thus, we're back to the solution of putting up an icon that represents it.

You're just riffing. You really aren't thinking about solutions to your "problems."

As I said, I've spent a fair number of years working on these issues. Focus Shift is an issue. Its not that you necessarily were "watching TV" - although the ability to preview the channel you are about to switch to via PiP makes even that a specious claim - but your focus of attention is on the TV.

The appeal of on-screen guides - especially ones that allow you to continue to watch the current broadcast, is that your focus doesn't change. And the remotes are designed to make it easy to do that without looking down.

If you think focus change isn't an issue, you've never spent any time working in a "fish tank" usability lab.

As for the rest, when you can get over your childish attempts at insults, perhaps we can discuss this further. You clearly don't understand the issues at play here and are trying to make up for it by insulting bluster.