An interface exploration for the Apple TV

Direct manipulation using a trackpad. With videos and prototypes!

Update (08/10/2015): Called it!

The new Apple TV is eternally imminent. People like me are waiting eagerly for it, since the current Apple TV UI hasn’t gotten any sort of relevant refresh in a long while, and it leaves a lot to be desired. I got tired of waiting. I’ve had this idea for a while now, and I finally decided to sit down and take a stab at it.

Every time Apple comes out with a new product category, it usually brings along a new interaction method. Here’s my idea for an interaction method on a remote, big screen.

Direct Manipulation

Remember what was great about the original iPhone? You could slide your finger across the slab of glass and the pixels below it would respond as naturally as technologically possible to your actions. You could pinch photos to zoom them and scroll lists like never before.

“You had me at scrolling”

If there’s one great thing about iOS’s UI (and there are many) it’s UIScrollView. It embodies the principles of natural, direct manipulation of objects on the screen with a very responsive and smart scroll experience and very natural-feeling inertia calculations, with bounces and everything.

Let’s go back to the current state of the Apple TV interface. Clicks is the name of the game. There’s 4 directional buttons and a center button on the physical remote, and there’s a Remote app on the iPhone that does its best to translate taps and swipes into discrete clicks — the only input method the current Apple TV can understand. It’s actually the Remote app that inspired this whole thing.

Wouldn’t it be great if instead of trying to turn finger gestures into artificial clicks, Remote acted more like a trackpad — directly responding to the position of your finger on the tracking surface? Selection in a complex button layout — such as the on-screen keyboard —would be much more usable if you could make the cursor move in any direction, instantly. What if, instead of relying on an app for this, the new Apple TV remote control had a smooth, clickable (or force-touch enabled) trackpad, that allowed for these behaviors without having to look at a second screen?

Well, that’s the whole idea. I thought it would be worth a shot, so I built a prototype of the interaction model that you can try using an iPhone and AirPlay. I don’t know if it’s the definitive answer for Apple TV interaction, but I found it to be better than the current state of things.

(This interface paradigm has been done before, but it’s only available to TimeWarner Cable subscribers, so you know it won’t ever have the global reach the Apple TV has. I probably won’t ever touch it, since I’m from Chile. Still, this is a fun exercise that shows the untapped potential living within the Apple TV today, and you can try it out for yourself, if you’re a developer.)

The Remote

This new interface is to be driven by a new physical remote. This remote would have a large tracking surface, plus some other buttons that would aid navigation, like so:

The proposed remote next to its current incarnation for reference

The big delimited area would be a ‘click-pad’, a trackpad you could click, not unlike the MacBook’s. It’s wider, to allow for comfortable finger tracking, making room for 3 buttons below the pad instead of just two, breaking down ‘MENU’ to a clearer pair of Back and Home buttons. It even has less buttons than the current one — 4 versus 7. I’m not an industrial designer, this is just a simple proposal based on the interaction model.

The Prototype

This is an interaction demo, built for the iPhone and AirPlay, that aims to show how a remote like the one described above might behave while controlling an Apple TV. It’s not particularly pretty and doesn’t do stuff. It simulates some current Apple TV interface patterns to evaluate whether or not direct manipulation works better than discrete clicks. (I think it does.)

I know you just want to see it in action, so here it is:

If you want to try it out for yourself, you need to be a registered Apple developer in order to compile apps to your devices, an iPhone to run the app, and an Apple TV to AirPlay to. You can grab the code on Github. Make sure you enable AirPlay Mirroring from your device to your Apple TV before you run the app. Trackpad clicks are simulated by lifting your finger and tapping on the tracking surface — on an actual device, you wouldn’t need to lift your finger to click.

Although these demos are simple, you can see the potential for more complex interactions using edge gestures, for example. Here’s what I thought a Now Playing screen could look like with a sort of Control Center popping up from the bottom, invoked using an edge gesture:

A back gesture — for example, swiping to the right from the left edge, like navigation controllers on iOS — could also be a possibility. The entire interface could not only get a new coat of paint, but also a quicker, more modern way of interacting with it.

Addendum (Dec 21): A clarification

Some people have expressed confusion towards the difference between my proposal and what the Remote app does today. As I explained to Dave Mark on The Loop, this is, first and foremost, about how the Apple TV handles control events.

For context, iOS on the iPhone and iPad can handle 3 major kinds of events: touch, device motion (shaking, rotating, etc), and remote control (for the tiny remote controls on the cords of headphones and the like). The Apple TV only really understands one major kind of event: a discrete button click.

The Remote app today most likely has a large UIPanGestureRecognizer, which as the name implies, recognizes panning gestures on the surface of the view. As you swipe on it, Remote does its best to translate your intention to discrete button clicks that the Apple TV can understand, even with an attempt at simulating inertia. This is a good-enough approach for a lot of use cases, but it does not provide the precision of directly manipulating scrolling views or cursors using remote touch events in a trackpad. As an exercise, try to use the Apple TV on-screen keyboard using Remote (hide the keyboard on the iPhone) and compare your experience to this prototype.

On the same vein, I would also like to remind readers that this is designed to be used with new hardware; the new remote design pictured above is an essential part of my proposal. This theoretical remote has a touch-sensitive surface that you can also click, which cannot be simulated on an iPhone at all (unless it had force-touch, like the Apple Watch). This has the potential to provide an even more fluid interaction, where you don’t even need to lift your finger to type a letter on the keyboard, for example. Clearly, if this ever became reality, a new Remote app should provide the same remote-touch functionality plus an even easier to use keyboard like it does today, but this is not about controlling the Apple TV with an iPhone.

The feedback and reach of this article has been impressive, which further underlines that users expect way more from an Apple product than what they are currently delivering on the Apple TV. Hopefully this serves as a friendly nudge that brings us closer to the great user experience we’ve been longing to see on the big screen for some time.