What are the ideal settings to take advantage of the hardware decode abilities of the processor and still steer clear of any features that would require software decode or inability to correctly decode in hardware?
For example, I know that B-Frames are often not handled well (or at all) on many mobile devices and the same is true for the CABAC encoding scheme.
To answer your question, any H.264 encoding will work perfectly on the Raspberry Pi. So you have to select (whatever profile you pick, best high profile or normal) the H.264 option in the tab Video under Video codec. For audio use AAC (supported) and pick stereo (Dolby surround doesn't work).
Using those settings (tested it myself) everything plays nice and smooth, no problems at all. Other options like frame rate can influence the movie a lot, a guide on frame rate: https://trac.handbrake.fr/wiki/FramerateGuide
I think that will help you, remember video codec on H.264 and audio on stereo AAC. Check your frame rate and other options will generally affect the size of the video.
For future reference of anyone who's looking a detailed answer to this same question, please refer to this post on the Raspberry Pi forums: https://www.raspberrypi.org/forums/viewtopic.php?f=2&t=4668
Summary: The RPi supports H.264 Level 4.1 High Profile. If you select these options on the Video page of handbrake, the Pi should play them.
You should make sure that the audio is encoded with AC3 rather than DTS, or you might run into playback issues: see this question: Difference between H.264 and x264 and how to play it
Cheers.
