Video function for Nikon D5100?

What option (example: scene, macro, program, aperture priority, etc.) should I use to record videos on my Nikon D5100 that has the best quality?

You have little choice. I have the D5000. HD will get you five minutes video. Then there's two LD modes. READ YOUR MANUAL! It has to be in the video mode! The others are for still shots.

Start reading your manual on page 107. When you get to page 109 it will tell you how to set it to 1920x1080, 30 fps.