I own nikon D5100. I own 18-55 kit lens & just bought a 55-300 VR?

Now I want to add a prime lens. 50mm 1.8 or 35 mm 1.8. My main aim to get reasonable tight shot (with some surrounding but not too much) of my family on vacation. Will be basically using is outdoors. Taking the distortion and ideal distance from the subjects in consideration what will be the best lens for me. I tried taking some pictures at 35 & 50mm from my kit lens in my house buy still can't decide which will really be the best lens for outdoor family photograpgy on vacations.

Added (1). If I'm on a 50mm Lens and shooting a group of 3-4 people, what should the distance between the subject and camera be?

You kit lens is sufficient if you shoot outdoor. Prime lens is primarily used indoor or where light is extremely low.

Rest Top Contributors will tell you the right solution

Since you "tested" the focal lengths of 35 and 50mm with your zoom, you know that there's not a lot of difference in the focal length, and you should have some idea which one is better for you.

However, if everything seems to be equal in your mind and are having a hard time deciding, realize that with the 35mm lens - like all wide angle lenses - you will have a small amount of perspective distortion that the 50mm lens will not have.

It is hardly noticeable at 35mm, but it is there.

Here is a webpage comparing the 35mm and 50mm lenses, and shows the perspective distortion (although it is a Nikon 50mm AF-D lens).

http://www.althephoto.com/lenses/35mmprime.php

IF you wanted to use this lens for portraiture, I would avoid the 35mm lens due to it's perspective distortion. Otherwise, either lens would be a good addition to your camera bag.

The advantages of a prime lens are:
1: smaller and lighter
2: wider maximum aperture (lets you get better subject isolation and shoot in less light)
3: sharper image compared to zooms (of comparable price)

If you want to trim down your camera to make it easier to take on family outings, a prime lens may be just the thing, but if you're not overly concerned with subject isolation or shooting in low light (and remember, your D5100 is a very good low-light performer), it may not be worth the extra money; especially if you plan on toting your kit lens around with the camera.

Your D5100 has a very high resolution sensor. You can crop the pictures with little loss of detail at home, so a zoom lens isn't really necessary. On your camera, a 50mm would make a good portrait lens but is probably too tight to get a group photo with unless you can get very far away. I would recommend looking at a 30 or even a 24 if you want a single lens to take everywhere.