Can't play Nikon d5100 video file using vlc player, gom player or quick time player?

Quick time plays it very badly, i converted the file into another format using youtube downloader, but the quality turned to be a lot lower than the original file?

now is there any way to change the file format using the camera itself? Or maybe a software that plays the video on my PC properly?

No, you can't change it in camera. Cameras produce a particular file format, and that is it.

Your footage may simply BE bad or perhaps your quality settings in the conversion program are set too low?

Converting via a "youtube downloader" sounds like you're asking for cruddy quality… Try a couple of different converters, I bet you can find a better one.