Nikon or Canon? (Or other brands)?

I want to get into photography and videography. I was looking at the Nikon 5300d but heard it was awful for shooting videos. I'm on the fence now on what to get. I have a budget of around $750.
I understand I should start with a beginner type camera but I don't want to spend money on a camera that is very poor quality.

If you want to shoot videos, get a video camera, not a DSLR like the D5300. If you want to shoot photos, the D5300 is just fine.

For still images, I would want a Pentax.

For video, look seriously at the Canon Vixia series.
This line still offers mic inputs and headphone outputs ---
two of The Most Useful features a video camera can possibly have.

Expect to spend a bit more than $750 to get both a still camera
and a video camera, but the added cost will be worthwhile.

You are correct, Nikon has really awful video performance except in their D500 which is $1,799.
Pentax is a very good DSLR, but like Nikon, has bad video performance.

Canon is known as the best DSLR for video, but you must understand that this only applies to their top-end models like the $3,000 5D Mark IV. So with this in mind, be careful about which model you buy if video is a big deal. The Canon T7i and the SL2 are among Canon's entry-level DSLRs around the $750-$800 price range with a lens. The 77D should be better for video, but like the T7i and SL2, it's not ideal because none have a headphone jack.

The Sony A6300 (body only) is $750. Like Canon, it has an external mic jack, but no headphone jack. BUT! Unlike most DSLRs, the A6300 can shoot in 4k. Therefore, IMO, the Sony A6300 is the best camera for someone who needs excellent results in photos and video. But really, the better solution is to buy a good camera for stills and have a camcorder for video. If video is going to be a seldom requirement, the don't buy a digital camera with video. Instead, get a camcorder which will be vastly superior in terms of features, capabilities, lack of limitations (e.g. Over heating) and overall video performance. No DSLR at any price can compete with a $700 camcorder.

A dedicated video recorder is best.