Why are a lot of high end lenses' widest aperture only f/3.5, f/4?

My 15mm 2.8 ef fisheye was stolen a while back, so today I just decided to check some out (mostly just "window" shopping). I noticed that the expensive L series version of the fisheye went only to f/4, and I noticed several lenses only go to that.

When I look for lenses, I tend to look for 1.4-2.8 for versatility. If I need a wide aperture for low light, I want my lens to be able to open up. That probably comes from the times I've had to shoot dark concerts, where a high ISO wasn't enough (and the even higher ISOs were just nasty). Furthermore, I just hated my kit lens at it's widest f/3.5 aperture. It might be fine for landscape photography, when you want the entire image clear and you're shooting at f/11 or f/22 anyway, but for portraits. Not so good.

So why do several of these expensive high end lenses have that same widest aperture? Are they built so differently that it is still excellent in low light, and still provides nice depth of field, or is it that their intended purposes simply doesn't call such a wide aperture and people are using them at narrower apertures anyway?

Canon's L series lenses are hand made (ground by hand). You are paying extra for the superior optics, and the extra time that has gone into producing them. Time = money so to speak.

The wider the aperture, the more grinding is required. Again Time = Money.

There's a video on youtube which shows the process

There's more to it than wide apertures. Get a cheap lens and stop it down to f/4 at a given focal length and compare your shots with say a Canon 70-200L f/4 at the same focal length using the same ISO and shutter speed. Often you'll see differences in exposure because the canon lens will let in more light at a given aperture. Not all f/4 exposures are the same; each lens is different. In other words f/4 on a good lens may yield the same exposure as f/3.5 on a cheap lens.

It all comes to benefit vs cost.

The difference in cost between the Nikkor 300 mm f/2.8 and f/4 is nearly $4,500 or about 4 times what the lens with the one stop smaller aperture.

If and only if there was a Canon EF fisheye 2.8, you can be assured that it would cost at least 4 times $1,499 of the f/4 lens or close to $6,000, How many photographers use a fisheye lens enough to justify that kind of investment?

The f/#'s tell us the ratio of lens diameter to its focal length. An f/1 lens will have a diameter that is the same as its focal length. Thus a 50mm f/1 will have a maximum working diameter of 50mm An f/2 15mm lens will have a maximum working diameter of 7 mm. Lenses are generally segments of a sphere. Think about cutting a slice of an apple. The cut is flat, the natural shape of the other side is a section of a sphere. To make a lens that operates at f/1.4 or f/2 or f/2.8 lens, the curve of the glass at the edges must be quite sleep. This is difficult to achieve as there will be lots of glass to grind away. Odds are when grinding off the glass, accidents and miss shapes will occur. In other words, lots of rejects will be the result. Additionally, the steep figure of the lens induces aberrations (lens defects) that must be corrected. To correct, the lens is made using several different density glass elements and several different shapes. The final lens will be a mix of positive (convex) and negative (concave) lenses. Some will be cemented together and some spaced apart. The bottom line is superior lenses require high precision both in the design and fabrication stages. Fine lenses are the no accident. They cost more because they are hard to make.

Quality goes beyond just wide apertures. But the wider the aperture, the higher quality the glasses need to be at a bigger range so it gathers light uniformly enough to avoid chromatic aberrations and such.
There are many high quality lenses at f/3.5 and f/4 but they usually have an even more expensive high quality lens with wider aperture.
Example:
Canon EF 70-200mm f/2.8L IS II USM --> $2500
Canon EF 70-200mm f/4L USM --> $709

The f/4 one doesn't have a wide aperture, but is made up of better glass than non L 70-200 lenses, just not all the way to support f2.8 apertures. Since the f2.8 gets lit at more parts of its lenses, it needs to have a quality built through every part that gets lit when the aperture is wide open.

Hope I've made sense