Issues with camera shake using dslr?

I'm using a Nikon D3200 with the kit lens 18-55mm currently, and have VR turned on.
Yesterday I went for a shoot at a local nature reserve, and found that when returning home many of the images had some camera shake in, which made many of the photos useless when zoomed in or cropped.
I was using an ISO of 400, generally keeping the aperture at f9-11 and the shutter speed then adjusts accordingly, it was usually between 40 and 100.
I was wondering what I can do to reduce camera shake, as I understand reducing the aperture can reduce sharpness and increasing the ISO can increase noise, and of course carrying a tripod is not always convenient.
Any help is appreciated, and many thanks in advance!

You will have to increase the shutter speed; 40 to 100 is quite slow, especially if you are using a zoom lens.
The unofficial rule with zooms is that you set the shutter speed to the focal length.So, if using a 300mm lens, you need to set 1/300th second. With a 200mm lens, select 1/200th second.a and so on.

Unfortunately with the faster speed you will get a wider aperture and less depth of field. So you might have to be more selective and focus more on a central subject and let the rest on the scene fall out of focus.

Increase the ISO; you will not notice much or any noise if you go to 1600.
I take your point about the tripod, but many wildlife photographers do manage to use them.

Consider a monopod or a bag of beans, these are far easier to carry than a full size tripod.

Using f/9-11 seems excessive to me. The lens should perform reasonably well at f/8, whatever little gain there may be at corner sharpness at f/11 certainly won't offset the motion shake problems caused by a stop less worth of light.

After all, opening up to f/8 from f/11 will enable you to double the shutter speed.
Here you can directly compare the performance of your lens on test charts at different aperture and focal length settings. Just move the mouse over the test photo to switch between the two settings:
http://www.the-digital-picture.com/Reviews/ISO-12233-Sample-Crops.aspx?Lens=662&Camera=614&Sample=0&FLI=4&API=3&LensComp=662&CameraComp=614&SampleComp=0&FLIComp=4&APIComp=4

You need something like a 300mm lens (450 in terms of the old film cameras) to photograph timid wildlife without needing to crop too much. Birding people will often use top quality 400 or 500mm lenses plus 1.4 teleconverter (like1200 in old terms).So as you're going to need to crop so much with your 55mm lens you need to eliminate camera shake and also subject movement.
Cameras with the newer 24mp sensor (Nikon D5200, Sony 58-77 etc) will help here.

So it's a balancing act unless you can get close to the subject. I use a basic monopod that is simply a telescopic hiking pole with a tripod mount or head on top. I'd also choose the 'sweet spot' of the lens - probably f8 - and try to have the shutter speed at 1/250 to avaid subject movement even if it means the ISO will be higher.

If you don't like the idea of a tipod or monopod then think about a Gorrilla Pod that you can fasten to objects like fences, trees and rocks. You can then take the photo with the 2 second self-timer to further reduce shake.

F/9 and f/11 are relatively small making shutter speed slow at 1/40s to 1/100s. If you want to get blur-free shots, set to aperture priority and use the smallest f/number possible. That will force the camera to use faster shutter speeds. Ideally you should use at least 1/500s but you can get decent action with 1/250s. Of course shooting at the thousandths of a second would be the best. If you want to go faster, raise ISO a bit. You should be safe up to ISO 1600. Higher than that you'll get noticeable noise or grain in your pics.

To clear things up, lowering f/number increases aperture size. Raising f/number makes the aperture smaller. Shutter speeds are normally fractions of a second so 1/100s is faster than 1/40s. If they were whole numbers, you'd easily catch on. Whole seconds are denoted by quotes (1", 10", 30").