I would suggest drinking your whites warmer than you currently do, and your reds should almost certainly be cooler. 

A simple rule for whites is that the fuller and more complex the wine the warmer it should be served. A few examples…

Sparkling wine – classically served out of an ice bucket. This is fine for cheap and nasty fizz when all you are after is giggles and a headache. The problem with serving better wines this way is that the glacial temperature will mask the aroma and numb the tastebuds. Serving direct from the fridge is just fine, perhaps a little warmer if you are indulging in something really posh.

Aromatic Whites – Riesling, Sauvignon Blanc, Chenin and the like are all quite happy in the fridge, no colder. 

Full bodied whites are perhaps the hardest to get right. They need to be served cool, not cold. Pulling a smart Chardonnay or Marsanne straight out of the Kelvinator will not show it to its fullest potential, yet too warm and they can come across as flabby and alcoholic. I tend to take these wines out of the fridge 20 or so minutes before serving to allow them to warm up a bit. This medium temperature is also ideal for Rose and light reds.

Red wines are more straightforward with serving temperature used as a balance to bring out the best in the wines. Cooler temperatures highlight the refreshing lifted characters of young light reds, yet mask the depth and complexity of anything more interesting. Big reds used to be served at room temperature, but that was before central heating and without the Australian climate in mind. In the colder months keep the wine in an unheated part of the house before serving. In summer you may need to juggle in a bit of fridge time. The bottle should feel slightly cool. If served too warm full-bodied reds can come across as hot and tannic.

Like everything in wine enjoyment, give it a go but try not to get too obsessed. The idea is to present the wine at its best and that means with a smile.