Should I switch from Skim milk to Whole milk?

I am trying to avoid clogging my arteries. I am drinking skim milk. I am trying to build muscle and I read that whole milk is better for overall health and nutrient absorption. Are there any adverse effects in drinking whole milk (1/2 gal day max) compared to skim? This brand has no hormones.