The confounding factor is milk fat. In my experience higher fat milk just tastes better regardless of any other factor and milk straight from the cow will have up to 5% milk fat compared to 3.25% for "whole" milk. Try drinking a shot glass of 10% cream sometime, it's amazing.