The Down Side. Switching to almond milk may be beneficial when you're training for fat loss, but it could have the opposite effect if muscle gain is your goal. The higher calorie content of cow's milk, be it skim or whole, can help increase your calorie intake and boost muscle growth.read more >>