There is a learning disability in mathematics named Dyscalculia. Current research suggests that Dyscalculia may be the result of mild impairment in the brain areas involved in mathematical cognition. However this impairment may be able to be remediated, especially at a young age.
Dyscalculia affects people from across the whole IQ range. Estimates of the prevalence of Dyscalculia range between 3-6% of the population. Recent studies show that Dyscalculia is even more common than Dyslexia.