Magic was not always around. The first that was able to weild it were given the power from the dragons. With the help of the dragons, the elves learned to control and weild the power bestowed upon them. This, however came with a price.
The dragons did not gift the magic to the elves freely. They demanded the elves' obedience. This demand eventually led to the Dragon Wars, where the elves fought long and hard for their freedom from their tyrannical overlords.
To this day, there are still dragons, though their numbers are not as prominent as they once were. Most hide in with the population of citizens, disguised as humans and elves. Most dragons are still angry at the elves for their disobedience, holding the grudge for what is now thousands of years.