This name for the character & is surprisingly recent, not being known before the late eighteenth century, though the character itself was in use long before printing was invented. It started life as a Roman scribe’s abbreviation of the Latin et, meaning “and”, and became common in the early medieval period. It was later taken over as an abbreviation for the English word and.
In some modern italic fonts (this is ITC Baskerville) the form of the ampersand is deliberately reminiscent of the medieval scribe’s style. You can see flourished versions of the e and the t.
Ampersand is a contraction of and per se, and. This sounds extremely peculiar, but it’s a continuation of a medieval convention in which Latin per se, by or in itself, was often added to those letters that could stand alone as words: A, I and O (as in “O for the wings of a dove”). A per se, a meant “a by itself makes the word a”. Since it stood first in the recital of the alphabet, in the fifteenth and sixteenth centuries it came to mean a pre-eminent person or thing:
In the one, that is for lands and possessions, you have companions many; but in the other, my good lord, you are A per se A with us, to our comfort and joy unspeakable.
In a letter to Lord Russell from John Bradford, 1554.
It was common enough that at this period it was contracted to apersey, meaning the first, unique, or most distinguished person or thing.
It was usual in the eighteenth century to have children end their recital of the letters of the alphabet with &, because it was so common. It was read out as and per se, and, which meant that the symbol &, whose name was and, stood by itself and actually meant “and”.
As it came at the end, people sometimes said from A to ampersand to express the whole extent of something, as today we would say from A to Z:
At length, having tried all the historians from great A, to ampersand, he perceives there is no escaping from the puzzle, but by selecting his own facts, forming his own conclusions, and putting a little trust in his own reason and judgment.
Gleanings through Wales, Holland and Westphalia, by “Mr. Pratt”, 1795. This, by the way, is also the earliest known use of ampersand. (Thanks to Fred Shapiro for finding this.)
In time, the character became known by this phrase, which became slurred through rote recital and oral transmission into all sorts of dialectal and variant forms, including anparse, empus-and, emperzan and amperzed. It was only in the 1830s that one form, ampersand, became dominant and conventional. By then, the old rote way of learning the alphabet seems to have been on the way out:
The expression and per se, and, to signify the contraction &, substituted for that conjunction, is not yet forgotten in the nursery.
A Glossary, or Collection, of Words, by Robert Nares, 1822.