Couldn’t you do the reverse – specify an everything-but-ampersand range for the body fonts and an underlying fancy one for the remaining ampersand?