Couldn’t you use a unicode range that excluded the ampersand with Arial, and list that before your ampersand font. That way the natural fallback would be Arial for everything?