Prince Andrew once secretly smuggled blacklisted Sarah Ferguson into the royal residence, defying Prince Philip, according to insiders.