Duff has some serious regrets about meeting the iconic Prince, but it seems he actually made a solid impression