Should I have reacted?

That's tricky ... only you know the full cirstances, but trying to defend yourself is often worse than saying nothing... that's the time for either silence, or a perfect witty reply. Let it go... most men are attracted to pretty women - doesn't mean a man will betray his own brother! The awkward silence was likely because they were embarred for the idiot!