The magic wand should have modifier shortcuts for adding/removing selection just like the marquee selection (Shift to add, Ctrl to subtract). If possible, I would also like these shortcuts to be exposed for changing. I would personally add Shift + Ctrl for intersect, for example.
Good suggestion, thanks!