Basically what it means when two pickups are "out of phase" with each other is that the waves that they are sending to the output jack are interfering with each other in a destructive manner.
Check it:
The first example is constructive interference. That's what you get when two waves are in phase (this is what it means to use more than one pickup on most guitars). The second example is destructive interference (out of phase; when your Mustang switches are set opposite each other).
That diagram is a little too simple for our purposes. Imagine that your pickups emit like really complex versions of those waves above. When they are out of phase the frequencies that they have in common (due to their placement, angle, et c.) drop compared to frequencies that are unique to each pickup. Usually this results in an overall decrease in volume but also accentuates certain characteristics of the pickups that would be difficult to notice otherwise. All in all it's a pretty sweet tool to have at your disposal.
