It can make a huge difference, I used to have a Trace Elliot 4x12 that always sounded far too dark for my tastes. One JCM900 2x12 later and I'm a very happy bunny. It can still get rumbly and deep as fook, but it's got a great top end if I need it.
I think it had a lot more to do with the difference in speaker type and cab build than the 2 fewer speakers- I've heard people comparing Marshall 4x12s and 2x12s and not noticing a huge amount of difference.
Generally, you wanna match your amp's impedance (ohmage) with the cabs, you'll get the best sound and efficiency this way. Some people swear by closematching the amp's RMS rating with the cab too, but I've yet to be convinced how much of a difference it makes.
Drag yer amp to a few decent big music shops and hear the differences.
What RMS and impedance is the speaker in the Hotrod? Closer matched to your amp's rating? The Hotrod is also openbacked, so it'll make for a looser bass response, and generally sound a little more mellow.
You may be seeking an open backed 2x12, dude!