When the thing is finally sparked, the white man is not there - he's gone. The merchant is not there, the landlord is not there, the one they consider to be the enemy isn't there. So, they knock at his property. This is what makes them knock down the store windows and set fire to things, and things of that sort.