The real issue lies in the North beyond the Wall. Stannis becomes one of the few characters fully to understand that, which is why in spite of everything he is a righteous man.