When we don't get that escape from our work selves, I think we feel its absence on a deep, almost primal level. Leisure is uncommodified, unoccupied time where we get to be truly free, so feeling bad about missing the weekend isn't just, "Damn, I didn't make it to the mall!" It's a profound loss.