This is something that the nimblest standup comedians learn, over time, to handle gracefully. They'll go between prepared material, then they'll respond to what's happening in the room and weave it back into the prepared material and so on.