You must not be afraid of small bits of silence. To use it well is the height of confidence and skill for a comedian. It increases the tension in a good way and adds contrast like a curveball complements the fastball of a good pitcher.