There is no 'thing' to let go of, but a concept, an idea of an ego that burdens us. As soon as we posit a 'thing' to let go of, we're in trouble. We need to change our view of reality, not attack a nonexistent entity.