Meditation is not a selfish thing. Even though you're diving in and experiencing the Self, you're not closing yourself off from the world. You're strengthening yourself, so that you can be more effective when you go back into the world.