You are allowed to draw lines even if your feelings are irrational. Part of the marvelous business of being an adult human is that you get to set your own boundaries for whatever reasons you like, without appending a sensible rationale to them.