Those who dislike fantasy are very often equally bored or repelled by science. They don't like either hobbits, or quasars; they don't feel at home with them; they don't want complexities, remoteness. If there is any such connection, I'll bet that it is basically an aesthetic one