He must have known I'd want to leave you." "No, he must have known you would always want to come back.
or simply: