Chunks of his life fell away, so that while we were moving ahead in time, he was moving back.
or simply: