"Of course I didn't forget," the Doctor remarked, very matter-of-fact, as if it was the most obvious thing ever. Duh, Clara. "I never forget."

And that's true. He tries his hardest to remember all those he lost, all those he traveled with him, all those who fought besides him. Because if he forgot one of them? The Doctor really doesn't know if he'd be able to forgive himself.