"I was so selfish," the actor, 43, admits in the new issue of Parade. "I liked to drink, and I had a drug problem, and that didn't jibe with Sarah Jessica, because it was the furthest thing from what she is."
At 19, "we quickly moved in together and played house," he reminisces of their life in L.A. "It was idyllic."
So did his drug addiction.
"She tried to help me," he says of his Sex and the City star ex. "She was so miffed when I didn't get my act together.
"Sarah Jessica would pull me out of a hangover," he recalls, "and we'd go pick out furniture together. She is a force of nature!"
Parker eventually ended their relationship after seven years.
"I was in love with Sarah Jessica," he says, "and love clearly was not enough. I was meant to move on. And, after some heartache, she was meant to find her home with a great star."
(In 1997, Parker wed Matthew Broderick, whom Downey says is "a lot more grounded than I ever was." He also calls their 5-year-old son, James Wilkie, "a great kid.")
Now clean and sober, Downey — who has a son Indio, 14, from his 1992 marriage to model Deborah Falconer — is wed to producer Susan Levin.
"She told me, 'I'm not doing that drug dance with you. I'm drawing a line in the sand here,'" he says. "She was absolutely clear about it.
"That doesn’t mean that other women, business associates, movie directors, insurance companies, judges and law enforcement hadn’t been clear about it too," Downey adds. "It was that, before I met Mrs. Downey, I just didn’t give a godd--n.
"What changed is that I cared."