Well in Air America he co-starred with Mel Gibson and the female lead was Nancy Travis. So who did Mel and who did Nancy? What does Robert do when it is a primarily male cast?
In Chaplin, he starred with Geraldine Chaplin so I guess she got a little bit of hat Robert Downey Jr. magic. The fact that she is 20 years older than Robert wouldn't have stopped her because as Robert puts it, "such intimate activity between co-stars is inevitable - because they spend so much together in such a close environment."
In Home For The Holidays, he could have had a reunion with Geraldine Chaplin and also thrown a little Anne Bancroft into the mix since she was the leading lady. I know some of you would say Holly Hunter, but I think Anne Bancroft was higher billed. Hell, knowing Robert, he probably went ahead and just did the whole cast. Had Steve Guttenberg calling him Papi by the end of production.
From this quote, I don't think multiple partners on one movie is too far off. "I remember certain colleagues having a lot of fun on the set with me." Hopefully he is talking about his time on Two Girls And A Guy and not US Marshals. I'm sure his wife loves the thought of knowing that Robert is off having sex with his leading ladies every night while she stays at home and wonders if he will stay sober during filming.