Are there good reasons for not using virtual sites?

Dear all,
This is a very general question, but I could not find an answer anywhere. My question is, given than they can double performance (by doubling the time-step from 0.002ps to 0.004ps), are there good reasons for not using virtual sites?
I know that there are force fields for which they are not implemented. And that implementation can be tricky. But when they are available, for “standard” MD of a protein system, should their use be discouraged for any reason? To me they have a sounder physics base than other techniques, such as using heavy hydrogens. I am surprised that, in general, they are not as widely used as I expected (that is my subjetive opinion, of course), considering the efforts that we all make to getting good performances.
Cheers,
Ramon