This reminds me a lot about discussions for the endurance sports that I have participated in (running, not anymore and cycling, 25+ years). The professionals always say that weights are a bad idea because it trains you to get used to going slower. The key is to ride fast, then supplement with weights at a different time, not when riding because it bogs you down and doesn't let you move freely...at least for cycling.
Now, I understand that this is not cycling, but perhaps the same concept applies. I think it is safe to say that the best players and the fastest players in the world have not had a need to use weights while practicing...they play, and a lot, and the speed and strength develop over time. I have been playing congas and djembe for about 20 years and people have always asked me how I move my hands so fast. So, I show them the trick, practice your technique nice and slow, get it down and FORGET about going fast. Just keep playing and it will come