I guess it depends how you define slow. I've never been big on VIM but have used it a number of times. My biggest complaint with it is going in and out of insert mode. With emacs, as long as you keep learning more and more about it, all the shortcuts become muscle memory and you can whip through things pretty fast.

In my experience, the extensibility of emacs was why I kept using it. I don't really notice any "slowness" compared with VIM. It's not enough to make me go to VIM anyway. But either way, to each their own. It's probably best just to know one really well and know the basics of the other. I've never been big on the whole debate. They each have their pitfalls and benefits