Absolutely this is the case. But not to single Agile out unfairly: Although people are doing a lot of great work researching processes and so on, Software Development as an industry is like this.
Consider Waterfall. If you try it, and it doesn’t work, proponents will tell you that you were doing it wrong with all the changes late in a project. If you try Ruby and your code is a mess, Ruby programmers will say it isn’t the language, you failed to write good Ruby code.
And so on, ad infinitum.
So, if I agree with you, why did I post this? Not because I assert that Agile will “work” if you just do it right, but that I think it’s wrong to modify it and then blame it for not working.
Come to think of it, if you do it “right” and then succeed, I also think it’s wrong to give it the credit. But that’s another story.
"If it didn't work, it's because it isn't really agile."