It's really, really difficult to do that without understanding (intimately) all the components in play from top to bottom.
I disagree. If a component is essentially a black box, where I know the inputs and I can test to check I get the expected output in known conditions, then that's enough. It has to be, otherwise there wouldn't be any way to use a middleware, or a third-party API, or anything too complex for one person to understand.
There are lots of software businesses where no one fully understands the internals of the product. That's fine. So long as there are people who expertly understand the part they're responsible for, and the APIs are well documented, everything will work great.
The problem with devops is that it gives people the impression that everyone can be responsible for everything, with everyone on the team having a hand in the design, the code, the testing, the deployment, etc. For a sufficiently complex product that is just not possible. At some point it gets too big, and people have to start accepting that they can't do it all, and that they'll have to trust their colleagues to get things right.
This works great when it works great. Black boxes everywhere. But then you run into some issue, like, I dunno, your monitoring service is telling clients the site is going down, even though you can't find any evidence of this.
If you know how to go deeper then you run a packet capture and start seeing that, whatdya know, your top-3 IaaS provider's network is shit and is often dropping packets, so those timeouts are actually occurring, though not on anything you control. Now you're able to try to setup a repro, then open a ticket with enough details to get something done. Versus opening a ticket along the lines of "Hi Top-3 IaaS provider. Some monitoring service says our site doesn't always respond, but I'm sure it's not me. Please fix your network, thanks."
Not knowing how things work at a deeper level leads to experienced teams working on high-profile sites to come up with stuff like: "We have 2 domains in use, but need to go SSL only. We only have a certificate for one domain. So please implement a DNS-level redirect from domain A to domain B to avoid browser SSL errors."
Or an experienced dev insisting that a major vendor is shipping invalid binaries because the runtime says so. And insists they aren't metadata-only libs, but isn't comfortable opening up the binaries to verify and thus spends a day trying to figure things out another way.
Being able to get to a baseline of what's working is a fundamental step in troubleshooting. Just like you would always ask a home user to make sure stuff is plugged in, verifying lower-levels of the stack are correctly operating should be a common step - it's unfortunate how common it is that something else is broken under the level you're working at.
You're confusing 'needs a deep knowledge of how components work' with 'needs a deep of knowledge debugging'. A suitably senior developer should definitely be able to deduce where a problem is most likely to be happening, and figure out a way to test it.
In the case you're alluding to, it'd be much more useful to know general information about why any monitoring service might think any IaaS provider is not responding, and have the skills to test that. Knowing details about the components you're using (eg AWS and Pingdom) is much less useful.
I disagree. If a component is essentially a black box, where I know the inputs and I can test to check I get the expected output in known conditions, then that's enough. It has to be, otherwise there wouldn't be any way to use a middleware, or a third-party API, or anything too complex for one person to understand.
There are lots of software businesses where no one fully understands the internals of the product. That's fine. So long as there are people who expertly understand the part they're responsible for, and the APIs are well documented, everything will work great.
The problem with devops is that it gives people the impression that everyone can be responsible for everything, with everyone on the team having a hand in the design, the code, the testing, the deployment, etc. For a sufficiently complex product that is just not possible. At some point it gets too big, and people have to start accepting that they can't do it all, and that they'll have to trust their colleagues to get things right.