A sincere thanks. As a software engineer I couldn't care less about what happens to my company's services/products outside the 9-5 time range. Don't get me wrong, I give myself 100% at my job, keep myself educated regularly and I'm rather on the "boring and stable stuff" side of things (instead of the "shiny/trendy and unstable" side). I have commitments outside work and no amount of money is going to make me give more than the (already exhausting) 40h/week my contract states. The "you build it, you run it" may work for people on their 20s (they usually are excited to earn "easy money" by being oncall). For people on their 30s and above the extra oncall money is not worth at all.
I certainly believe that's true for you. But in the case where engineers choose not to ever run what they build, how do you reconnect the feedback loop?
Put differently, I think one of the ways somebody goes from the "shiny/trendy and unstable" side to the "boring and stable stuff" side is by experiencing the operational pain of their choices. If the pain falls on others, will they still learn?
Of course, the way you talk about your job makes me wonder if you are already experiencing so many systemic/managerial issues that there the feedback loops are already pretty broken, so this one may not make a ton of practical difference.
Engineers can run what they build during normal working hours.
Oncall is a scourge not because of the experience of technical problems, but because people already working full time have to arrange their lives outside of work around a second "oncall job". A job which occurs after hours, one out of every X weeks.
A dedicated, pure "Ops" night shift (perhaps in another time zone) would be more humane.
Even if it were built perfectly, if engineers are still on-call, they would have to arrange their after-hours time around the possibility of an incident.
That's true but it's just a reality of being employed by a saas company these days. Customer support, sales, etc have those too (and usually less formalized and unpaid) so why are engineers immune to this? You can still probably find some shops that ship an offline distribution but that's becoming more rare.
> But in the case where engineers choose not to ever run what they build, how do you reconnect the feedback loop?
Personally, if I get paged at 3am due to a bug, I'm going to fix it regardless of what the 'backlog' and 'prioritisation' and 'sprint goals' and 'feature roadmap' and 'product owner' say I should be doing.
But some would say I should not be bypassing the process in that way, and that the feedback loop of external stakeholders making requests to the product owner is more than sufficient.
> If the pain falls on others, will they still learn?
I think that depends on the seniority of the individual/team. In my experience, of course one can still learn.
To give you a real example: years ago one of our systems went down on a Sunday morning and our team had no oncall people. The infrastructure team was the one who fixed the issue (don't remember the exact underlaying issue, but it did make clear one aspect of our service we didn't properly: signal handling). Next morning the team wrote down a Jira issue to improve the way we handle signals. Ticket got prioritized very high and was fixed the very same day.
Now, what would have happened if the issue that Sunday morning was due to a bug in the software our team wrote? The same thing. The difference is that infra team would have no clue on how to fix the thing and would have to revert the service to a previous stable version. Would the business be fine with it? In our case, yeah. As a matter of fact, they didn't want to spend the extra money hiring ops people for each team to be on call. You see, if the business really cared, they would immediately have hired a software engineer willing to be on call... They just didn't care that much (and they couldn't force the current team to be oncall because our contracts didn't specify so and the average age in our team was around 35, and nobody wanted to be on call).
I believe they can still learn if they are senior enough and compassionate enough. And if they have management competent enough to let that work. But what percentage of teams would you guess fit that? I suspect that leaves a lot of on-call staff suffering from bad software.
> I certainly believe that's true for you. But in the case where engineers choose not to ever run what they build, how do you reconnect the feedback loop?
My company goes with option 3 from the list, “It’s not part of the job outside business hours, but we might still try to reach you during those times.” and it's working fantastically for us.
One out of eight weeks my only job is handle alerts, incidents, and questions from other teams. I and my coworkers dedicate this time to burn down technical debt and to add documentation to the codebase. This keeps error rates quite low on its own but we combine this with a scheduled release cadence (3 times a week) and a reasonably sized QA team that tests the major feature flows of our product for each release (~1 QA per/ 25 engineers) and an ops team whose job is built around being available to rollback to a previous release.
Even though we're a large company with millions of DAU most teams get pinged out of working hours less than once a year. My experience here has really pushed me to the opinion that continuous delivery has been hugely destructive for most of the industry, eroding our pool of experienced engineers who can't be bothered to do on-call.
“How about the whole team makes engineering decisions as though you’re unable to contact us after hours, or as though doing so were particularly costly.”
What, and break down all the monitoring and alerting silos we built by hiring a Devops engineer to come in and break down the development and infrastructure silos that were built when the company went ham adopting "Capital A" Agile?
