Microservices are a solution to a social problem, not a technical one.
A team of N engineers requires N² coordination. Large teams get mired in endless meetings, email, design reviews. Small teams are more effective, but struggle to maintain large systems.
Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination.
Yes, microservices add complexity and overhead, but this approach enables a large organization to build and iterate on large systems quickly.
> Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination.
This does not happen at all. When you break a system into subsystems, all the previous connections that get remapped to new connections between subsystems still need to happen, in order to solve the fundamental problem that the system solves — except now instead of just making the connection directly, there has to be a "cross-functional" meeting between teams and a complicated communication layer between the systems. And if somehow you find a breakdown that requires minimal connections between subsystems, then those connections wouldn't have existed in the original system either, and the N² problem doesn't exist.
It's all fun & games until product wants to add a feature that doesn't map cleanly to your micro servies architecture. Then you end up hard coding you services into a macrolith. Good times.
If that's the experience then you're doing services wrong. Each service should have its own datastore and a single API. The interface between services should be a single connection.
There should be maybe one meeting where the caller defines what they need the service to return to them.
The problem with this is that you have to be really damned careful how you split things up. If your separate data stores end up having to be joined together later on because some new business feature requires them to be cross-referenced, you're painted into one of two corners:
1. Merge the two services (and their data stores) into one and cause havoc downstream of either service
2. Burn through your network latency/throughput budget trying to reinvent a DB join across RPC boundaries (and god forbid if you can't batch multiple lookups in a single API call!)
Exactly. Software developers will never learn that the separation of concerns is a myth. In reality, UI, business logic and data are deeply entangled. Hence, moving these things apart makes everything worse.
I do not like appeal to authority either. But based on my experience, most of these issues were resolved in one to two meetings (I said maybe one, after all).
The team that needs data goes to the team that has that data and says "can you make an API that vends the data that I need?". Then they make it. And then maybe there is another meeting with a follow up, or perhaps an email asking for the data to be presented slightly differently.
But if you come to a team with a good use case and some examples, it shouldn't be too hard for them to create an API that vends the data appropriately.
This of course is all predicated on a good microservices architecture where you already have systems in place to handle API discovery and communication between services and proper data separation.
your "argument" was literally just telling the person that they were naive for thinking things could ever be handled by one meeting.
turns out, they inarguably have a whole lot of experience at high levels in the biggest companies in the world. Either they're straight up lying, or it is in fact possible to resolve things with a single meeting. Why wouldn't that be the case?
I can tell you many other wisdoms: code should be tested, code shouldn't contain bugs, everything should be done before deadline, etc.
The thing is in reality it's not that easy.
> Either they're straight up lying, or it is in fact possible to resolve things with a single meeting.
Something can be done in a single meeting but not everything. But you gave me only two alternatives without anything between so there is another useful link for you https://en.m.wikipedia.org/wiki/False_dilemma
oh sweet summer child; have you not heard of the fallacy fallacy?
snark aside, just because a logical fallacy exists in an argument, it does not mean the argument is untrue. My experience echoes jedburg's - services should maintain their own datastore and provide api access. This allows them to be decoupled and allows the service to alter their datastore as needed. When multiple services leverage the same underlying datastore, things can get overly coupled. This becomes increasingly important as organizations grow because the coupling means more overhead communicating and aligning around changes.
Jedburg is an expert because he has seen this happen in multiple companies. I have too, but I don't have the speaking/consulting breadth of experience that they have. Credentialing and appeals to authority work because we are squishy humans and we can't verify everything so we lean in on others. That does not mean that authorities are always right, but when their experience and advice mirrors other's experience and advice, there is something to it.
And for the record, in public companies (two of them), I have exactly seen this play out: a well defined api that does not allow others to muddle in their datastore leading to _single_ meetings where new functionality is discussed and agreed upon and later implemented. I have also experienced the pain of multiple teams sharing a datastore when the datastore now needs to be altered which led to dozens of meetings across several teams and a rollout plan that was measured in months and quarters due to coordination needs. I can firmly say that the latter is a much harder (and expensive!) problem to solve in each case I've experienced.
> Of all the things wrong with micro-services this isn't one of them.
It is, exacerbated by the over-use.
> The "complicated" communication layer is always either REST/JSON or GRPC.
Going over network, which is slower, unreliable, poorly typed. It's orders of magnitude more difficult to refactor a published REST API in comparison to e.g. Java interface within a monolith.
Microservices are generally far from easy to debug. In the best case scenario you have the whole stack locally and can put breakpoints, add logging immediately. But that happens rarely, debug port is often blocked (security), you can't easily modify the code to add some diagnostics without a complex CI dance etc.
But I've never worked on a monolith project that I could run entirely locally.
And this idea that APIs are slow, unreliable and unable be to be strongly typed is nonsense. This is 2023. We have plenty of tooling and techniques to make this robust.
The (monolith) product I currently work on has about 200 engineers working on it (backend + frontend + devops). I'd say it's of medium size, certainly not small. And yes, we can run it locally easily. The monolith starts up in like 20 seconds which I consider quite acceptable.
There are few narrowly defined responsibilities which are handled by dedicated services, and it is often more awkward to get them working locally. But because they are so specialized, you need them only rarely.
> And this idea that APIs are slow, unreliable and unable be to be strongly typed is nonsense.
Network APIs are inherently slower and more unreliable in comparison to an in-process method call. Some typing solutions are there, but they are way more awkward (and in some ways weaker) than statically typed monolith interfaces.
For a monolith it's having to run the entire platform locally to fix one bug.
That includes databases, caching, auth, ML models, mock API endpoints etc all pre-populated with sample data then the actual application which for complex, JVM based ones can often fail to launch if there isn't enough memory to pre-allocate.
Many systems I have worked on all of that would simply not fit on a 16GB MBP.
I've never seen teams organized around microservices like that. What I've seen, again and again, is one huge team where "everyone is responsible for all the microservices" (meaning, no-one is responsible for anything).
On a theory level I would agree with you - I've just never seen that happen in practice.
I'm not that much of a supporter of microservices, but my experience is the opposite: in every company I've worked for that used microservices, each team had their own set of microservices they were responsible for.
It should be like that but in practice sometimes other team just too busy to implement required feature. I can say my PM that we have to wait for a month but in reality I will have to implement it myself.
The complexities of software development could be solved with this one weird trick - if only programmers remembered FOLDERS. What an absurd thing to suggest.
Sure teams in large organizations allow other teams to randomly create folders? Also I've rarely seen an internal code base that is easy to grasp. You can create your own microservice and set up an API in 1/5th of the time it takes to understand a foreign code base and make a small adjustment.
I'm honestly not even super convinced that small teams struggle to maintain large systems. I've been on a team that was only 7 good engineers maintaining a 3.5 million line project that had both a web UI and thick client. It supported 2 different databases and had a horizontally scalable job runner.
At one point it was 35 engineers, but layoffs took it down to 7, at which point we started to get a lot more done. There was just so much less time spent keeping everyone aligned. So many fewer meetings, sign-offs, reviews, plannings, retrospectives, management meetings, etc. Developers had a lot more agency, so they just got stuff done. Technical debt repayment became 50% of our time, as we easily knocked out features in the other half of the time. We kept ruthlessly cutting complexity, so it got faster to add new features.
I'm sure some projects just need more bodies, but I think there's an upper bound to how much complexity can be added to a system in a given unit of time. Adding developers over a threshold will result in the same amount of features per week, just everyone does a little less and spends a little more time on communication.
Repeat up to thousands of developers where adding a single field takes months.
>At one point it was 35 engineers, but layoffs took it down to 7, at which point we started to get a lot more done.
Years ago I did two back to back contracts for two different pharmaceutical companies. They were both about the same size but one had an IT group that was ten times the size of the other. You can guess which project was late and painful.
The problem microservices try (tried?) to solve isn't about namespaces, it is about too tight coupling between code. Whether that tightly coupled code sits in a subdirectory or in a different repo doesn't matter.
It can be benefitial to maintain well defined interfaces at boundaries between certain parts of your code. It can also produce a lot of work and add complexity. But beyond a certain scale adding systemic boundaries and honoring them isn't something you should avoid.
Devs who do microservices just tend to go too far too early.
>The problem microservices try (tried?) to solve isn't about namespaces, it is about too tight coupling between code.
You can use folders to decouple code by making code private within that folder. Only publicize parts of the code with your languages version of exporting.
Or you can make it private behind an entire new repo and behind an entire service. Only publicize parts of the code through api interfaces like http and make everything 10x harder for the sake of decoupling code.
Many languages misses that abstraction to make things private in a folder (namespace) hierarchy to outside consumers. This is something I like to see solution to (integrated with your IDE of choice)
Usually the most important designation of the separate system is that it uses a separate database. You only have immediate consistency within one service.
At the point where you have to write an API call instead of simply joining to a table, you have introduced orders of magnitude more complexity, failure points, testing challenges, race conditions, coherency issues, code duplication etc into your system.
I don't mind stateless microservices too much. This forms more of a hub and spoke model (many services talking to 1 database). But the minute you bring separate state into the equation it's a chaos factory.
Joining a table is nice until that table is owned by a separate team. Becomes either gridlock or constant breakage, performance can get precarious, and testing involves populating a whole fake DB vs just mocking a few RPCs.
I really like relational DBs. I'll make a system rely heavily on complex joins. But I'd rather call some other team's API than share DBs with them.
Nah but there are real instances where you have to use a different database for performance profiles.
The biggest one in web is regular entity databases vs. timeseries databases designed for analytics.
For this case entity data is just synced to to time series database every so often. Not super chaotic. It does make sense to divide services along those lines.
Though I would argue a folder still works for the web app part.
If you have centralized data then you have broken the abstraction and are in for a world of hurt. You shouldn't need centralized data, just a service front door where that service is the master of that part of the data.
Each service is authoritative on its own data, and it works neatly. Any medium to large size system is going to have natural places to separate that, or else will buckle under complexity or even hardware constraints if you try to keep it all in one DB. I've seen it repeatedly.
Nah this doesn't work out. Because requirements are so chaotic and modular design isn't an axiomatic science things will for sure go wrong with the design both because you got the design wrong and because the requirements shifted to a point where another design was better.
You encapsulated data under one service and you find that it's actually utilized much more in another service. This happens a lot.
The longer you can centralize your storage the better and easier everything is.
Except I've seen this work at dozens of companies?
This thread is very painful, feels like CS101 coming in to tell us why we don't need relational databases and that we can just shove it all inside mongo.
It's not the thread it's most likely you. I don't know if people tell you but dealing with you is likely to be extremely painful if you say rude garbage like this to peoples faces.
Multiple databases can "work" but you get the problem I described above.
You know when you go to a company and eventually you see idiot practices or technical debt? Unfortunately this belongs to that category. You seeing it at dozens of companies doesn't validate anything except for the fact that bad patterns are common.
It also validates the existence of people who often do things poorly and don't know it (aka you).
Multiple DBs does work, and large orgs do it this way with good results. But I'm not here to insult people for disagreeing. It's just software, nothing to get heated over.
No worries. You didn't call the thread painful. That kind of comment causes heat. It doesn't belong here.
Of course it "works". But you can do better.
Splitting data into several dbs when one db suffices is called "extra work". Good results can arise from "extra work". But the extra work wasn't necessary.
You have to think like this: Sure multiple dbs work. But would 1 db in it's place work just as well?
The top commenter here from Netflix one of the people who promoted the whole microservices thing is telling startups to go monolith for a reason.
Our large org has been familiar with the 1 DB approach for over a decade. At some point the need for microservices became clear, and years after that, they actually switched. Even our subteam is feeling the need to divide up our service because its scope has kept expanding.
Startups shouldn't start with microservices. I've worked for (or led) a few different ones, and I've always created a monolith while avoiding other kinds of premature scaling (sometimes even "folder, bro" is overkill). Doesn't mean I'd run things that way with 200 people owning systems with evolving requirements for 10+ years, or even three separate dev teams of 5 people.
Right. But you likely split the db for performance not just for decoupling data and code. Microservices is an option for scaling. My argument is more against it as a way to decouple and organize logic.
Performance wasn't the issue; mono DB was actually fast enough. Multiple teams' services sharing tables became effectively a fragile API between them with hidden dependencies. Small new business requirements were causing too many breaking changes, and the org relied too heavily on a few experts who understood it all. Plus, schema changes were behind a ton of red tape since they affected everyone.
Also if by "decoupling data and code" you mean decoupling data from code, that's not the goal. It's to decouple one team's data from another team's data, and to properly abstract it.
The only reason our mono-DB even lasted this long was because previously, very few requirements were changing over time. So when people say their big org is monolithic, I can believe it. It works in some scenarios, but probably not most.
I know, but our DBMS doesn't support schemas. Instead, it's easy to spin up more "databases" that all share one instance but have separate ACLs, quota, backups, etc. Postgres has something similar but with less isolation. It doesn't really matter how exactly you slice it, point is you aren't sharing tables.
Exactly. And having more than one source of truth for your data complicates a lot of things.
Where I work we run a massive MySQL database that has a pretty simple sharding rule, so I don't believe that you need to split the database even at scale.
It's not primarily about the size of the DB, it's about what's in it. If your massive MySQL DB is only used by one team, then that's fine. If separate teams' services are speaking through shared tables on it, that becomes a mess of an API easily.
We have thousands of developers on it. I’m not saying it’s a silver bullet, but it works and it works well. Also, the service writing on it is only one (monolith).
Not really. You can't really reduce the blast radius of crashes or bad deployments. You need to have the discipline of a good CI/CD instead of siloed but decoupled workflows.
Just keeping things neat doesn't go nearly as far as a separate process on separate machines. Monolith might be better but I don't think it's a situation where you can have it all.
This argument always confuses me. It depends on what you’re doing but if you’re doing web services, as most here are, the crash is limited to the request being served.
The blast radius is a single request, right?
In every likelihood it _is_ a separate process on a separate machine.
If schema migrations take non-zero time and you want zero-downtime deployments, then the both service versions need to be compatible with either the new schema or the old schema, regardless of service size.
The software should be compatible with both the old and new schema until all of your database servers have moved over to the new schema. All rollouts are going to be somewhat staggered even if you go straight to 100%.
The same problem exists in microservoce land, only you now have a load of separate teams managing their own database, perhaps with different solutions.
I think you misunderstood what I said, which is you still have databases and ergo need to manage migrations and such, only that this now typically falls to the specific teams.
That said, loads of folks do microservices with a shared dB.
Even for a web service, imagine a small feature is causing the app to crash to OOM, or too many file handles or something that will take out the whole process and not just the one request.
If you have long running requests (something beside small rest calls like large data transfers) then any being served by that same process are taken down.
Yes, only it takes a while to figure out the right level of abstraction for your organization. It's difficult to start right out of the box with the right type of modularization.
FT.com did a recorded seminar session on their microservices architecture and one of the benefits they extolled is if someone wanted to improve on a feature they could just make it all over again and replace the old microservice with a new one. No need to look at the last persons code, just blow it away like it never existed.
I gathered their site is actually a black box filled with hundreds of black boxes of microservices. All a mystery, they either work or they don't and if they don't they fail gracefully quickly.
Microservices are not a solution to the problem you describe. Nothing in your problem description requires the micro part.
Microservices are about splitting the application into very fine-grained sub-applications. It's not about modularity in general, it's about making things as modular as possible (up to one function per service). That's why they are micro. Otherwise we'd just call them "services" and nobody would have any problem with that.
>Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination.
Assuming coupling is reasonable. If you have a "distributed monolith", you still end up with all the meetings because every microservice change risks breaking interfaces other people are using.
In the context of coupling, I'd argue the same applies to monoliths. Multiple teams can successfully work on a monolith given architecture where they're not constantly stepping in each other's toes (each team works mostly in their own modules/classes/packages)
Exactly, I missed this completely in the article. In my experience, microservices attempt to solve organizational problems and not technical problems. There are technical downsides to microservices that may be outweighed by organizational benefits. With monoliths you might have a large amount of hidden opportunity cost that technically never surface.
I do agree that this is less of a problem for startups so it doesn't make sense to start with a complex microservice architecture. But in large organizations, especially corporates, it absolutely makes sense.
Ummm, extremely large teams develop monolith (single binary) things called OSes or databases etc. The modularity for scaling developers cross-communication comes from.... modules! duh! Aka libraries.
Yep, I work in a large org that used to be a monolith (which means single DB really). Was a mess for the reasons you'd expect. Even our subteam of 10 needs to split things up more.
"Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination."
The coordination is still there because a microservice team does not live in a vacuum. They build services based on demand from other teams that typically build web apps, mobile apps, sometimes server-to-server.
Hence, the "independent" team now becomes a roadblock for higher order features.
I get the point about creating boundaries to document the dependencies - however if your language supports packages and private keywords you can do that without having microservices.
And once you've split up your app into lots of independent microservices who owns the arrows between the microservice boxes? ( The actual app ).
A team of N engineers requires N² coordination. Large teams get mired in endless meetings, email, design reviews. Small teams are more effective, but struggle to maintain large systems.
Splitting a system into subsystems allows each team to focus on their piece of the puzzle while minimizing the amount of peer-to-peer coordination.
Yes, microservices add complexity and overhead, but this approach enables a large organization to build and iterate on large systems quickly.