Episode Transcript
Join us as we gather around the hedge, where we dig into technology, business, and culture with the finest minds in computer networking.
Well, hello, Tom.
Hello, Russ.
You pushed all the articles to one side of the whiteboard.
There must be something written there but we can't read it.
Yeah.
So It's an invisible dry erase marker.
Yes.
It is.
Yes.
It is.
It's an invisible dry erase marker.
You had put lemon juice on it to make it visible.
Is that it you want?
Right.
Right.
It's a it's a list it's a list of all.
Black light.
It's a black light.
Yeah.
I like the black light idea better.
That's cool.
And then today, we also have Yvonne Hello.
Hello.
With her fog lamp.
And Yvonne, I was at a cookie company yesterday.
It's gonna sound really not yesterday.
Last week.
It's gonna sound really weird, but they had this really great cookie tin, and it said, well, bless your heart.
And I thought of people.
I thought of you.
I don't know that we said that so much in our in our family growing up.
We had other we had other weird sayings, but not that one didn't seem to be as as common.
I have a a lifelong loathing of terms of endearment, you know, honey, baby, sweetie, all those things.
Oh, yeah.
I I just it's it's not my favorite.
I I heard too much of that in a syrupy, disingenuous way growing up that, nope.
Not for me.
Yep.
Yep.
I went to a concert a a Christian rock concert one time, and the guy who was the guy who was leading the band got up on stage and said he was talking about something, and he said, shot who in whose backyard?
Didn't even know they had a cow.
And I was like, he's from Georgia.
And so I went and I looked it up, and, yeah, he grew up about thirty, forty miles from where I grew up.
It's just Sometimes you can tell.
Sometimes you can tell.
He's a little bit too and phrases are a dead giveaway.
So today, we are talking about modularization and mitigating people problems.
Now I'll just say, I have often thought that if you took everyone and put them in their own little padded cell, the world would be a much calmer place.
You'll be quieter.
Yeah.
But you still have to build interfaces between modules for us, so it's not like it's not like the padded cells are gonna make the problem go away.
I'm not convinced.
You'll have to work harder.
I don't know.
Does anything work if you have ero interaction surfaces, Russ?
I know we talk about reducing interaction surfaces, but I don't think ero interaction surfaces is the right, choice either.
The older I get, the older Depute your interaction surfaces you want.
I can relate.
Yep.
That's pretty much it.
So but it's interesting because he starts off talking about microservices, right, and how we use modularization to, what, to, like, create little spaces where different things can take place?
Decouple services, you make them work independently.
You can change one without having an impact on the other.
Yeah.
Reduces blast radius.
Yep.
Yeah.
All of that stuff.
Okay.
But, I mean, how do you apply that to people problems?
I know, Yvonne, you had some things to say before we jumped on there.
Oh, well, one of the one of the points that he makes in very early on in this article is that, adding modularization might not might mitigate people problems, but not necessarily solve them.
And I think, I think that's true.
But but something that I've said a lot lately in my world is that there are some problems that you solve and some problems that you manage.
And sometimes changing how you organize people or organize teams can either increase or decrease the friction based on how you do that.
So there's there's some truth there.
But the other the other thing that happens, and we've seen this in microservices as well, the more you modularize, the more those services have to communicate with one another.
And so you create this this mesh, this communication mesh, and, and just like in our systems, like the more of those that you have, the more energy it takes to maintain those systems of communication.
So you really gotta think about how you're gonna divide up your services or your teams and what those communication structures are gonna look like because ultimately that will shape what you make.
A la Conway's law.
Well, right.
So, normally, we think of Conway's Law, and we think about how the organization follows the software.
But what what what this article is advocating is you do the opposite.
You use the software to drive the culture in the other direction, or you use the culture to drive the software, whatever you want to say it.
Like, you are actually intentionally choosing an organizational method in order to or an organization of software to cause the human organization to work a certain way, which is microservices.
Right?
And so this is this is interesting because it is true that we do spend a lot of time and this is the being this is the role I kinda play.
One of the roles I play at Akamai, I owe them being like this, forward looking, trying to dream up new ideas, but also the, okay, I've got this alert over here, and I've got these people that provide the alert and these people that consume the alert, and I need to get them in the same room to talk to one another.
And so that's part of the role you play as an architect.
People don't think about it as being an architectural role, but it actually is an architectural role.
And so there's a lot of bickering that goes on in trying to figure out, oh, is that really what you meant?
Where should that really map?
How should that when should that alert happen?
When shouldn't it happen?
All those kinds of questions come up.
And so modularization, as you said, Yvonne, creates more of those interaction surfaces, but you also don't have the it's not my it's not my monkey.
It's not my circus.
Right?
Well and we we don't The more people that have to agree on a thing, the harder it is to get it done.
Right?
And so I think part of what the argument he's making here in this article is if you can split a a team in two and have them work independently, there are fewer decisions that need to be made about how you're gonna build that particular thing.
Or the decisions are, or they can, or they don't have to be the same.
They can make different decisions that work for each team.
That is great until you have tools for all or, you know, you you still have to have some agreed upon standards, but also providing teams with the flexibility and the autonomy to work is super important.
And I think the autonomy argument is the other big thing here.
Is that we're gonna let you decide the tools that you use.
We're gonna let you as a group figure out how you wanna approach this.
And your goal is to accomplish the thing whatever it is.
And and to do that as independently as possible and I think there's a ton of value there.
One of the one of the examples he used in the article was, you know, a team like part of the people on the on the team working on module want to want to introduce integration tests and the other ones don't and so they all affect each other.
You know half the team is is what they perceive as negatively affected by the other half of the team and so the solution there is divided up into two modules and then if the ones that want to do test, fine, here you go, the ones that don't, don't.
So there's kind of the thing that highlights there for me is the, the the conflict there is managed by by not having a certain group of people affected by another certain group of people's decisions and, you know, that's I think I think that's interesting and whenever you work in a place where somebody's doing something and you just have to live with it and they can make a poor decision and make your life worse, That's that's pretty hard.
That's that's one of those people problems.
And, you know, just dividing up the pie and saying fine, this is yours and this is yours, doesn't entirely solve it because as soon as you do that, you have to have standards not necessarily for what runs inside the modules, but for how what runs between them, and how they communicate with each other.
And so you still have to agree on some things, but just not all things.
And the problem is just what you said, their standards.
Right, Tom?
Like, if one team's doing one type of test, an integration test, and the other isn't, then there could be a quality problem you have to deal with down the road.
And one thing he doesn't really talk about here, which I think is really important, is the depth of those interaction surfaces and the breadth of those interaction surfaces.
If you can just hand the problem off to another team, you're actually in a pretty good place.
But if you're in a position where you have to say, I'm going to provide you with lots of information, the alerting example is one where the people producing the alert and the people using the alert have different standards about what's important and how to mix these things up.
So the integration surface the the the yeah.
The integration surface is very deep in this case in many ways.
Because if the people producing the alert change anything, all of a sudden, the people on the other side of it well, you know, it's like I aggregate out a route.
But every time the route the metric of any component of the route changes, the route itself changes.
How much good have I done with that aggregation point by abstracting something out?
And it's gonna be the same thing with people problems.
We just don't think about it in those terms.
Right?
But it's going to be exactly the same thing.
There's part of me though that as I read this, it's it's it feels a little bit like you're you're trying to solve a problem by not really solving the problem.
It sounds like, you know, and and there are times when there are irresolvable conflicts or things like that that that where, you know, splitting up teams is is gonna make sense.
But I do think, like, ultimately, you have to figure out how to get folks on enough of the same page that they can work together.
And sometimes that has to be by fiat and sometimes it's through compromise.
But, you know, I think like and he even says here that, you know, these kinds of mitigations are only a second best solution, and we should always prefer solving the people problem instead.
I I wanna, you know, double click on that and say yes, yes, yes.
But then he also says most hard problems in software development are people problems.
And I think that's been my experience in general, that ultimately most hard problems boil down to people problems.
And, you know, sometimes we can solve them with systems and structures and organizational structures, and then sometimes we have to solve them like the old fashioned way.
Face to face and having a conversation, and and sometimes, you know, like, sorry, we're not gonna do it your way.
We're gonna do it this way.
And we all have to be adult enough to, like, manage.
So it's, it's but it's it's never a one size fits all answer either.
So it's interesting that you say that because I'm not convinced that all problems are strictly people problems.
They are I don't think they are.
They're people problems in the sense that there are trade offs, and everyone wants to make different trade offs.
Mhmm.
Right?
And so it's it's this doing the modularization does help different people make different trade offs.
If they want to make different trade offs, have at it.
You're within your own little bounded world.
Given I can define it in a very clean way and this goes back to interaction surfaces and the depth of those surfaces and and how they and how they work together So long as you can actually define it in a way where it is its own little thing, you're all good.
The problem, as you say, Yvonne, is when things get messy, when you can't, like, you just can't solve it by dividing it any further.
You have to solve the people problem.
You have to solve the the conflicting ideas and the conflicting information sets and the conflicting ways of of someone trying to solve this problem.
Yeah.
I think I've seen some organizations, not not so much at a software development level, but for example, a company will have three or four different lines of business.
And one line of business will have one CRM system.
And another line of business will have another CRM system.
And then all of a sudden as an organization, you don't have a unified record for your customer to know everything that customer is using across your entire business.
To me, that is a modularization solution that's gone too far because you've segmented your data in a way that it's no longer as useful as it could be for the business.
And so those are the things, those are the trade offs of modularization you have to think about and determine whether or not like, hey, is is are these hard breaks here and letting all these teams or encouraging these teams to work independently?
Are we still gonna be able to get where we need to go?
Again, that's not as much a software development problem, but, you know, what if what if you got one team using one messaging bus and another team using a different messaging bus, and then all of a sudden we have two different ones that we have to maintain and two different license agreements, and you you know, like, it it gets it gets messy.
So those are the those are the trade offs for modularization you have to think about.
Yeah.
And and you can go farther than that.
You can actually say things like, testing platforms.
Who does fuzz?
Who doesn't do fuzz?
Who does integration?
Who does system?
Those are places where things can get messy.
And even the tools that are used, languages that are used, compilers that are used, all of this stuff gets really it's very much a struggle to figure out what the right thing is.
I'm thinking of being in an environment where corporate VPs and above say, I want to use this tool because it really wraps up all the projects into a nice visual representation.
But then the people actually working on the projects go, yeah.
But that tool doesn't have any doesn't have any detailed information.
I can't It doesn't actually help us manage our project.
It doesn't actually.
That's exactly right.
And so then you end up with two different tools, and you have people who are doing, basically, just spending their time putting status between two project tools because because you have two different people who want two different things out of the tool.
Because we couldn't agree.
Right.
Exactly.
Because we couldn't agree.
Yep.
That's exactly right.
So, yeah, I I think that is all very, very valid and very much a part of the real world that we see.
I don't know.
Any other thoughts on modularization to solve people problems?
I'm telling you, padded cells Once you start talking about straight jackets, Russ, you've gone too far.
People still have to be able to type.
Crooked jackets.
Okay.
Crooked jackets and How about retirement?
That's sounding better and better every day.
It really it really Well, unless you've looked at the market lately, but that's another conversation.
Yeah.
That's It's not the right topic for this podcast.
Yeah.
I know.
I know.
And I don't know.
We're on a quest to anyway, whatever.
It's another entire problem.
So the next thing we're going to talk about is this comes up in Ireland.
But it's actually coming up in a lot of towns all over the world, which is that the energy draw for building data centers is so great that cities are now saying, and countries are now saying, feel free to build your data center anywhere you want as long as you provide your own energy.
Is anyone scared by this?
Is this like a I don't think I'm scared by it.
I think it's a pretty natural reaction to, you know, in most places, power is a shared public utility and to have a private entity come and gobble up like way more than anybody ever has before and pay the same rate that everyone else pays.
Or increase the cost because now the demand is so much higher.
Sure.
Sure.
And then that's that's a typical a typical reaction.
Yeah.
Yeah.
No.
I think I think the I think the reaction is reasonable.
I don't I mean, the power's got to come from somewhere and asking the people to provide it that are doing most of the consumption.
Like, I don't even know do I don't know do you all know?
Like, how many houses does it take to consume what a typical data center consumes?
It's probably a massive number of houses.
Right?
Like Well, so so being in the process of building, again, seems like this is what I've been doing with my life is trying to build houses.
I know when I fill built the first house that that we built, that the builder was like, let's just put it put in two fifty k w and be done with it, because that'll be big enough.
Okay.
Well, the builder I talked to this house has already said, oh, dollars 400 to $4.50 is pretty average for a house now.
Now, in terms of what that means from a data center, a lot of data center racks are, from what I understand, 125 kw.
And they'll get as big as two fifty kw.
So For a single rack.
Yeah.
For a single rack.
So if you're and that's not counting air conditioning.
That's not counting environmentals.
That's just feeding the processors sitting there in the rack.
And so if that is if those are correct numbers, which I believe they're they're rough.
They're in the rough territory.
Even if you're talking about 400 k w, four fifty k w to a house now, which seems ridiculous to me, but okay.
I mean, something somebody brought up the other day.
We all went to LED bulbs and Energy Star televisions and everything else, and yet our power bills keep going up.
And yet, somehow or another, we keep increasing the size of the main panel in all new houses.
And when I was growing up, the house I grew up in had a 25 kw.
A hundred it was either a hundred or a 25 kw, panel.
And so we built a little place up on the on the on the lake, and we had about a hundred k w in that place on the lake.
400 w for a house?
Like, what has happened seriously?
Our house is that much bigger.
What's going on here?
But even if that size, even if 400 w for a house, you start talking about 200 k w per rack.
You're talking about a house every other rack.
How many racks can you put in one of these data centers?
I I Found a I found an article, quick Google, says that, basically, data center, and it doesn't provide details here, but can consume the same amount of energy as 50,000 homes.
That's a crazy big number.
Yeah, it says, one one megawatt equals 200 homes is the math that they're doing.
So it's it's assuming a 250 megawatt, data center.
K w.
Oh, 250 megawatt data center.
And they're assuming they're assuming like a 200 k w or two fifty k w house.
Sounds right.
So so so you can play with those numbers.
Yeah.
Yep.
It's a lot of that's a lot of impact for, you know, 50,000 people, 50,000 lives being, not even lives.
I mean, that's you're in the hundreds of thousands of lives probably at their at their Yeah.
50,000 homes is probably what, you know, 7,500,000 people.
Yeah.
Right.
Right.
Yeah.
It's a lot.
Yeah.
It's a lot.
So, yeah, it's it's wow.
It's a lot.
So it's really not amazing to me.
And this now, Ireland has said, basically, overall, in the country.
If you build a new data center, bring your own power.
So that's like a countrywide thing.
And then, Well, the only so this this will have a centralizing effect because how many people can actually bring their own power?
How many people need data center connectivity?
That how many of those that need data center connectivity actually can do it?
None of them And and that that's interesting definition too because I've seen customers define a data center as a closet with a rack a single rack in it.
Right?
Like, what in but with this definition, what constitutes a data center?
Like, you know, how much power does it have to pull?
Because, you know, a a single rack is manageable.
Right?
I mean, there are lots of folks that we know that have them in their basements.
Right?
In in their homes.
But That's why they have four four hundred k w power fees to their house.
Right.
Right.
Right.
Well, like even companies like Equinix though, are they gonna build their own power plants?
I don't know.
It seems like a reach for a company like them.
Well, I think that's part of why we've seen, you know, a lot of our hyperscalers announce partnerships with small form factor nuclear power companies.
And I think that'll be a new industry that we see, you know, crop up in the next decade.
We will have small form factor, you know, nuclear power where, you know, this data center is gonna have its own little mini power plant right right next door that's gonna gonna provide the power.
And it it it it will be even more interesting if if those facilities are able then to start selling that power back to the local power grid.
Right?
Because it's, we we see that now with with homes that have solar.
You you you install solar, and then you can sell that power, you know, back to your local grid.
So I think, you know, we're we're gonna see some innovation in what the power industry looks like in the next decade.
And then it's necessary.
Right?
We can't continue to run all these GPUs and do all this fun AI stuff without it.
Yeah.
Yeah.
Absolutely.
I I think it's absolutely gonna happen.
And I think you'll also see, as you said, Tom, it's not just centralizing.
I would actually say it is you know, if you go back a hundred years in The United States, whatever it is, take your pick of of however many years in The United States, every kid knew the name of every river because there weren't roads.
And so every major city was on a river.
And every you know, that's how you got things from point a to point b, was rivers and lock systems and canals.
So everybody knew where they were.
And they knew if they followed this river up, they would get that city and that city.
Well, we're gonna start seeing, I think, to some degree, data centers needing to be located near or in a place where they can build their own nuclear power plants, their own mini nukes and stuff like this, which is gonna come down to not just centralization in terms of who can build them, but where they can be built.
And and and that will a lot of that will depend on, countries and states and how they regulate their power industry.
It's you know, you we look at, for example, the Equinix model of data centers that they really built out those data centers around existing communication networks.
Right?
And what what we're seeing now is that the rate limit limiting factor isn't necessarily I mean, the availability of connectivity is still incredibly important, but power is becoming a much more, what's the word?
A A gain.
Unavailable resource.
More of a gain.
It's yeah.
So it's interesting that you said about bandwidth because back in the day, we had 10 gig links.
And we thought we were hot when we had a hundred gig.
Right?
Oh, we got a hundred gig across the country.
I was reading the other day that IBM, I think it was, just pushed a 1.6 terabit optical link between two cities.
We're not talking across a data center.
We're talking across tens, fifties, hundreds of miles.
So just at the time when we're starting to see power become this gating thing, we're actually at the same time seeing data speeds and optics be not being the gating thing anymore.
It's like you you can put data a lot of places now.
And just like inside of a a physical piece of infrastructure, as technology evolves, you move the bottleneck.
Right?
Sometimes it's your storage, sometimes it's your RAM, sometimes it's your bus speed, sometimes it's your processor speed.
Like, there are all these different areas where latency can happen.
Just like the bottleneck moves around there, we're seeing in our physical infrastructure the bottlenecks moving.
Now, it doesn't move as quickly, The innovation doesn't happen as fast, but it's it's the same process.
Right?
And right now, it's power and is probably gonna be that for a decade or two, I would suspect.
And then a question that always comes up when you talk about this is, so let's say everybody moves their data centers out someplace where they can build their own nukes and uses these high speed links.
Now we have all these old data center buildings that we've built in the middle of these communities.
And we'll just turn them into peddlers malls.
It'll be fine.
Because they're gonna be all the old Walmarts all over the country.
They'll only be open at Halloween and Christmas for for decorations.
Some some really cool, you know, apartments or some sort of new and, you know, new decor and industrialization of, you know, of of those spaces.
They have plenty of cooling.
Yeah.
Industrial apartments.
I like it.
Yeah.
The world is ugly enough.
Thanks.
I am oh my goodness.
I saw this chart the other day showing the number of colors that are used by paint makers and, like, just how much color each paint each color paint they sell.
And it's amazing that they go back a hundred, two hundred years, and the variety of paints they sold was so much broad much broader than what we sell today.
Nowadays, almost everything they sell, 50 80% of what they sell is, like, gray and white.
Is there's very little red, there's almost no yellow, there's very little blue.
Like, we become we become a monochrome world, and that's that's just a little weird.
Yvonne's like, I'm sitting I don't know what to tell you.
My house is blue and and and, kind of an off white, and blue is my favorite.
So if if I would if I was gonna do any color palette, it'd probably be a monochrome blue.
But, you know Or shade shades of blue.
I'm not a fan of yellow.
You can keep the yellow paint.
I lived in a kitchen with yellow paint for ten years, which is my fault for not resolving in a house once that had black and white stripe, one inch wide black and white stripe wallpaper in the kitchen.
Mhmm.
And that lasted about two days.
No.
I'm not doing that.
I'm sorry.
So yeah.
So anything else on this one?
No.
I think it's, I mean, we're gonna see, an an industry that that spawns up here around this problem.
I think, you know, enterprises have a way of solving problems that get between them and their revenue, and I think this will be one of those.
So, well, it'll be interesting to watch it and see what happens.
Awesome.
Okay.
Well, I think I'm done.
Unless you all wanna talk about something else.
I'm good.
Very good.
Alright.
Awesome.
So, Tom, say it.
LinkedIn.
Okay.
LinkedIn.
Yvonne.
Are you there?
I've been a little quieter on the socials lately, but still on LinkedIn and, Twitter.
Still gonna call it that.
And, you can also find me over on Blue Sky.
But, yeah, mostly, LinkedIn these days.
And you still have a blog going.
Right?
Or have you stopped kind of because construction's over?
I I do.
I do.
It's been really busy at the day job, and so I haven't written as much, but I got a couple ideas I need to need to get down.
Yeah.
I have that same problem.
Day job plus I'm recording I've recorded about fifteen hours of a new training series I'm working on.
And, like, oh oh my gosh.
I'm like, I'm gonna die now because I just have so much going on.
But, you know, it's coming to the end of the semester.
It's coming to the end of the recordings.
Maybe I'll be more casual in the fall.
Who knows?
I always say that and then things come up and it never is.
You have a way of filling your schedule with stuff to do and always staying too busy.
So Thanks, Yvonne.
I'm just saying.
You're in charge of your schedule, Russ.
I'm not sure that I am, actually.
Okay.
Alright.
I won't argue with you on your own podcast.
Not about that.
I have given up on being in charge of my schedule.
Alright.
So I'm Russ White.
You can always find me here at thehedge@rule11.tech on LinkedIn x at routing geek.
I don't know.
Here and there.
Whatever.
You'll find me if you're looking for me.
We know that we live in an attention driven economy.
We thank you for listening all the way to the bitter end of this podcast.
I'm not bitter.
And, we will we're again, we appreciate you listening, and we will catch you next time.
Time.