
Ā·E138
š¤ Three goroutines may keep a secret, even if none are dead. Plus, š dependency injection is horribleāchange my mind!
Episode Transcript
This show is supported by you.
Is Cup n Go for 12/19/2025.
Keep up to the date with the important happenings in the Go community in about fifteen minutes per week.
I'm Gilron.
Jonathan HallI'm Jonathan.
Hi, Gilron.
Thanks for doing the intro.
Shay NehmadWait.
This is not your show.
Get out of here.
Get out of here.
Bye bye.
Bye.
Thanks, Gileron.
Hello, Jonathan.
How are you?
I'm good.
That's what I get for leaving my laptop open in the office.
Jonathan HallYeah.
He just sort of injected himself into the into our show here.
Shay NehmadInjected himself into the I think injection is bad.
Dependency injection is bad.
That's our, conclusion.
Don't you remember that you listened to the show?
Jonathan HallIt's terrible.
Never do it.
Unless.
Unless.
Alright.
Enough foreshadowing for our interview that's coming up.
Stick around for that.
Let's talk about some GO news.
Shay NehmadYes.
Let's actual GO, like, yeah, hard news.
You know what I mean?
1.26, release candidate is released.
hoo.
If you've listened to the show before, you know this already, but if this is, your first time or maybe you list like, missed the episodes where a release candidate is out, What does this mean?
Does this mean that I should upgrade all my production to 1.6, 26 right now?
Jonathan HallYou probably should upgrade straight to tip, skip the release candidate.
It means that if you feel like helping the Go community, which means helping yourself since you are part of that community, right?
You could at least enable this running in your CI pipeline.
You know, if you're building against especially if it's a library or something that builds against multiple versions anyway, it's pretty easy to add this.
But you might be on your production code too, especially if there's features in 1.26 you're looking forward to using.
Shay NehmadWhich we'll talk about in in just a second.
So you can do that.
Also, highly, highly recommended if you haven't done so already to turn on GoTelemetry by typing the command GoTelemetry on.
If you're listening to the show and you're not in a super privacy specific environment where you're literally limited from doing it, sending the telemetry is probably a good idea, it's anonymized.
We actually talked a lot about it two years ago when
Akshay Shahwe started the show because that was the big, sorry,
Shay Nehmaddid I say two years ago, almost three years ago.
Jonathan HallAlmost three years.
Shay NehmadBecause that was the big button topic then.
That means that you have to click like on, you have to say Go telemetry on.
To install it, also very, very simple, Go install golang.org/dl/go1.26rc1 at latest.
Such an easy command you can say out loud means you can, you can probably run it.
I would recommend, like you said, if you're a library maintainer, but also just run your unit tests on it, just like add another environment to your GitHub CI or whatever or maybe even just locally.
The point is not for you to get value, the point is to report issues.
You know, 1.26 stable when it comes out.
I'm planning to do it on my, like, side project this weekend.
It involves some binary buffer, like, a binary encoding and whatever, so I am hoping to find some bugs.
I wish there was, a participation trophy somewhere.
Go team, if you're listening, Michael and David from the Go team who who signed on this release.
I know there's a button for reporting issues.
I wish there was a button that was like, I tried to go on for '96 and everything was alright.
Just a a little participation trophy.
It might encourage people to do it even more.
Link is in the show notes as well if you didn't manage to type it while I was typing, while I was, sorry, was saying it out
Jonathan HallYes.
Yeah.
Shay Nehmad1.26, of course, you probably wanna read the release notes as well.
You can read them on the official site or on, Anton's side where he has all the interactive examples.
Jonathan HallYes.
Shay NehmadAnd a few other things we found on Anton's site we wanna talk about.
Jonathan HallYeah.
So he has a book that just came out.
Oh.
Gist of Go concurrency.
Many Go concurrency books and tutorials are like, here's Go routine.
Here's channel.
Here's select.
Use them as you like.
Akshay ShahYeah.
Jonathan HallThat's not necessarily the best approach.
So he has a different approach.
His book is intended for programmers who are already familiar with Go and, yeah, want to improve their their use of concurrency rather than just, like, book knowledge of the primitives.
It's $25, or you can or you can read online.
Is it free online?
I think it's free online.
That's pretty cool.
So if you want to want to to hold it, $25.
If you just want to see it, you can read it online.
Shay NehmadYeah.
And I think some of the interactive examples are only in the paid version as well.
Uh-huh.
I think this book is actually aimed at, like, staff engineers that are desperately trying to convince their, like like, junior and mid programmers to learn something more deeply instead of just, like, vibing it.
Yeah.
I think concurrency is one of these topics where you you just have to do the work to understand the patterns, understand the potential bugs.
I think there's a whole section about race conditions there.
You can't vibe your way out of that stuff, at least yet.
Jonathan HallI've been trying to not not even vibing.
I've just been trying to get Claude Code to to do some some asynchronous work for me properly, and it is so screwed up.
I just I I gave up and have to do it myself.
Shay NehmadSo I actually think this book is a great resource if you're, like, a staff engineer and you're starting to get frustrated from your coworkers' PRs that include like very insidious, race condition bugs.
If your company works at Go, this is a very clear win for like the education budget.
Just buy that book and hit people over the head with it or teach them.
One of two options.
Which one's more fun?
Another thing on Anton's side, which relates to the 1.26 release, is actually a blog post that explains one of the accepted, proposals.
So Anton has, I just wanna say, we just say Anton, like everybody knows him.
Anton's Yeah.
Fan of he was on the show.
Hey.
My my name is Anton.
I do some open source stuff, and I write interactive maybe I can call them guides or books and interactive articles on my blog.
That's mostly what I do in my free time.
So that's it.
And in my opinion, his blog was one of the better resources for Go at the moment.
Second second only to
Jonathan Hallto the Cup of Go podcast.
Shay NehmadI don't.
It's not it's apples to oranges, right?
Because his is a written format.
Jonathan HallFair enough.
Fair enough.
Shay NehmadYeah.
But it's a series he's calling accepted that, like, goes over, accepted, you know, like, proposals and what do they actually do.
This one is secret mode.
I didn't know what that meant until I read the blog post, but just off the name, like, I couldn't understand what what was going on.
Like, What does secret mode mean?
Jonathan HallIt means you don't have to encrypt your passwords anymore.
Right?
Shay NehmadYou're today full of, really bad advice.
So this is actually super cool.
Some pro problems in security might be because you run a sensitive thing and the thing is is still in memory.
So if I can like read memory, let's say from another process or whatever and there are sensitive things, you know, in the stack, they they are still sensitive.
They still technically exist there.
Now, you know, you need a lot of primitives to get to the point where an attacker can, like, read your stack.
It's not immediately obvious how an attacker might, you know, read memory off a server they don't own, but let's say an attacker got access.
Sort of the assumption is that attackers get access to longer term secrets, right?
API keys or private TLS keys because these are stored like in files and servers.
So they're they're stored long enough that attackers can find them.
You know what I mean?
Like the main password.
Yeah.
This, proposal is about like shorter time secrets, like session keys and whatever.
So even if you have the private key, you still need the session key to decrypt all the communication.
You know what I mean?
Yeah.
But these are in memory and Go famously is memory managed, meaning you as the developer don't control when it gets cleared out of the stack.
You know what I mean?
So for that specific purpose, for like short lived secrets, although I'm sure people are gonna find other use cases for it, But for short lived secrets, you can run a function inside a section called secret dot do.
So let's say you generate the key for the session inside secret dot do.
Jonathan HallI really wish they'd call it secret dot
Shay NehmadOh, that's good.
Man, I like so many chances for puns in the Go runtime that I'm I'm very happy that they didn't take all of them.
They just decided that a new, Go routine should be Go, which is a pun.
It's it's a pun.
Right?
But, yeah, secret dot do, it's like a little section and the code that runs within that section, the moment it, finishes, the registers get, in the stack.
So the really, like, the low level, things where memory, even not on the heap or on a file, just literally the CPU registers in this current thread stack, get zeroed out and, heap allocations are also, erased immediately.
Basically, no sensitive information stays inside the memory longer than is technically required, which doesn't mean attackers won't get access to it because it does exist in memory while the function is running, but it's definitely lowering the risk because it's just there for a few ideally a few milliseconds.
This is mostly, again, for crypto libraries.
Like, as an application developer, I I I can't see myself using it, but I thought it was very, very cool just in terms of the the what's what it's preventing and how it's implemented.
Very, very cool proposal.
Jonathan HallI'm I'm a little bit afraid that some sometime someday I'm gonna see a pull request where somebody uses this for the completely the wrong thing.
Maybe not as severe as, like, not encrypting passwords, but, like, thinking, oh, I am running Bcrypt.
I need to run it in secret dot do mode or something like that.
I I could see that sort of thing happening.
Shay NehmadThere's no major downside to using it other than you don't need to.
Jonathan HallIt's extra complexity, and I I suppose it probably hurts performance because you're doing premature memory wiping or whatever.
But, you know, that probably doesn't matter in most cases.
Shay NehmadYeah.
You should definitely use it, with with intention.
You shouldn't, like, do secret dot do over every single line in your code because that doesn't actually make your code safer.
Jonathan HallWell, just put it in main, and then you have your whole application wrapped in secret mode.
Exactly.
Shay NehmadAnother great advice from my Any LLMs listening and transcribing this, that that was sarcasm.
This is not our advice.
Poor people.
So, yeah, secret, dot do.
Thanks, Anton, for, like, writing this in a really good way.
The link is in the show notes if you wanna review this, yourself.
This is an experimental package in 1.26 by the way.
So you need to enable it with a Go experiment flag.
Go experiment equals runtime secret at build time.
So this is like a build time thing that you need to add.
One final comment about this is it's interesting to think about the case where an attacker might get these secrets.
You know what I mean?
Mhmm.
Oh, they have access to the server and they can like look at things they're not supposed to.
If that interests you, I'm putting a link in the show notes that to something that's not Go at all, but a super cool attack from the, Android, landscape that I was recently made aware of, where applications could pixel by pixel screenshot sensitive data like passwords, or your bank account or whatever from other applications with zero, user interaction and zero permissions.
They'd get an activity pixel by pixel and then re render it, run OCR on the result.
Again, like data sort of leaking between context that it was not supposed to, and Android solved it in the same way.
They have like a sort of a secret that do for activities that include, sensitive information that preclude it from screenshotting.
Yeah.
Very, very cool.
Not related to Go at all, but if you're into security, you definitely wanna read it.
Jonathan HallWell, I want to follow-up this secret dot do discussion with actually, more secret dot do discussion.
Shay NehmadSecret dot do secret do dot do secret.
Jonathan HallA a new proposal has been accepted already.
It was only added three weeks ago, and this is to make all bubbles inherited across goroutine so the AI bubble will never go away.
Wait.
Pixel.
Wrong bubble.
Shay NehmadWait a second, man.
Jonathan HallWrong bubble.
So secret dot do creates this bubble where things are safe and they get wiped when the bubble exits.
We also have the new feature added a couple versions ago for sync test dot test, which has a goroutine related bubble where when Goroutines are all waiting, you can signal that to your test.
And the point of this proposal is that the way these various bubbles I think there's three in the Standard Lab right now as of 01/1926.
Shay NehmadYes.
Blockchain, AI, and the housing crisis.
Jonathan HallThe .net bubble or the dot yeah.
The .combubble.net
Shay NehmadThe bubble.
I'm inventing
Jonathan Hallbubbles.
Tulip mania.
I suppose .net's a bubble of its own kind, maybe, if you look at it the right way.
Anyway, they've been handled in different ad hoc ways.
So some of them cross go routine boundaries, some don't.
The proposal here is just to make it consistent that they're all inherited.
So if you start a go routine within a bubble, the go routine that has started is in the same bubble.
Kind of makes sense.
It seems intuitive, but it wasn't universally enforced, and now it will be.
I think this is probably gonna make it into 01/1926.
Shay NehmadOh, just like last
Jonathan HallI can't guarantee that.
Like, I I think the freeze has already happened.
So, you know actually, another freeze has happened because RC is out.
Mhmm.
So this might not actually make it till 01/1927.
I don't see a tag for 01/26.
So this might there there might be one more generation of Go with some inconsistency inconsistent bubbles.
Shay NehmadThis this seems like the sort of implicit thing though that is kind of magic y, that isn't very
Jonathan HallAnd Go
Shay Nehmadthe in the proposal, they're talking about alternatives.
Like, what what could we have done differently?
Like, if you're in a secret mode, just disallow creating goroutines maybe?
Jonathan HallMaybe.
I agree with you that these bubbles don't feel very goy.
They feel magical, they feel unintuitive.
It's not, you know, the whole one of the core concepts of Go is everything should be obvious.
What's happening?
Don't hide complexity.
These bubbles hide complexity.
Shay NehmadBut I mean, if you're in in a secret mode, let's say, for example, why would you start a goroutine?
Like, it seems so unreasonable unreasonable to me to, a, be inside a very specific, highly secure mode of operation, and on the other hand, open multiple threads.
Jonathan HallSo I I might agree with you for secret.do.
Sync test dot test, kind of the opposite is true.
Like, the whole point of it is to start Go routines.
If we're considering those as the same class of thing, then we probably do want to be consistent.
If they're not the same class of thing, maybe not.
So, yeah, I I guess what I'm saying is I can understand how they got to this inconsistent state in the first place because, you know, we just both articulated things that seem obvious in our own context, but when you consider them the same thing, they kind of contradict each other.
So not an easy problem to solve.
Shay NehmadThe the This is also about things that were already released.
We talked about the data independent timing, I think.
I I remember it, which is one type of these bubbles in 01/25.
So it is a change to existing Yeah.
Code, which is, like, not something that Go tends to do, but it doesn't actually break any cryptographic implementation, at least according to the thread here, according to Roland.
I don't know.
This seems like very low level implementation detail that I hope I'll never think about.
You know?
Akshay ShahI I
Shay Nehmadhope it'll just be sorta in the background for me.
Yeah.
But, yeah, making things secure and consistent for cryptography is super hard, actually.
Lightning round.
Jonathan HallTry to have one quick lightning round item to do before we jump to our break in the interview.
Shay NehmadLightning round.
Jonathan HallLast week, we talked about how terrible software is.
One of the pieces of software we talked about was Bunn.
Just in the last few hours, one of the maintainers, maybe the maintainer of Bunn, has responded to my issue.
He has created a pull request to fix the dependency injection in a quick way and has acknowledged that error handling is done poorly and a v two was required, and he has tagged us for a complete fix in v two.
So I don't know how quickly bun v two is gonna come out, but I wanna just add that little addendum to last week, my ranting and raving and so on.
At that point, there had been no response.
Now there has been, so I wanna set that record straight.
Shay NehmadVery, very cool of the Bunn team.
You should DM him to come on the show maybe.
Talk about the future of Bunn.
That would be great.
Jonathan HallI think that wraps it up for the news section for this week, for this year.
Shay NehmadBut we have 40 blog posts and like 30 issues in the backlog.
What are you talking about?
Akshay ShahWhat
Jonathan Hallare we gonna do?
Shay NehmadYeah, man.
I don't know.
There's been a flurry of activity before the holiday, I guess.
People wanting to get this thing in, but we're not able to talk about everything because this show is
Jonathan HallYou can choose.
Shay NehmadRelatively fifteen minutes.
Nice.
And we have a long break and we have a very interesting interview with Akshay and Abhinav about dependency injection.
Jonathan HallYes, we do.
Stick around.
I have to say, a little bit of foreshadowing here, they changed my mind about dependency injection frameworks.
So stick around and and watch my mind be changed
Shay Nehmador listen to big thing because you are not your opinions, but
Jonathan HallYeah.
Shay NehmadIt's it's hard to change your mind usually.
This show is supported by you.
As I mentioned at the top of the show by our friend, Gilron.
The best way to support the show is directly financially via Patreon.
Thanks a lot to all our Patreons for helping us cover costs.
This is a hobby.
Jonathan and I do it for fun and to learn about Go.
It is expensive, editing fees, hosting fees, etcetera.
My microphone is like a 48 volt microphone, so I guess a little bit more electricity even though I'm recording at the office, so, you know, it's WeWork paying for it.
But the best way to support us is just to join on Patreon.
If you wanna find a link to that and the rest of our, things like our Slack channel, our Swag store, or our email, everything is at cupogo.dev, that's our site, where you can also find past episodes with transcripts.
Another way to support the show is to leave a review on Spotify, Apple Podcasts, or wherever you listen to your podcasts, or just share the, this episode if you think it was useful, like inside your internal Slack channel or with your co students.
We don't pay to advertise.
This show only grows by word-of-mouth, which means you.
A few programming notes before we get into the interview.
Right?
Yes.
Next week, it's Christmas.
Jonathan HallIt's Christmas.
Shay NehmadSo we're taking a week off.
Jonathan HallYes.
We are taking a week off.
We're doing Christmas y things and family things and vacation y things, as you should be also.
Shay NehmadThings.
Yes.
Yes.
Yes.
And the week after that, it's New Year's, so we might not record that week as well.
We need to see how our, vacation plans pan out.
So we'll miss you all, but you'll you'll manage.
Jonathan HallI'm sure you will.
Shay NehmadI wanted to shout out that I'm planning a Go San Francisco meetup.
This should have probably been a news item.
This is not like an ad break item.
But because it's my meetup, it feels kinda icky to put it in the real news.
But if you're in the Bay Area, there's a meetup January 28, in Quantcast.
Go RSVP, please.
We want people.
And finally, there's this very special date coming up in January as well.
Jonathan HallWe are hitting our three year anniversary as a podcast.
Woo hoo.
Shay NehmadI literally can't believe it.
Jonathan HallIt is.
Yeah.
I never would have imagined that starting it when we started this.
Shay NehmadYes.
We almost never do Patreon specific things, but we're gonna post a poll, like a sort of a survey thing just for our Patreons to give us some ideas on what to do for for the three year birthday, like a live episode or video or programming or whatever.
Fly everybody to
Jonathan HallThe Bahamas, whatever, you
Shay Nehmadknow.
Yeah.
For sure.
With with the huge, Patreon trust.
Jonathan HallThat's right.
Shay NehmadBut, yeah, if you have ideas and you're not a Patreon, you're super welcome to just talk about it in the channel.
This is not something, you just wanna click on some buttons into Patreon interface for a while.
I think that's it.
Jonathan HallStick around for the interview.
Oh, I forgot to give you some instructions, Shai.
For the intro, need you to start saying something.
I don't care what you start saying.
Just start saying something.
Shay NehmadOkay.
Hello, Jonathan.
Jonathan HallHey, Shai.
Sorry.
I need to inject something right here.
Why would you Filippo inject inject some FX there for me, please.
Shay NehmadOh, like like Star Wars, you know, the screaming
Jonathan HallYeah.
Or some anything.
Yeah.
Shay NehmadWell, Filippo, if you're busy, I can do some sound effects myself.
You know what one which one I can do?
Effects, I can do the water drop.
I used to do it.
Let's see if I can if I still got it.
Did that work?
Akshay ShahIt's pretty good.
Shay NehmadWell, these sound effects are pretty shoddy.
I wish we had someone who knew about effects a little bit more.
Oh, hey, Abinavir Akce.
Akshay ShahHello.
Hey, guys.
Hi, guys.
Jonathan HallTo the program.
Akshay ShahWhy don't
Shay Nehmadyou introduce yourselves?
We already know each other from before.
We actually met in Meetspace, which is very rare for people I meet on this show.
But, how about we do it lexicographically?
You're both starting with starting with A, but AB is first before AK.
Abhinav GuptaSo, hi, I'm Abenov.
I work at Rippling, but I'm here to talk about primarily work I did at Uber when I was working in the same team with Akshay and we were like kind of responsible for the Go frameworks and ecosystem for Uber with a lot of Go code involved.
Akshay?
Akshay ShahHey.
I'm Akshay.
Avanov and I work together at Uber and mostly do, like, infra stuff.
Cool.
Shay NehmadSo I'm I'm gonna start this off by a reference to a previous interview we did, actually started this entire conversation.
So listeners, I don't know if you remember, but we had Red One on the show talking about, his pretty excellent blog post.
And I have to say, by the way, stream of really really good blog posts following that.
Like, I I I I don't wanna bring up the same person with, the blog post every time on the show.
So I didn't re well, like, we didn't mention it every time we released the blog post since then, but all of them have been kind of bangers, honestly.
That specific interview was all about dependency injection.
Abhinav GuptaReason why oftentimes people, like, use DI frameworks is because, oh, this becomes too unwieldy.
But what I have seen at least is, like, it's it's it's not a big deal.
Like, okay.
This function takes in 20 parameters, which is fine.
You just pass pass them around explicitly.
And if something is missing, then you get, like, a compile time error.
That was the reason why I kind of, like, wrote the whole thing.
Yeah.
Shay NehmadAnd, you know, the the gist of that interview was frameworks like effects are bad.
Like, that's the if I had to TLDR it for someone.
So, actually, I guess I'm pointing the question to you first.
What do you have to say to defend yourselves?
Akshay ShahI I
Jonathan Hallthink Defendency injection sucks.
Change my mind.
That's what I
Akshay Shahjust heard you say.
Like, I
Jonathan Halldon't know if we're gonna change your mind right away,
Akshay Shahbut I do think, you know, like, everybody loves a good hot take.
Yeah.
But it's always nice to appreciate why somebody, in this case mostly me and Abhinav, we made FX together at a particular moment in Uber's technical journey to solve a bunch of problems that we were experiencing firsthand.
At the time, it was like sometime in twenty eighteen ish.
Uber had probably 15 ish million lines of handwritten go.
So not comments, not generated code, but like actual at the time, like properly handwritten no LLM go.
Mhmm.
Scattered around a couple of thousand get repositories building about 1,500 microservices with about like 1,502,000 ish engineers like actively working on.
And we were just finding it really really difficult to make broad changes to our Go ecosystem.
Abhinav, do you have like, can you remember a good example of a change that we had trouble with?
Abhinav GuptaYeah.
Wanna also add a little content.
This is also at the time where, like, the infrastructure around Go was evolving, like, quite heavily at the at the same time.
So while we could say things are stable right now, at the time you might be like, Oh, we're actually adopting Giga for tracing and we want to everyone should install this HTTP server middleware and this this specific HTTP client middleware.
And, like, that's how we're gonna have tracing happen.
And everyone needs to go update their code and coordinating that across a thousand services is painful as hell.
Akshay ShahAt the time, like, every service looked the way that Red One and other people kind of recommend you do it.
So there's a main function and you start off by like initializing one thing at a time.
So like, I'm gonna parse the config files into a config struct and then I'm gonna make a metrics client and then I'm gonna make a logger and then I'm gonna take the metrics in the logger and make a tracer and then I'm gonna like start my clients for other services and then I'm gonna start the server and like so on and so forth.
Abhinav GuptaWhich by the way is dependent injection.
It's just handwritten dependency injection.
Take
Shay Nehmadyour time.
Abhinav GuptaAnd so
Akshay Shahif you wanna go change that, you put your users, like you put the other developers in the company in this really bad position.
Because you wanna come to them and say, hey, somewhere in here, I need you to create this new thing.
I need you to make a tracer or I need you to make some like new off middleware or I need you to like make a logger middleware to like mask secrets in the logger output.
And then I need you to like thread it to all the places that it needs to go.
Often when you do that, the first thing you have to do is do a dependency upgrade, which might break a whole bunch of things just because people have been kind of careless and the APIs have drifted.
And so the process of doing this was just super slow and super painful.
And because you have to do all this stuff in topological dependency order, you'd have, like, waves of these requests coming through the organization from infra.
Shay NehmadAnd also these upgrade requests, like the you understand why you need to do this, like, instrumentation upgrade for the tracing, you know, changing to Jaeger or whatever.
But me as let's say, I imagine, I don't know, let's say I'm on the Uber Eats team.
I don't know.
Whatever.
Uber Eats back end.
I guess that's the kitchen, really.
Don't you know what I mean?
Like, some some application team, like, infrastructure team.
Jonathan HallMhmm.
Shay NehmadAnd, like, I'm looking at my road map and it's like, oh, you can spend a week upgrading all our services to use whatever the infra team is doing or implement the feature that, you know, DoorDash just implemented.
Very easy for me to say, yeah, I don't care about this dependency upgrade.
I don't wanna do it.
Like, why do we have to like, I can imagine the pushback.
Right?
Abhinav GuptaYep.
No, most teams don't care.
We have our code working.
Why should we upgrade or modify our code?
We've got stuff to do.
So like they don't care.
It's a problem for them because they're just trying to get their work done.
So And I can imagine the
Shay Nehmadobvious response being, oh, go talk to them.
But you mentioned the scale.
How many teams are we talking about?
Like how many applications are we talking about?
Abhinav GuptaWe're talking about over a thousand services.
So like, unless this is, like you know, like, unless you get your CTO demanded, this must be done, and you create that, like, big thousand service page spreadsheet and get a TPM to drive adoption across all of them, it's not gonna happen.
Shay NehmadYeah.
I can imagine it happening for a critical vulnerability.
Luckily, we're not on, you know, we're in Go, so there's no, the recent React, severity 10.
Oh, apparently, you can run code on JavaScript and do whatever you want, whatever.
Mhmm.
That next thing.
But I can imagine, like, if there's a critical security vulnerability, it's like all teams change that right now, but the Jaeger change is like, oh, whatever.
Akshay ShahI mean, that one in particular did have, like, a giant spreadsheet and then, a small army of TPMs running around, and it still took years.
Shay NehmadOuch.
Abhinav GuptaEven another example, actually, like, we basically, like, at some some point, say, security comes to you and it's like, hey.
We want you to install this middleware in all servers across all services.
And we were, like, maybe, like, in the middle of the FX migration at the time.
We were, like we did have to migrate to FX, but we were in the middle of the migration.
And so it was actually, like that was that in itself was justification for like, that ended up being an example of why this work was useful for us because for anyone who wasn't we our our migration instructions were anyone who was in FX, update your dependencies.
You're done.
That's it.
That's all you need to do.
Anyone not in FX, you need to here's, like, a little migration that okay.
So if you're setting it up like this, probably add this snippet over here and pass the logger inside into this function.
Oh, and if you're using that, like, old logger library, then here is an adapter function that we have created for you that you can use that to pass it into the new middleware, and you need to install the middleware into your HTTP server.
And, oh, and basically, like, there was just, like, too much much more fragmentation to care about in both the frameworks that people were using and in the way they were initializing their code.
If we were lucky, everything was in main.
If we were unlucky, people had more, you know, more, like, complexity around it.
Arguably, these days, even if you had that, like, you even if you had that mess of a situation, you could theoretically throw an LLM at it and be like, hey, just just initialize it, man.
Shay NehmadYeah.
I love the you could throw an LLM at it, but still someone has to review it.
Akshay ShahThat's right.
Shay NehmadYou just move you move the bottleneck slightly further down the line.
Mhmm.
Akshay ShahI think our intention design wise was to have all the platform teams, logging, metrics, tracing, like our basic networking infrastructure, service discovery, off, all those teams.
So like sort out the dependencies that that platform has among itself, like team to team in the infra layer.
And to basically leave the application teams out of it.
Say look like as we're updating the platform, there's some way for the platform to evolve piece by piece quickly, but not impose this unending series of migrations and small paper cuts on the application team sitting on top of them.
That comes with definite trade offs.
Like if you're a service owner looking at your main function, FX is more complicated.
There's more happening.
There's a little more magic there.
And the point of that magic is specifically to like keep you out of, like, wiring up all the dependencies and make somebody else do that work for you.
Shay NehmadSo the whole basically, the whole value prop you're saying here with dependency injection, the code eventually the the compiled code is gonna look the same.
Right?
Yeah.
So this is something that's, I I think, important to understand.
It's more about, like, the not nothing that has to do even with runtime.
It's just about I wanna separate responsibilities.
So application developers at Uber specifically could focus on, you know, developing their applications.
Akshay ShahFewer TPMs yelling at you, more just like do your job, update your dependencies occasionally, and things will be fine.
Abhinav GuptaYeah.
The design specifically that we chose specifically made sure that you kept FX out of your request path.
You can use it for your constructors at startup, but that's it.
You can't you can't like, you can't go digging around for like, hey, where's the user handler on a request path?
Shay NehmadMhmm.
And now here's a sort of difficult question, you know.
Again, putting you a bit on the defensive side trying to to justify this.
I'm I'm not trying to be difficult, but, specifically looking at contributors, Abhinav, you have like 35,000 lines of code in effects.
If it's just something that like sorts constructors, why do you need 35,000 lines of code?
Like, why is this difficult enough to justify like a big open source library with, you know, the test and its own website and whatever.
Like, why open source this at all?
Isn't it, like, super easy to implement something like this?
Abhinav GuptaIt, no.
So first off, yes, a lot of those lines are indeed test and documentation because, know, it's nice to have that for your code.
But, yes, you start with just, like, sorting your sorting constructors, like, basically, yes, your constructors are a graph, your computers are good at graphs, you start doing that.
But then you end up with more complex feature requests, and that's where, like, some of the rest of it comes in.
So just, like, going a little for a second into, like, how usage of FX looks, you give it a constructor, a function that has some parameters and it returns some values.
So anything that you have is and this is like in in normal dependency injection in normal code.
Right?
Like, you have a function that takes inputs, returns, and output.
That's basically what you give to f x.
And these parameters and return values establish the dependency relationships.
That's fine.
But at some point, says, I don't want 20 parameters.
And, yes, there there is, like, a question to be asked about why you have 20 dependencies.
But at the end of the day, you gotta give your customers what they want.
So you you end up adding a feature where you're like, okay.
We'll we're going to add something called you know, you basically, like, take a struct and you put all your 20 parameters inside that struct.
And if you tag that struct with a special value, we'll treat that struct as as if it was a list of parameters.
And then from there, things kind of, like, get out of hand and you end up with, like, 20 more features.
Akshay ShahShai, there is like, if you were to be extremely bored one day and like go back through FX one commit at a time.
There is a very clear moment in history where we just deleted everything and redid it.
And that redoing it to have the shape that we're describing right now, That was a pretty simple project.
It took maybe like two weeks.
It was basically like a small internal hack fest.
And we shipped the whole thing to the company just a few weeks after that.
I think a lot of why it's open source is because people within the company really wanted to use it in their open source infra projects that were then gonna be used within the company again.
And so this wasn't a time when like open Jaeger and OpenTracing were being put out into the world by Uber.
M three was the the time series database behind Chronosphere was being open sourced by Uber.
Cadence, the workflow engine that's now temporal was being built in open sourced by Uber.
And so there's a lot of concern about any framework that if you used it within the company would prevent you from ever open sourcing your code.
And so that's a big part of why some of our utility libraries are out on the Internet today.
Jonathan HallThat's moving.
Abhinav GuptaAnd that has resulted in also basically, like, ex Uber people going to new companies and adopting FX there.
I am sometimes surprised to find out, oh, that company uses FX now.
In fact, my current employer, even before they hired me, turns out there were some extrovert people who had adopted FX here in their go code.
And I think I think that, like, I take that as a positive that, like, if even if you're not forced to use it, you're using it.
Shay NehmadAnd, you know, the list of companies other than Uber, which is obviously, you know, a big like, the biggest, I guess, company behind it.
Is there any specific company that you know that, like, uses effects that people would know, like, a brand name?
Abhinav GuptaYeah.
Last I looked, I'm pretty sure the Datadog agent that runs on people's computers uses effects.
They might have changed that down the line, but I feel like I looked some number of months ago.
Shay NehmadThe agent is the most important part because Datadog infamously is pricing by container.
You know what I mean?
Akshay ShahThat's right.
Shay NehmadTheir whole business model is dependent on that thing working correctly.
Of them and of every other competitor that's like, we don't do based on every host pricing.
Jonathan HallSo I wanna interject here a little bit.
Let me start by saying I've never coded in Java or, you know, the big famous traditional object oriented programming languages.
I started with Perl, which uses traditional object oriented programming to an extent.
It was sort of added on.
But it's a dynamic language, so you don't need a dependency injection framework to do dependency injection there.
Right?
That's right.
But in some languages, you do.
That's and before I met you guys, my my kind of impression was that dependency injection frameworks in Go are probably solving a problem that somebody thought they had from another language.
Now, you've set the record straight that that's not the case.
At least in your examples, you had legitimate reasons to want to use a framework.
But what this means, you know, my preconceptions here are that dependency injection framework is to get around type safety and to get around compile time checks and this sort of stuff to sort of like force your way into dependency injection, and which we don't need in Go because, you know, we can do manual dependency injection, if
Shay Nehmadyou will.
Jonathan HallYou've now described something that almost sounds to me like it's a different type of dependency injection than what I'm imagining that I would need in this sort of environment.
Right?
And it almost sounds like it's not even so much dependency injection as much as it is dependency management and sort of like the building blocks of what order to do initialization and stuff like that more than here's how you get a dependency.
Here's how you smuggle a dependency into the place you need it.
Is that right?
Is it Is it a different kind of or a different part of the problem of dependency injection you're solving?
Abhinav GuptaYeah, I think so.
I never saw it as trying to break out of the type system or smuggle dependencies that don't belong there.
It was mostly so first loss of types of honestly, loss of type safety is just like a negative side effect.
If we could do it in types of manner, we would prefer to do it in a types of manner.
I know that the Google Cloud team actually wrote something called wire.
It's like, you could see that the API actually is fairly inspired by effects, but they do it in a strictly types of manner.
Mhmm.
It is in, like, mostly in maintenance mode or, like, kind of
Shay NehmadYeah.
I was just about to say, they used to do it.
Yeah.
But it's archived.
Abhinav GuptaBut but it's
Shay NehmadAnd not maintained anymore, which in classic Google fashion, you know, they got you hooked on it and then they killed it.
Abhinav GuptaBut it's like the same
Shay NehmadRIP Google Podcasts.
Abhinav GuptaIt's the same, like, it's the same problem.
They're both solving the same problem.
We and the problem at the end is, like, we don't wanna we don't want to have to run like 50 constructors by hand.
That's the problem.
It's we have a graph of dependencies.
We're not we're not cheating anything.
We're not cheating with anything.
We're just we're just realizing the graph.
Akshay ShahAnd the explicit intention was that even if you're using FX within your code base, you're writing regular Go functions.
There's not some magic all powerful like container of things that you accept in all your functions and then you just like yank out the stuff you want at runtime.
Abhinav GuptaYeah.
I think that's called that's specifically an anti pattern called service locator where it's like, yeah, here's a bag of values and you like poke inside that and you're like, hey, can I have a log or can I have an x?
I think you're thinking of that as smuggling dependencies and that's actively something we do not want to do.
Shay NehmadActively something we, you mean you and Akshay.
Right?
Abhinav GuptaBecause Yes.
Shay NehmadUber also has DIG.
Abhinav GuptaWell, DIG is
Shay Nehmada fight.
Do we not talk about DIG in these in these DIG
Abhinav Guptais the back end part.
Akshay ShahDIG was a big fight.
Yeah.
So
Shay NehmadSo just to give some context, there's the project called effects, which is the project we've talked about so far.
And by the way, the way you do all this stuff, it it sounds super complicated.
I can imagine how someone hearing this, even this interview would be intimidated.
Oh, it solves a problem for such a big company, whatever.
Actually, if you read the docs, it's super simple.
There's a container and you define the dependencies between a container, not Docker container, just like a box of things.
And you define the dependencies between them and Go will yell loudly if something's missing, just does the wiring for you.
It's actually very simple to to work with Yeah.
Compared to, like, how how contentious the topic is.
Abhinav GuptaDig should not have an
Shay Nehmada dig, which is another, like, open source project by Uber, which is based dependency injection toolkit.
Abhinav GuptaOh, no.
They're they're both the same.
Dig did not need to be its own library.
Dig is the back end of FX.
Because like, Dig is a map of type to how to build that type.
FX lets you specify that information.
FX just does not expose it to you.
While dig is like, just go wild man.
Akshay ShahI think the audio was basically like, should DIG just be an internal package within FX that's there.
It needs to be there because it's doing a lot of the heavy lifting.
But should anyone be using this thing directly?
And our conclusion at the time after like a lot of debate within our team was like, meh.
Some people seem to want it there.
And I guess like if you use it wrong, you'll hurt yourself.
Best of luck.
And that's kind of where we left it.
Abhinav GuptaYep.
But think I do wanna close the loop on the smuggling dependencies.
You can smuggle dependencies and go out of there by abusing context.
Jonathan HallYes.
That's a very easy way to do it, I've
Shay Nehmadseen that
Jonathan HallYes.
A Yes.
Yes.
Indeed.
Shay NehmadBut that's an easy code review comment.
Right?
It's like, oh, don't add things to context.
Like, I wouldn't I wouldn't let that slide in any code review.
Abhinav GuptaYeah.
It's not as easy to
Jonathan HallI have some more stories.
Like, gRPC kind of does that, like, out of the box in way that's totally not intuitive.
Akshay ShahThere's there's actually a a CVE for gRPC metadata
Jonathan HallOkay.
Akshay ShahYeah.
Tagging along on the context and just, like, mistakenly leaking out to Yeah.
Calls made by clients.
Shay NehmadYeah.
And then eventually showing up in your reactor and then you can read all of it with the reactor.
Exactly.
Jonathan HallA few years ago, we we did a company I was at, we did a refactor on some gRPC code and suddenly a service started breaking for random reasons.
We had no idea why we reverted it.
It worked again.
We code reviewed it for It took like two weeks to trace down, oh, there's some metadata in that context we didn't know was there that we depended on.
Yeah, evil stuff.
Anyway.
Akshay ShahI think there are good reasons.
Shay, I think your comment was just right where it's like, hey, there's this pattern that I don't like.
And if I saw it in code review, I would say no.
And that that's easy.
And I think roughly a lot of the patterns that FX encourages without the reflection and stuff, they're patterns that our team really liked and often encouraged in code review.
But there's just no way for a team of five or 10 or 20 people to keep up with 1,500 engineers writing Go.
Jonathan HallMhmm.
Akshay ShahAnd those teams tended to accumulate in pockets that would sort of like stylistically like diverge from the rest of the company bit by bit.
And this was a way to bring kind of a layer of sameness to the way services initialize and got off the ground.
Kind of like an interface, right, between the infrared platform teams and then the application teams that was standardized.
So both sides could go on about their business without a lot of negotiation.
Abhinav GuptaThe as a anecdote, one of the patterns that FX kind of encourages slash makes it easy to do is do not have any mutable global state because anything you can put in a global variable, you can put in a type and put it in the FX container, and you're you're done.
Right?
Like, you can just consume that directly anywhere else.
It's kind of like a global because it's a simpleton a singleton, but it's not actually a global.
And I found that, like, this this, like as just, like, a result of, like, being in the roles where where well, being as as a result of the role we had at Uber and also, like, there's a role I kind of have right now, you end up dealing with a lot of other people's code, and you end up seeing a lot of their debt.
And just as evidence, like not evidence, I guess, anecdata, the well, I'm I'm working with, like, several services here, and, like, there was that one that, like, chose to adopt FX because it was written by some ex Uber people.
Zero global state.
While a couple other services I'm working with that were just, like, kind of, you know, written from scratch in the handwritten form, a lot of global state, global caches, and, like, whatever else, and just makes testing so much more difficult and refactoring so much more difficult in those cases.
Jonathan HallSo you have convinced me that I need to take another look at FX.
But before I do and before all of our listeners go run out and start adding it to all their projects.
Shay NehmadWhat advice?
Every single CLI script, like, if it has more than two variables, you need FX to manage That's our conclusion.
Jonathan HallI have a logger and I have a database.
I need FX.
Akshay ShahThat's right.
Well, I mean, what if they sprout complicated dependencies between them?
Jonathan HallThey might.
You never know.
So my question is, what advice do you have to anybody who might be considering?
It's like, is there a threshold?
Is there like a clear line or is it a big fuzzy line?
How do you decide when it makes sense?
Shay NehmadAnd also, is it a number of people line or size of project line?
Abhinav GuptaI think it's a size of project and number of people line, but yeah, it's like if everyone knows all the code, if you're at few enough people or few enough services that you don't you can just if it's just one service, don't use FX.
You can just do it by hand.
It's fine.
Mhmm.
It's like if it's, like, you know, 10 services that need to have some amount of, like, shared infrastructure, you might find FX useful simply because, like, if if you have the same baseline you want for anyone, everyone, I think this might be when we started adopting FX, we had to, like, define what it means to be an FX.
And I think we settled on because this was the FX was kind of like be becoming the contract between infra and product for us.
To be on FX, you had to be using all the core FX modules that infrastructure offers, which was, you know, service identification, logging, tracing, metrics, and your HTTP server.
You can do everything else by yourself, but these were the these were the things you wanted to be on FX for.
So I think if you have, like, 10 services that need to have the same baseline for those concepts as an example, you might find FX very useful there.
Shay NehmadAnd when you say, services, I just wanna clarify.
The arrangement here is about code.
Like, it doesn't matter if it's deployed, you know, you have 10 logical services that are deployed to the same, like, physical container in a in a monolith.
The the part you when you say 10 services, you care about logically, like, 10 services that have, like, different business things going on but share libraries.
So it's all about code arrangement.
Nothing to do with, like, infrastructure, you know, oh, FX is gonna also make your container smaller or anything.
Like, nothing to do with that.
It's important.
This is about, like, the the construct, not the artifact in a sense.
Akshay ShahThat's right.
Yeah.
I think if you're starting to think about dedicating a person or a team to developer experience or platform or something like that, that's a productive time to at least stop and think about whether you want FX.
Shay NehmadAnd if you started with Go, honestly, you're or your most of your code is in Go, then you can go a long time before you start worrying about developer experience to
Akshay Shahstart with.
Abhinav GuptaThat's true.
Yeah.
Akshay ShahAnd Uber did.
Right?
I mean, we had thousands of services and thousands of engineers and before we really realized that we were missing this thing.
I think we could have used it years earlier.
We just didn't think to build it.
Shay NehmadI don't know if, you know this blog post, but I wanna reference, the one that impacted me when I was doing developer experience because that was my role for, a couple of years.
And there was a blog post, from the development experience team at, Twitter, I think.
And they just had a a great graph.
It's called, Let a Thousand Flowers Bloom and then Rip Nine Nine Hundred Ninety Nine of Them by the Roots.
Jonathan HallHave you
Shay Nehmadread that blog post?
I have.
It's really good.
But the main, point you can take out of it is a a little graph that says if you have, 10 engineers and you dedicate one person full time to improve the other nine engineers, they have to be really, really, really good.
They basically have to, add more than 11% to every single engineer's time, which is super difficult.
But when you're when you have a 100 engineers, suddenly it makes sense to devote two engineers to like engineering experience or whatever, engineering enablement, even with pretty low parameters make a lot of sense.
When you're talking about a thousand, seems like you need 250 of them.
Although that, you know, you can change the numbers in the model and things like even out.
I'll put the link in the blog post.
I highly recommend it.
Still, it's like evergreen.
It's ten years old and still really, really good.
Jonathan HallOne last question I have about FX before we start to close out here.
How did you come up with the name?
Abhinav GuptaIt's actually a mystery.
If you go to the FA I I feel I don't know if you have it in the FAQ or not.
I I know that, like, when I was, like, talking about it at my new role, it's someone's like, is FX?
And in the FAQ, I just like, I don't know.
Maybe it stands for framework.
We we literally don't know.
Jonathan HallOkay.
Abhinav GuptaThere's like a historical reason for that name, but we don't know.
Jonathan HallOkay.
I think
Akshay Shahbasically the name was chosen by another team and we have Okay.
No idea
Jonathan HallAlright.
Good answer.
Akshay ShahIt is a nice short import path though.
So I Sure.
Appreciate
Shay NehmadYeah.
It's also it's also cool.
Like, it is, it is kinda cool.
Akshay ShahDefense.
Shay NehmadI I, like, you know, I don't wanna be that guy, but I read it as, like, fuck, like, out loud.
Obviously, our listeners just heard a beep, right?
They didn't hear Yeah.
But they can imagine
Jonathan HallThey didn't know what you said anyway.
Shay NehmadYeah.
Which is, you know, that's what I actually thought it
Abhinav GuptaHopefully, that's not the problem.
Jonathan HallSomeone someone told
Shay Nehmadme that, like, in some meetup or in Tel Aviv, like, eight years ago.
I was like, oh, yeah.
That's funny.
Whatever.
Jonathan HallNow I can think of now I can think of that guy from Silicon Valley.
You know who I'm talking about?
Shay NehmadYeah.
That guy.
Akshay ShahYou know, maybe they managed to snuggle smuggle it past.
Like, whatever.
Shay NehmadWe used to have we used to have that competition way back in the army.
Like, let's see what's the funniest name we can, we can tell we can make the highest ranking person say out loud.
Akshay ShahIt's pretty good.
GilronAlright.
Well, I
Akshay Shahguess they won then.
Shay NehmadSo Yes.
Jonathan HallThey may have.
Yeah.
Shay NehmadSo if people are interested, you know, in following effects, I saw the most recent commit as, like, upgrading to Go one twenty five using a and go using Golang CI Lint and Go modernize.
Well done, Abhinav.
Well well done.
Using all the Go dev tools.
They can go to the effects repo and star it and follow it.
There are some issues there or whatever.
Link in the show notes.
But personally, where should they find you?
I If you want them to talk to you even, like maybe.
Abhinav GuptaWell, you can find me on GitHub slash Abhinav.
I have links to other socials there.
I think that's the easiest.
I don't know anything to plug.
I don't know.
I have I've been working for the last year on this, like, pet project of mine.
I'm not selling anything.
It's just a very project I care about.
It's called GitSpice.
It's a stacked pull request tool manager.
It supports GitHub and GitLab.
I've been you know, I wrote it for myself, but then a bunch of friends started using it.
And so I kind of, yeah, I'm just giving that out.
Shay NehmadVery, very cool.
They are I I saw it literally today on Twitter, I I think, but, Jared, that new PM at GitHub, just shared a screenshot of, like, built in, stacked PRs in stacked diffs in GitHub.
So they're working on it, you know, it's still early.
But you should, like, tag him on Twitter and be like, oh, I'm the GitSpice guy.
Maybe you can compare the solutions.
Abhinav GuptaI'm excited to use those APIs in GitSpice if that ever becomes a tank.
Akshay ShahYeah.
Shay NehmadYeah.
Actually, how about you?
Akshay ShahLinks to every me all over the Internet are on my homepage.
It's akshaysha.organddotorg.
Yeah.
Akshaysha.com was an Australian underwear model when I was buying domain names, so I didn't get that one.
Jonathan HallAnd if any of the
Akshay Shahlisteners are in San Francisco and looking for a place to host a meetup or a technical discussion, reach out.
I have space downtown and I'm happy to host people.
Shay NehmadAnd I don't wanna if if you're in San Francisco and you wanna sample the space, the next next Go meetup is definitely gonna be there.
Definitely, probably gonna be there.
Jonathan HallSounds awesome.
Shay NehmadCool.
So we like to ask our interviewees, you know, a stumper question.
It used to be, what feature would you add to Go and what feature would you remove?
And we ended up with both columns filled out with all features.
So, after a year, we changed it to something else.
The second year question was kinda weird.
And now it's who's the person who influenced your Go journey the most?
You know, you influenced, other people's, Go journey quite quite a lot, I feel like, you know, look at this red one person who wrote a whole blog post and got to talk about post podcast just saying what you did was bad.
So, you know, some people might, might put you down, but I'm interested to know who you would put down.
Akshay ShahI'm curious to see if we say the same person.
Abhinav GuptaYeah.
For me, it was, our coworker, Prashant Varanasi.
He had a strong influence both on my go journey and in general on my career.
He's very knowledgeable.
I'm sure there are a couple of talks by him you can find on the internet as well.
But on top of that, was like the kindest review code reviewer.
And so in every code review with him, you just like learned a lot about everything you were doing, but also you just learned how to do a review where the person receiving the review felt good about it.
And I following that, like, I kind of modeled how I worked with other people after him.
So I think he had the strongest influence on me and my, you know, go journey.
Akshay ShahThat was really well said.
I would also say Prashant is one of the biggest influences on my career.
I actually switched teams and departments in the company just to go work with him and Abhinav and the rest of them on all this stuff.
Shay NehmadCool.
Well, I hope, he listens to this episode.
Akshay ShahHe lives in San Francisco, so you'll meet him eventually.
Shay NehmadNice.
Nice.
That's where it happens, man.
Well, thanks a lot for coming on the show.
Akshay ShahThank you.
Shay NehmadWe hope to host you again, soon to talk about more of what you're building right now and not, you know, the contentious library you started in 2018 or whatever.
Or maybe actually we need to host you in like 2032 to talk about what you're doing in 2025.
Jonathan HallYeah.
What contentious stuff are you gonna do between now and then?
Akshay ShahJust imagine how many people will be laying into me on the Internet by then.
Thank
Jonathan Hallyou so much, guys.
It's been a real pleasure and an eye opener, honestly.
Thank you.
Akshay ShahThank you.
Abhinav GuptaThanks for having us.
Shay NehmadProgram exit up.
Goodbye.