Episode Transcript
: Tobi: Hello, friends.
Tobi: This is the Alphalist podcast.
I am your host, Tobi.
The goal of the Alphalist podcast is to empower CTOs with the info and insight they need to make the best decisions for their company.
We do this by hosting top thought leaders and picking their brains for insights into technical leadership and tech trends.
If you believe in the power of accumulated knowledge to accelerate growth, make sure to subscribe to this podcast.
Plus, if you're an experienced CTO, you will laugh the discussion happening in our Slack space where over 600 CTOs are sharing insights or visit one of our events.
Just go to alphalist.com to apply.
Tobi: Welcome to the Alphalist Podcast.
I'm your host, Tobi, Tobi: and today's episode has the show title Tobi: Two CTO Dinosaurs versus today's Tech Hype.
And I'm talking to Res, and Res is the city of Ox Money.
Tobi: And yeah, we we fell in love, uh, Tobi: when we first met and and and thought it it makes sense to record an episode where we like talk about all the myths and all the the the the hypes, um, that that, um, waste resources in, uh, modern life and and companies, Tobi: um, especially in tech.
So, um, welcome, Res.
Raz: Thanks.
Thanks for having me, Toby.
Absolutely.
I think, uh, our first talk was, uh, Raz: kind of led into this podcast because both of us were basically kind of venting to each other about, uh, the us being dinosaurs in this new environment, new age environment.
I'm super happy to actually do this and just a, I don't know, like a very big by the way.
I I used to have a podcast myself, Raz: and I have not recorded an episode for so many years because I got sick of speaking in podcasts after recording, Raz: uh, 30 episodes or so.
So, weirdly enough, a bit stressed about doing this.
It's been three, four years since I recorded an episode, so that's kind of nice to, I don't know, put myself in a position of, uh, putting myself under stress a bit.
Tobi: So then I'm super proud to have you here.
Um, and and you're CTO of Ox Money since May 24, so that's like a bit more than a year.
Um, and you personally have like a, Tobi: 17 year history in tech, uh, and are currently arm wrestling, uh, PHP monolith Tobi: into a more modular monolith or like, do you kill microservice again?
Or what's what's your job right now?
Raz: I mean, I think my job right now if I'll need to summarize it is to sign off on risks from tech side, honestly.
Raz: So to the monolith versus non monolith and that will be part of our discussion.
Raz: One of those risks for me was to sign off.
Is it a rewrite?
Is it a straggler pattern to microservices?
Is it a modulus?
Raz: Um, that's also applies for, you know, 17 years, CTO is also a lot of cloud costs, cyber security, data and so on.
So all the fun topics.
So that's basically my day-to-day summarize, I guess.
Tobi: And and before you worked, um, at at Delivery Hero.
Um, so I think you also captured some knowledge about like how to build a platform and how to not build a platform maybe.
Tobi: Um, I don't know if you want to talk about that as well.
And and and you're from Israel originally, right?
Raz: Correct.
I'm from Israel.
11 years in Berlin, but, um, born in Israel.
Tobi: Cool.
Um, so yeah, let's let's get it started.
Super, super pumped to have you here.
Maybe we start with your early years, like, I don't know, did you Tobi: serve in that 2800 unit in Israel?
Or what's what's your history?
Like how did you end up in tech?
Like why why are you here?
Raz: No, no, I'm I'm, uh, my military service is not in the, not in the tech part of the Israeli army.
Raz: Um, honestly, I started, like, I guess many of us, very cliche.
Um, started as a kid, got interested in computers.
I think my parents got me like a 286 when I was nine Raz: with a CGA screen where you basically kind of, and I I pretty sure remember Norton Commander, like the blue thing with no mouse that you can actually have your file system with a with a UI.
And I just kind of fell in love weirdly with trying to figure out how it works.
So like many kids, Raz: tearing up the the the computer and trying to assemble it again was number one task that I started doing.
And then after that trying to figure out how can I write my own software.
Raz: There's like a, if I remember correctly, I remember there was like a doctor something that you can ask questions on DOS and basically give you answers to what it could be, which is just a very nice decision tree for the early 90s.
And I learned Pascal to try to make it work and realized that this is really difficult.
And kind of that's how I started actually writing code as a, I don't know, 10, 11 year old on my free time.
Tobi: And then you accidentally stumbled into the web at a certain point or?
Raz: You have probably 100 years plus experience in PHP.
Do you need anything that PHP doesn't do?
And people are like, not really.
But you know, if we write new services, we can look into that and do something else happily, but even with that, Raz: Basically.
I think the web kind of happened mainly more when I went to university and and I finished my my degrees and so on and job, you know, life brought me into the web.
Raz: Uh, but uh, the passion to write, you know, just nonsense code, C++ and Pascal was uh, was uh, how I kind of got into the idea of even studying computer science, honestly.
Tobi: What happened then?
Like you you got your first job as an engineer or what was that?
Like where what was your first, first, first stop?
Raz: This is where maybe us being a bit grumpy old people.
That's a good story.
Raz: Um, so my first job was actually a QA engineer.
Um, I think I was Raz: in my studies and um, I remember it was approximately 2007.
It was relatively after Raz: um, the bubble burst kind of, so hiring was back again in 2007, but for juniors it was quite tough to find jobs.
So if you're a student developer or a junior developer, jobs were not existing and I will actually paralyze it to today because I know it actually is quite a thing still today.
Raz: And I remember a lot of the people I I went to university with told me, don't don't accept the QA job.
Basically, wait, wait it out because otherwise you'll be pegged as QA.
Raz: Um, I really didn't care because I just wanted to start my career and actually try stuff out and and learn on the go.
I I always find myself to be more of a person that learns by doing rather than learning theoretically.
So I joined as a QA.
I did my role as a QA and then through, you know, the job itself and and showing eagerness, Raz: possession in software engineering kind of came to be.
I also did tons of, you know, automation just played around as QA to say, hey, you don't need to do this thing every day.
I always say that we engineers are lazy mofos.
Raz: We will try to automate everything we can anyway if we see it.
So that mentality came.
And I think this is a really great path for people that are trying to get into tech today Raz: to also be a bit more flexible on the positions and roles and not to just chase a dream job, but rather have an opening, prove yourself and then within the org, it's much easier to move, especially in times where it's a bit tougher because we are currently in a bit of a tough time, especially for juniors.
I think seniors in general when you're good, you'll find the job always.
So kind of like as a as a advice from an old person that done it himself, Raz: the opening to a position is is really great right now.
Tobi: But we we we explicitly said we we don't want to talk too much about AI today.
And you said like, job job market is hard for juniors.
Do you think this will stay like that?
Um, I'm I'm like really again and again thinking about like, maybe it's also a slight chance for juniors because many, uh, like, uh, let's say, older engineers, Tobi: uh, or very senior people, um, do things how they ever did them, right?
Uh, or like to, like to, like to, like basically, uh, continue writing the path, which is now maybe super like, super, super disturbed or super, super hit by a bus, basically.
Yeah.
Tobi: Um, or do you see that?
Raz: Oh, it's a good one.
One, we're now touching the AI topic.
Now, like every cliche podcast for one minute.
No, I'm not complaining.
It's all good.
Uh, I actually think, so it's such a tough topic because we are in the beginning of this, we don't know.
The positives, one, let's let's put it out there, right?
We all both of us know, if you put three seniors in a room to give you a solution, the three of them will have six different solutions and each solution will not work with the other because Raz: senior people, they have history and opinions and, you know, the way I do stuff doesn't necessarily mean that's this is how you would want to do stuff and that's why we need to have a better, I don't know, team dynamics, right?
You want to have maybe 30% of the team is senior, 40% are mids that can do stuff and the other 30% are people that want to learn.
So in in general, we want juniors and um, now comes the other part.
Raz: The good is, I think it's it's never been easier to start as a junior.
It's vibe coding, it's actually testing stuff out.
Raz: I think the path to become a senior will be quite tricky.
I remember when I joined, we just started with Stack Overflow and we were copy pasting stuff from Stack Overflow and just making minor changes.
Yeah.
Now you don't even need to do make, you don't Tobi: need to copy paste anymore.
Yes.
It's Raz: basically writing itself in.
But comes the question, when will you learn and also what will you focus on, but also don't want to be like an old grumpy person because I can share my wife's father, for example, was a developer back in the 70s, right?
And and it's a different world.
And I talked to him about, you know, more clean code principles, craftsmanship principles, solid principles.
He was like, this is all nonsense.
Where where do you manage memory?
How do you like, where is your garbage collection and all those things.
No, no, it's all about readability because we don't care about those things anymore.
It could be that the future will be another obstruction layer where we don't care necessarily about Raz: the bits and bolts of of software engineering.
It's going to be more communication with the machine and trying to figure out what's the business logic.
I don't know.
It's it's uh, it's a weird, crazy environment and I don't, I'm not going to be a person to be quoted then to later say something stupid.
Yeah.
Because I don't know where we're going honestly.
Tobi: Yeah, me me neither.
Me neither.
Let's let's let's skip that for today.
I mean, even if it's a nice topic to like just think about it.
Tobi: Um, when you entered the company, um, when you started at Ox Money, there was, I think no CTO for how many years?
Raz: Basically forever.
Um, I mean, I know that there was once one for a bit, but in general, no, no actual CTO as the definition of CTO that I have for the 17 years of the company.
Tobi: And you basically do peer-to-peer lending, so the, I I could imagine that the, um, there are some requirements, uh, like regulatory requirements, uh, and and also quality requirements that that you need to hit.
Like, did that always work without someone?
Like, what's is is your CEO also like super technical or how how how does that work?
Raz: I mean, just to kind of clarify, Rox Money doesn't do peer-to-peer.
It used to, it started like that.
Right now, it's more, it's consumer lending, so it's basically kind of doing, uh, kind of simple, fair digital consumer lending for for people.
Um, Raz: and now to the question of how, I mean, one, um, we also have a mutual, uh, colleague.
Um, so he had a very, very strong, uh, vice president of engineering, who was basically in touch on the day-to-day software engineering and built it up to the position where we are right now, which is great and amazing to have.
And I'm also, I have the pleasure to work with him still, which is also fun and amazing.
Raz: And of course, there is the parts where, and and this is where I always claim that there always needs to be a C level in the C suite.
That's the position where you sit and you can actually have more of a strategic role where we need to go.
Those things are the things that I'm trying to kind of tackle right now.
But if you're asking me as far as regulatory wise and and what feature-wise and what got Ox money to where it did, I think the team did a phenomenal job.
Raz: Uh, where we want to go is now where I think the position was at some point required that to have as more of an overview of more than the software engineering topics as we all know, data, infrastructure, costs, um, and regulation and implementation of regulation.
These are all parts of the CTO boring, terrible job.
Tobi: So you're now the, the, the, the one who annoys everyone in the room because like, all of a sudden, like things get harder and you like don't continue with microservices or like what's what's your first, what's the biggest battleground then, uh, that that you you hit?
Raz: So, to be fair, when I joined, I hit a battleground quite immediately.
Um, let's start with the first battleground.
no matter what is very cliche to say, but still is building trust.
But the problem is when you join and there's a decision that you completely disagree with, uh, is how do you build trust by shutting down something that the team was pushing for.
So when I joined, Raz: there was a big shift into doing a rewrite of the system or basically disassembling the monolith into microservices.
Raz: My general gut feeling is, as a person who, again, we both are going to this from this point of dinosaur CTO, right?
I remember since 2011, every job I took up to delivery here in 2019, all of them had the same story.
We're hiring people, we have a monolith, we want to, uh, bring the tech team to now disassemble the monolith into microservices.
Raz: In that span of eight years, I've never seen any microservice replace, any rewrite replace the monolith.
Never.
It never happens.
So I was like, unless there is a really, really good business value that we can quantify for this two year insane project, I would say let's pause and think and rethink about this thing.
Tobi: Let's rewrite the whole thing.
Let's, um, I I I would say it's not necessarily microservices, but it's like the CTO who enters and, uh, then starts the rewrite, right?
Like, I don't like this language, I don't like this, I don't like that.
Uh, I like microservices more than monoliths.
Um, or like a team that comes to that conclusion at a certain point, like, hey, we have to get rid of Elixir, we want to move to Node.js.
Tobi: Um, and that takes years and is often like doesn't help anyone, right?
Raz: Um, Absolutely, absolutely.
So just to be fair, when I joined as well, I think, uh, I think in my interview process even someone asked me, what do I think about PHP?
And I said, it works.
Raz: I wouldn't take a tier one critical service and write it in, I don't know, I'm just going to give an example of whatever trending now, Rust, where we can't hire anyone that does Rust.
We have ero experience in Rust.
Our infrastructure, also and every tooling that we've created is not really built for that, so we need to probably rethink a lot of what we have.
Raz: Observability, CICD, yada yada yada.
So all those things becomes hurdles.
It's not going to help us.
So unless Rust solves a problem, I would say let's choose the hammer we know to the nail we already know that works with the hammer and move on.
But you also want to keep it fun, right?
Like we are also engineers and I also want to give the people chances.
So in a good way, Raz: for example, the team decided to try out Golang for some, you know, tier four services.
And that's great because then they can try stuff out, we can learn, and then maybe we'll when, whenever we will need Golang for the tier one part, we can do it.
Raz: But yeah, in general, CTOs coming in and starting with this like, I don't like the language, I don't like this thing, let's rewrite.
Uh, that's um, I don't know, maybe that's uh me being again, dinosaur.
Raz: I find it to be where techies don't understand as a CTO and any management of tech, it's a, it's a business role.
You need to basically look at stuff and say, what is the business value of me now reinvesting all the people I have and their time to do this something?
And what do I get in return?
What's the ROI?
And the ROI for usually those things are nothing, it's vanity.
Tobi: And, um, coming back to like monolith versus microservices, like, Yeah.
what was like, how did you, I mean, you, like, for sure didn't want to do it, um, because you, you went through it like a hundred times and, uh, what did you end up with?
Like, is it like a, like a single monolithic application or is it, is it, is it more like, are there like a few services and more like the, uh, medium sized services, uh, or like, what's your philosophy now?
Raz: So, we are currently still a work in progress.
Uh, when we paused it, we are also pausing to look into it.
Raz: Um, we still have microservices because microservices were built around the monolith in general, so that's great.
That's good progress.
Raz: In general, what we started doing is more figuring it out, what is the obstructions of the team, right?
What is where a team starts and ends as far as our code base and try to really separate that.
And once we can separate it, we basically can create a modular monolith.
From that point on, we can separate the database also, to have databases per something.
And then we can ask, do we need those services to be disconnected?
Then it's easy to disconnect.
We can then decide to do event driven or API based of synchronous or asynchronous topics, or we can just be okay with having a modular model.
I am generally a big fan of modular monolith.
It's uh, people think microservices are, I don't know, the best thing since sliced bread.
Raz: I see overhead, right?
Overhead means more deployments, more infrastructure team, more infrastructure problem, and it's great if you want to scale and you have like, you know, crazy, crazy scalability issues where you need to scale one part but keep the other part low.
But in general, Raz: like, like monorepo, modulus for the scale we're doing is actually not a bad fit.
I don't want to now say it and then everyone in my team will stress out.
It could be that we'll separate them into microservices.
I'm not going to guarantee where we're sticking with a modulus, but I think before you go microservices, Raz: you know, DDD exercise or domain driven, uh, design your your your situation first because that's the first step.
Then separate your your your modules, separate your data and then you can actually see if you need to have any any more work.
That's kind of my take on this.
Hmm.
Tobi: How big is your engineering team now?
Raz: It depends how we look at it.
Raz: Um, our software engineering team.
So fun story, we just transition all our QAs to be software engineers.
We moved away from manual QA with the word we say we will not talk about AI, so we'll have AI actually assisting our QAs to become software engineers.
Um, and at the end of the year, with some hiring, we'll be around 40 software engineers.
We also have 20 people in the data team and about uh, 20 people in the infra.
So that's about 80 people right now.
Tobi: So that's a, that's an effective team, I would say, or like it's it's it's the upper, the upper end of an effective, effective setup, I would say.
Raz: Um, yeah, trying.
Tobi: Um, and um, still you have some services, uh, like basically separated, or like, how do you partition your team these days?
Like how how does that look like?
Raz: Yeah, we had this really awesome exercise before I joined, um, where we basically split the team into value streams.
And the value streams are basically looking at the value chain and everything that every bundle we can create of an, let's call this uh, Raz: in the most extreme and best way, like a cross functional team where we have software engineers, we have POs or product owners, but we also have business people and we have all the necessary functions that are working for the same KPI and we bundle them into one team.
And we basically have four different bundles of that.
So we have customer acquisitions, we have, uh, the the banking relationship, we have the customer management after we convert.
And those teams actually have everything we they need to succeed, not just from software engineering, but all around and and that's also what we're trying to do now with the monolith, to modular it to those functions.
So we basically won't be violating the Conway's law because for 17, well, 16 years, we basically had one team doing everything.
So the model looks exactly like that and now we've Raz: reorg, but our code still looks the same.
So we need to now slowly but surely make the code also fit the reality we live in.
Tobi: So you basically like follow a flow driven approach then now, right?
Raz: Exactly.
Yes.
Tobi: Yeah, yeah.
Sounds, sounds, sounds meaningful.
Tobi: Um, and um, one thing that I I suspect you also hit, uh, when you when you came in was that, uh, like for sure there was Kubernetes.
Raz: A bit, yes, a bit.
Tobi: And do you like, are you a fan of that?
Raz: To be honest, uh, look, uh, I am not a fan of Kubernetes to start with.
I think it's by definition an overkill and it's a waste of time.
Raz: Uh, I can share from, I mean, we talked about my previous employer, deliver here.
Kubernetes was a game changer for deliver here because of the spikes.
So the the interesting part of food delivery quickly is I can put all my money, all in every day.
Um, and bet and I will win most of the time because I know when you are going to eat lunch and dinner, approximately.
So that that spike exists everywhere.
So you basically need that ability to deal with spikes and and and ramp up, rapidly scale up and down, so you need elasticity.
Kubernetes is really, really great at doing that.
Raz: And I would say if your business is like in this spikiness and elasticity of the system, then by all means, go for it.
Raz: Otherwise, it's just another abstraction.
It's another learning point, it's another failing point.
And I if if I will find myself again talking to software engineers that are using, um, liveness probe or readiness probe wrong and killing their system because of that, I would lose my mind.
I I had so many incidents in my career where people were doing a liveness or readiness check on the database and then the database was basically slow.
So the thing just started killing everything instead, which is the worst thing you want.
So we basically say to the customer, due to the database being a bit slow, we will we will take off our software offline.
Yeah.
We take it all offline.
Great.
Great win for me as a customer.
Tobi: I I think, um, my my first Kubernetes project was actually like, uh, moving some WordPress over over to Kubernetes.
That was also horrible and took so much time.
What was the motivation?
I I back back in the days wanted to learn it, uh, which was like absolutely the wrong approach and not not really like, uh, like very business driven.
I mean, ultimately, would say didn't pay off.
Raz: Um, Tobi: and I would never do it again.
And I think that's like the, like really the extreme form of like Kubernetes first syndrome, right?
Raz: Absolutely.
So, I mean, at least you learned something.
It was fun.
Tobi: Sure, it was fun, but, but like these days, like if if someone tells me in an interview, hey, um, yes, I would like to use a vector database.
Yeah.
Tobi: For whatever, I don't know.
Uh, and I don't know what it does.
Um, it it potentially is, uh, the wrong techie to hire, right?
Um, Tobi: how would you approach it these days?
Like would you like, if you, if you, if you could start, if you would start your business from scratch, would you, I don't know, go to AWS and like use as much platform as a service as you can or like manage as much, as many services you could, you could rent or would you go to Hetzner and and buy some bare metal or like what would be your your like those two extremes are or maybe like even one more with I don't know, Fly IO or whatever.
Uh, where would you, where would you go?
Raz: I don't know, to be honest.
I don't know because it depends on what business I would go to.
Raz: Again, like I think, um, I'm a big fan of cloud for elasticity, scalability, and the ease of use, but oh my God, it's so expensive.
Um, and and it gets really expensive really quickly and you when you start with it, you're not even, I don't know, I have a general feeling that Raz: the business model of AWS and GCP and all those cloud providers is also built on the idea that people don't know what they're doing and people will just do dumb mistakes as well.
And that's why it's really profitable.
And also large companies kind of don't care because, I don't know, optimizing 10% of a, Raz: whatever, 10 million, uh, annual cost for a for a company like that where they're probably making billions and billions is not like a big cost factor for them.
So they don't care.
So all those things are tricky because I've been to the bare metal world where I had people working on bare metal and it's tons of work and it's, you know, Raz: I don't want to find myself again in life where I need to go and buy hard drives and go to a server room and then basically increase the disk size by literally putting more hard drives.
Yeah.
Yeah.
But on the other side, I think Hezer is solving a really good problem there where you basically find your midterm solution.
In general, unless I really am going high scale, super elastic environment, I would probably go with like one of those head ones.
I'm just gonna rent a wreck and start from there and then maybe I'll move to the cloud.
I've seen there is uh, what's his name?
Um Tobi: DHH or Yes.
Exactly, exactly.
Where he's like pushing hard and also, god damn it, sorry for the language, but the Twitter story of Elon Musk going in, taking servers with like you whole truck from uh, from a data center and and moving away from that to to their own bare metal data center is also kind of crazy and it was a crazy cost reduction as well.
So Yeah.
Tobi: Yeah.
Um, so but but I think like you can say like it really depends on your workloads, right?
And you now have relatively stable workloads, I I I assume.
Or do you have like really spiky workloads?
Raz: No, I have, I have a stable workload.
Tobi: And and that means that like if you would start from scratch now, like you had the chance, would you go for bare metal plus Kubernetes or would you go for something different?
Like what would be your, Raz: I would probably go Heads now, um, just because I would like to kind of at least reduce my, my overhead as far as employees.
Like I don't want to go into a server room.
Raz: Um, but yeah, probably bare metal relatively easy to use.
I would don't I don't think I'd find myself on AWS for that specific.
Tobi: So you're also private equity backed, right?
So that's that's a difference.
You need to you need to like earn some money or uh, like even generate.
Raz: Yeah.
Exactly.
In general, companies should aspire to should really yes, should really.
I mean, that's what we learned in the last two years, I would say, right?
Absolutely.
No longer, no longer we are pushing for burning every capital possible without regards of anything.
Yes, that's a nice thing to actually have companies making money, profiting.
It's it's not a bad thing.
Tobi: I I remember, like, I I I don't know, I I don't, I don't, I don't, uh, mention at which company it was, but I I once had a team where someone was really in an all hands, uh, developer all hands presenting that, um, they are now top one of the, um, most cloud cost producing team list.
was like someone was very proud of that that fact like, hey, we're now we're now on on number one.
Tobi: And I said, what, what is wrong here?
Like, um, amazing.
How do you, I guess you also then like introduced FinOps, um, at at Ox Money at a certain point.
Like if there was no CTO, then most likely there was no one really like creating awareness for cloud costs.
What what was your approach there?
Um, and um, like and and how high was the cloud cost when you started, how high is it now?
Like just roughly?
Raz: Yeah, um, so my approach honestly was, uh, figure out what are the the low hanging fruit because there is the, you know, when you start with a with this type of role, you also need to figure out what are the battles you want to fight and what are the battles you want to let your future self handle.
That wasn't really a battle that I wanted to start.
Raz: Um, but there were tons of low hanging food.
So on that side, at least, I, um, just talked to the platform team.
Raz: We reviewed our cost.
Our cost was approximately 120k a month.
Uh, and we just started just basically in a very simple level saying, we can actually reduce it quite heavily by almost not doing anything.
Raz: And very simple approach started looking into Raz: our costly levers, right?
So you're going to your AWS cloud account and you basically start seeing and I saw S3 cost was 10k a month.
Why is S3 cost so high?
Like storage is really cheap.
We don't store a lot.
We realized that we have backups of the database that were just not rated.
Yeah, forever.
Raz: Boom, 10k reduced approximately.
Um, so we did it like this really simple exercise.
Just looked every place we have and as we're just seeing that the costs are dropping without interfering to the team because we just never had a Fps, right?
So we never actually had proper tagging where we saw exactly what we have.
So we just look at it from like a macro level, not a micro level, not what the team owns, rather RDS is this, uh, EC2 is this, we don't see what's another deeper sack into that.
Raz: But this will be the future for us.
It's actually proper tagging, um, allocating costs to the team.
And in my, in general opinion, part of software engineering in a good way, and that's a muscle I got from former guest you had here, Christian Harrenberg, uh, is reviewing your costs is part of the engineering work.
So at deliver a hero for example, um, every two weeks, I sat with my vice president or later when I reported to the CTO, to the CTO, and we looked at my costs.
And why they're high and if they're trending up or trending down, but I'm not looking at the absolute cost.
I'm looking cost per transaction.
Raz: And in general, they should most of the time trend down because with large scale and with proper engineering, you can have spikes if you're building something new, but when you're a big company and most of the things you're maintaining, you're building a service per X amount of time, you should basically also be able to, you're in the third phase of development where you build it, you build it right, you optimize.
Raz: Big companies in many cases, you're in the you optimize phase.
Raz: Um, so yeah, that's kind of Ox money but also more on the, maybe I'm kind of trying to to warn my team.
Raz: Very soon we will start having proper tagging, my friends.
And we'll start actually looking into costs.
Tobi: And, um, then you also introduce a metric like, uh, I don't know, cloud cost per per, uh, dollar you rent, you you lend, right?
Um, Raz: Yeah, exactly, exactly.
Cost per transaction would be great.
Yes, cost per loan, let's say, would be a good one.
Yeah.
Tobi: Cost per dollar.
Raz: Cost per dollar, yeah.
Raz: Yeah, that would be amazing.
But yeah, but on the good side, I think we are now currently at approximately 85-ish K, uh, AWS, and I'm pretty sure we can get to 50k a month.
Tobi: And if you move to Hedster, you get to 10, right?
Raz: That would be, no money.
The the yeah, I need to nuke my roadmap for the next two year, but basically yes.
Tobi: Okay, I moved everything to one single server.
It's like a bit over dimensioned, but it runs like the database plus the web service services.
Tobi: Yeah, good idea.
Um, good idea.
Tobi: Uh, I think, um, I mean, you work for deliver here, like, I think one smart guy, like, I once asked like a very smart guy who worked there, like, um, if you were me and now you would, like, I don't know, maintain 22 SAS companies, like, what would you centralize?
And he said like, nothing.
I don't believe in centralization.
Um, and I think you also centralized a lot there.
Um, that's I think in like an, uh, open, open secret.
Tobi: Um, and there is a lot of platform engineering happening.
Yeah.
So, um, Tobi: after, after like data science being now, I don't know, uh, maybe dissolved a bit through like also LLMs entering, like is platform engineering the new ivory tower?
Raz: Oof.
To be fair, um, let's start with your quote here with this person.
I guess I know the same person and I know the same quote.
I think I would still centralize some things.
Um, I think Raz: when you talk about SAS topics, it's easier to centralize where you create like APIs, for example, that many companies can use because it gives a similar service, more configured configurable, easier-ish.
But someone has to maintain it, right?
Correct, exactly.
You need to own it.
Yeah, exactly.
I think the general thing is, oh, we have a team that does it.
Now we will have a team that will do it for 22 companies.
You you probably need three teams to do that for 22 companies.
The complexity is crazy and there is a metric of configuration that becomes really difficult to test, to maintain, and you also want to run properly and you also want to build features for companies.
So it's not like a cost reduction one to one.
It's really far from it.
And in general, unless it's a really huge cost, I would also not centralize just to kind of be fair.
I'm not a fan of centralization as well.
Raz: I think the point where I really, really, really wouldn't touch are the platform engineering part.
My general impression is, I don't know, you are a group CTO, I'm also a group CTO myself.
Every team uses different software, they have different processes and platform engineering is like, like software engineering is the reflection of how customers use your software and your organizational design.
Platform engineering is how developers write code and the processes they have.
Raz: So then when you try to centralize platform engineering, it becomes a massive nightmare in my opinion because you try to now silver bullet the deployment process that will touch many platforms, probably a couple of different cloud providers, code bases, observability, all those things.
I would not touch that really.
I would just make sure that platform engineers are sitting with their team, kind of keeping them close to the reality because the other part that I've seen, you kind of touch base on it, you said it's a ivory tower.
If the platform engineers are now no longer working with the developers, it's the same as, I don't know what old school developers that don't understand their customers and just build stuff for fun.
You'll get the same thing, only you're just destroying your developer, developer experience, which is just insane.
Tobi: Yeah, I think, um, if you, if you do that, you have to basically try to solve business problems really, right?
Or or like, let's say cost problems, right?
Like, let's say I have, I don't know, 800k in S3 costs at one company, then I can tackle that.
And maybe ultimately, it ends up in like an S3 close, close to S3 service that I can offer to to others as well.
But it's more like a marketplace, it has to be more like a marketplace than uh, and like finding a silver bullet, right?
Raz: Absolutely.
I mean, the best thing to look in those things is is you mentioned it and and I think that's great.
It's our job at the end of the day as C levels and CTOs, but still C levels is it's a business job.
If there is a business ROI that we can quantify, then it's worth it.
And we, it's also our job.
I mean, I it's my favorite thing to say about our job is just managing risks.
Raz: If the, if it's not like insane risk and insane effort, by all means, I'm all for it.
I want to reduce cost.
I'm very, I'm very frugal person myself in in real life, not just in the in my job.
Raz: Uh, but always there is I I if there's something I learned and that's something the cost we save versus the opportunity that we can do to invest on this on the business side on something else, it has to be compared and that's where I think having a good product counterpart or a really good C level team to also say, oh, you can save 800, but Raz: we can also invest in this thing and there is a market cap here of I don't know how many billions and we can tap into 50% of that, so it's worth more investing.
So it's always that dance and it's it's a good dance to have.
And I think maybe a good advice for managers in tech in general is to Sure.
look at this from a business side.
Tobi: Sure.
I mean, often there is not that uh, billion market you can quickly tap into, right?
Tobi: I mean, I'm just getting and that's also really hard to to really like, um, attributed correctly, if you if you do that instead.
Like I see many companies like investing into diverse things and never really looking at the ROI.
Um, but but yeah, I I I assume you you're different at Ox money.
Um.
Yeah.
Raz: I mean, I'm I'm I'm we are not different.
I mean, to be honest, there's a lot of gut feeling and I don't think everything needs to be fully quantified.
Um, I'm generally though would always come to try to say, look, at least for me in tech, it's easier because I can say I can save this amount.
And this is the data I have.
And then someone will say, well, we have this, I don't know, this insane ability to do something and I, of course, it never hits really and it's always like questions, but Raz: the always like there comes the, let's say that there is a 1% chance that this is true and a 99% that it isn't, it's still worth exploring probably more than, I don't know, most of the time may cutting some costs because I can always cut those costs later.
Tobi: Yeah.
Raz: Yeah.
Tobi: Yeah.
Makes sense.
Tobi: Um, next, next buzzwords, uh, next buzzword I wanted to touch with you is, uh, velocity.
Tobi: What's your, what's your immediate reaction?
If you, if someone tells you about the velocity, like, what's your, what's your def velocity?
Raz: I laugh, uh, as I did right now.
Um, for me, velocity honestly is, um, one of those things that is actually kind of great from what it was supposed to be.
It was supposed to be a discussion, right?
Like some of us talk about something, we give numbers because, I don't know, let's say I think a feature is a three, you think it's a seven.
And then I say, why isn't it a three?
exactly.
And then you say, oh, you actually are right.
I didn't think about this thing.
Amazing.
Raz: And then later came, you know, the agile coaches and the industry and they started to say, oh, this is a metric.
We can now look at all those numbers and say, this team produces 42 points every sprint.
But the reality is, what is 42 points?
What is a seven and what is a three?
It's nothing.
It's just common language we created.
And in different team might deliver 13 points.
And then we say, if the 42 points is bigger than 13, that's a better team.
But in reality, maybe the 13 point team is actually defining their three and sevens differently, or they're actually touching different systems that are a bit more complex.
So we kind of started assuming weird stuff on velocity.
So I'm generally a big fan of estimations.
I know there is a trend to not to.
I think estimations for the sake of conversations, let's do it.
estimations for a metric, I'm not a fan.
That's I'm grumpy CTO, say let's kill it.
Tobi: No burndown charts for you.
Raz: No, I mean, in general, not a fan of any of those things.
I I have a lovely story if I may.
Uh, 14-ish years ago, so when I was in Israel, so more than 11 years for sure.
I remember and I I've told that to every team I've been to since.
I had a CTO that decided to do a productivity metric, had screens everywhere in the office where it was personal commits per developer and you can see who's number one every day.
Raz: For a month until he gave up on this insane task, every developer, all of us, every line we wrote, enter commit.
We started using C# type of like, uh, indentation where you basically, you you do the curly bracket, you break a line.
You don't like have it on the same line.
You break every line.
Everyone had like hundreds, if not thousands of lines of commits every day.
And then the the thing is the minute you start giving numbers and you tell people that it's productivity metrics, Raz: we will just gamify it, all of us because at the end of the day, we all know it's ridiculous.
We all know it doesn't mean anything.
And in my opinion, most of development work is done when you think about things and not when you necessarily write code.
Best developers I've known were people that wrote less code and solved more problems and or reduced code that I didn't need.
So it's it's tricky.
I wouldn't touch those things.
Tobi: So what's a, what's a valid metric then to, to measure the the productivity of a team or the quality of a team?
Raz: To me, and that's very much to me thing.
I think Dora metrics is is a really good system and I would recommend to people to explore it.
Um, I'm not like a Dora metric as a as a religion type of person.
I from Dora metrics, I like two metrics specifically.
I like, uh, change failure rate.
So the idea is when you go to production, Raz: um, did you have to revert, did you have to roll forward, there was a hot fix or you just fire and forget?
I think that's a percentage of good quality and good basically automation.
It kind of reflects a lot about the processes, the level of automation and and everything.
everything.
So it basically kind of summarize it into one thing and for me that's positive.
And the other one for me is I like to look at the team when things are tough.
So Raz: when there is an incident, what is the meantime to recover?
So for me, a meantime to recover means the team, if it's a good one, like let's say a 15 to 20 minute, a number would be a good number for me.
Raz: It means that one, we have the enough testing and we we can roll back quickly, we response quick, we respond quickly, the team is on it, so they're like motivated.
We have good tooling to also inform them that something is happening.
So it kind of presents that they have probably good observability or they have something that actually is um, good error logging and so on.
Uh, and they're able to adjust quickly and fix.
So if it's not the case, I'm not saying it's good or bad.
I'm just saying it lets me ask questions and and for me, those metrics, again, I'm not going to say that the team with a 15 minute MTTR or meantime to recover is the best team.
It's just usually will be a team that I would ask questions, get answers and be like, think it's fine.
If I see a team with a two hours meantime to recover, I'm just going to say, hey, what's going on?
What is missing?
What is the response time?
So it helps me kind of look deeply into the team.
Because and you know that is again, like you have tons of people, uh, in your org, uh, with what is it?
22 companies?
Right?
Raz: It's your time, you need to spread it between so many different people.
So I don't know, what what are your, what are your go-tos?
Because I also want to learn here what what do you do with such a diverse kind of org?
Tobi: I I think like what you, what you just mentioned, so meantime to recovery, like really like looking at incidents per team and then the time to to to recover is a good thing.
Um, plus I believe that, um, I mean in Dora metrics, Tobi: I'm I'm not sure if it's really like valid.
I mean, you can, I don't know, if you're looking like really top down at a at a at a company and you want to figure out if there are any people not contributing, then yes, you can look at commits etc and requests.
I I also like like pull requests as like a general like orientation.
Um, but but I think it it's also not a silver bullet, so you can't really like do it long term, but rather like that's like if you want to like do a quick health check.
And then, um, Tobi: I think it more, it's more like actually business metrics, uh, that get more important.
Like, I like output or outcome per per employee, basically.
Like what, what, like how many people do you need to generate a dollar of revenue?
Um, and, uh, how, yeah, like the, the relative cloud costs, how does that look like?
To get like a, a feeling for the, the health of, of, of a company, basically.
Yeah.
Um, Tobi: and then yeah, I don't believe in story points.
I believe in also surveys, um, or survey driven approaches, uh, like, uh, Spotify health check, basically, that's, um, one thing.
And also recently, like, I mean, the question is really like, how do you, do you want to actually encourage people to use AI to code or not, right?
And and how can you drive that change, right?
How how can, can you like generate appetite in something?
Yeah.
Uh, that for me is really crucial.
And and and really like what, what, uh, right now, like bothers me a lot, like how can you, I don't know, I I I like for example, um, the the command line tool, um, Anthropic's command line tool called Claude code, right?
Um, I I like that a lot.
But if if I like it, it's not, it's just me.
Like, how do I teach others to to use it properly and how do I drive adoption of that?
Yeah.
How do I drive adoption of best practice, right?
That's really a key question that bothers me a lot.
Raz: Yeah, yeah, that's that's a, that's one of the tougher things, right?
Like, in general, um, something that we're also kind of in a in a good way are playing with right now at Ox Money is Cursor.
Um, and I think in a good way, and I think usually adoption is is that mix of top down and bottom up.
Um, Raz: we had people who were excited about it.
I was excited about it and you just pair with those people and you get to let them and let let their excitement kind of like be spread like wildfire in the org, right?
So what we did is we had a very strong tech lead that uh, said, okay, I've been playing with Cursor at home.
I think it's a game changer.
Can I on board it for a POC with my team?
And I I'm already a friend.
So I was like, Raz: absolutely, yes.
Um, that team started using it and then other people started reaching out saying, hey, can we also use it?
Can we also join this POC?
And then, um, as I mentioned already, we're transitioning the QAs to software developers and we said, okay, let's give it to the QAs.
Why not?
Let's every QA can get it.
And yesterday we opened to the entire company.
So everyone now has a cursor account.
So it kind of just spread bottom up in that sense.
Yeah.
Um, it's not always the case.
Bottom up doesn't work all the time.
Um, but in that case, I was lucky to, you know, find couple of people that were super into it and basically became the people on the ground that were like, uh, you know, spreading the word, spreading the the Lord's word with everyone else.
Tobi: Yeah, I think, um, that that's a good approach because like, I I don't know, like many companies, I don't know, give co-pilot to everyone and then like you end up having basically no usage because you don't like really see the appetite and basically it potentially it's also not, it doesn't have product market fit really.
Like I don't know about like co-pilot these days.
Like, um, no, like, no offense.
Uh, but maybe it it it's not so easy.
It's not like a tool you can throw over the fence and then everyone will start using it, right?
Raz: Yeah, it improved immensely.
I think I I played with CoPilot a month ago, so nothing insane.
It's way, way better.
It's not no longer like a upgraded IDE like it was in the beginning.
But honestly right now, I'm Raz: cursor for me, at least from what I checked, is really above and beyond and the hallucinations are also, I think the biggest problem I had with co-pilot is it had weird hallucinations about a very old and weird code base.
A 17-year-old code base is is is a tricky one, no matter what you do.
And you know, no no throwing shade at anyone who building it.
It's a representation of the business changing itself.
As you mentioned, we were a peer-to-peer company and we kind of shifted, we pivoted.
It reflects in the code, but in the code, like many things, it's uh, when we shifted and pivoted, most likely it was quick and we had to quickly change the code and those things still exist, they linger and Raz: so with co-pilot, I had tons of times where I was kind of like trying to figure out how do I do something and it told me about classes I didn't have and how I can I can improve stuff by removing something that does not exist.
Um, I think cursor actually has a much better way to index that and and at least from the team that I'm, the the response and the feedback I'm getting right now from people is it's much easier to work with.
So let's see with co-pilot.
Honestly, I wouldn't write Microsoft off, but uh also, Raz: I don't know, I would I would also not put them, put it beyond them to basically buy cursor at some point because it's just like a Visual Studio code fork.
Who knows?
Tobi: It's a, it's a, it's a crazy, crazy world out there.
I mean, basically, I don't know, cursor could already be dead in two weeks and like someone killed it through just like having a new approach that everyone jumps on then.
It's, it's, it's worse than like this like, you remember like the single page web app world when it like all started off and everyone said, oh, Google is doing this.
Let let me also jump on that, right?
Like same how how also, I don't know, Kubernetes was initially hyped.
Um, and there was like a new build tool every day.
And to me it feels like there's a tool you can subscribe to for 20 bucks a month every day.
So your tool, your tool belt as a typical developer these days could cost you easily like 500 bucks and like that would be an okay tool belt, right?
Raz: A phenomenal one, honestly.
Um, yeah, you kind of touch base.
I I have those memories of, um, moving from knockout JS to Angular, to Angular 2 that became Angular four, to React, to then Vue, to then React again in a span of four years.
Um, so and then I used all the, you know, the the terminal deployment, the grunt, the the whatever other ones, every couple of days we change those.
Yeah.
There there was a new one.
Yeah, crazy.
What a and yeah, it feels like this is the our version of what's going on with AI right now.
Only Raz: I don't know, I've the AI part is the four years I just talked about happened in the last six months.
Yes.
That is insane.
Tobi: Yeah, and it will, and it will continue.
I mean, everyone's jumping on that, uh, like code generation train.
Uh, so I think that that won't stop, um, in the next years and like really interested, curious to to know what will what will happen there.
Tobi: But yeah, we don't, we didn't want to do a, we didn't want to do an AI podcast, like not not another one.
Um, how about like, I mean, a Fintech company that exists for 17 years without a CTO, like how does does security look like in that setup?
Like what was, I don't know, did you have like a, like any any crazy security moments, any crazy finds, uh, any crazy picks, like anything?
Raz: Um, no, to be honest, on the good parts of this, we have a really good enterprise risk management team and we had a really good security person from the cyber security side of the tech team.
Raz: Um, so for those years, very good, very, I mean, very good job done.
Raz: Um, as I said, it's always the question is what got us here will not get us to the future.
So now we also in a very good way hired an information security officer.
Uh, and we are now working for the future part, which is to fortify us a bit more.
And I know it, I know we said no AI, but this is one of those AI problems that everyone, now when you use AI code, there is this supply chain hack where, you know, you have those libraries that the AI hallucinates, and hackers now actually are piggybacking onto that and now this is will be a problem, a real problem when you write software, where you have a team that accidentally just calls a library that doesn't really exist.
And then someone piggybacks on that and it becomes a Trojan horse, so we need to figure out how to fortify those type of hacks now, which is crazy if you think about the, so AI is such a great thing, but also it's tons of different problems from tons of different angles if you talk about GDPR and data, but also if you talk about cyber security.
It's uh, I read recently, I don't remember where it was, it was a huge supply chain hack with uh, fake libraries of hallucinated libraries basically, which is crazy.
So you need to, it's insane.
Six months, again, we're talking about six months, the world is changing.
Tobi: Yeah, that's super crazy.
Like, uh, like looking looking forward or not so much, uh, to what will, what will happen with security in the next like two months.
Raz: Absolutely.
Absolutely.
Absolutely.
But also in a very much cliche way to be fair, um, we will also take the route of ISO 27,001.
So we now have an information security officer.
Yeah, I know.
Not, not the topic I would love to talk about, uh, you know, with having a beer with a friend in a bar.
Uh, but for a fintech company, for, you know, to building trust with, uh, with the customers and also, you know, going back into this direction of we're no longer a startup, we are now a profitable company, we're doing well.
We need to also start adhering to those standards.
So it's a 27,001 is something that we have now information security officer and we're starting the path of getting there.
Uh, and anyone who's not in fintech, you're all lucky that you don't need to go through that.
It's not fun.
Tobi: So what was your last, let's say, tool you you you discovered and then like basically annoyed everyone with like, in your, in your dev team, like, is there, is there anything?
Like I I tend to like every once in a while discover something and then like tell everyone about it, like, hey, you have to look at this.
This is like better than sliced bread.
Uh, is there anything that that you figured out recently?
And everyone should know here?
Raz: Not really.
I'm trying to think about that.
Raz: Maybe, yes.
I'm not sure.
I'm I'm annoying my team quite a lot, so it's hard.
And oh, I know, I got it.
I got it.
I know.
Raz: Incident IO is something that we are now, uh, bringing into our org.
I'm a big fan of incident IO.
Raz: Um, we were looking into Jira manage service.
Raz: And I think I had the two, like my director, Robert and and Hugo, one of the team leads, were like showing me how it looks like and all of us were like, someone over a weekend will need to go and click in Jira to create tickets for an incident.
And I thought I will never let an engineer go through this nightmare.
I'm like putting myself in my in my engineering head when I was waking up at 3 a.m.
or something because of an incident and I was like, I would never want to go to Jira to do that.
And I think putting everything on Slack, the thing just explains itself, takes all the information you're putting in a channel, summarize it for you.
I'm a big fan of incident IO.
I've been annoying a lot of people about incident IO.
So yes, actually I did.
Tobi: How big of a fan of Jira are you?
Raz: Not a fan.
We're using it.
It works, but if I could nuke it, I would.
Tobi: Yeah, switching, switching, uh, like ticketing solutions, like also like a a classic thing that And you can't trust the ones, right?
Like, yeah, that that's absolutely.
But I had this a long time ago, I used Pivotal Tracker.
I really liked it.
It's dead now.
It doesn't exist anymore.
So I mean, Atlassian, you know, for better or worse, it's probably going to stick around for a while.
Tobi: But for many, GitHub issues is also good enough, right?
Raz: Um, yeah.
Tobi: I'm I'm a fan of that.
Tobi: So, um, Rest, we are slowly coming to the end.
I I still have a little surprise for you.
Um, it's uh, I bought a new car.
It's it's like a DeLorean.
and it's not just a DeLorean, it's the DeLorean.
Um, and uh, like super, super straightforward, we can now use it to travel back in time.
Um, and um, like, where should we travel to?
Like, what is more exciting than Tel Aviv in the year of 2010?
Um, and you're you're at that point still working in QA.
Tobi: Um, starting Selenium scripts and wondering if you should jump to death.
Yeah.
And um, if you had like one, one sentence you could whisper to your younger self, uh, what would it be?
Raz: Honestly, um, don't jump into leadership before you got the dev bug out of your system.
I I did that and then went back to being a developer, which was a mistake.
And also don't look at the QA time as a fault, like as because it took me, I was like two, three years as a QA.
Raz: Um, it will rounded me.
So see it as a positive thing because I think it made me a much better manager.
I look at things from the customer side and not from the how the backend talks to itself.
So I would actually give myself a nice, you know, cheer me a bit because I think at that point in 2010, I was so eager to become a developer.
I was basically annoying every manager possible to give me the opportunity.
And now I look at it and it was I think one of the best educations I've received in my career.
And then later, yeah, jumping into tech leadership before Raz: I had this thing where I was like, I still just want to write code, I was a really bad manager for for my first goal.
I really just didn't care about the people part of the tech leadership, right?
You have the product, the process, the people and the tech.
I didn't care about the product, the process or the people the first go on.
I just wanted to write code and um, this is where you kind of start actually impacting other people.
So semi-optimal, yeah.
Tobi: Okay, now many, many CTOs will go back to coding.
Uh, thank you, Res for that.
Tobi: No, but I I think like actually it will, it will happen anyway that many CTOs will jump back to coding in a way like if you look at the AI hype then or not hype, uh, AI AI happenings then like for many of us, it's super interesting to to get get our hands dirty again, right?
Raz: Absolutely.
At least at least vibe code, right?
Like this is the different part.
Tobi: Right.
Right.
Right.
Um, and I mean for us like vibe coding is a bit different, I think.
Like, uh, thanks a lot, Res.
Um, Thanks so much, Toby.
and, uh, like hope to see you soon at some dinner somewhere in Berlin or Hamburg or whatever.
Um, and have a great day.
was really fun talking to you.
Raz: Same here, same here.
Thanks for having me.
Tobi: You weren't super grumpy, but grumpy enough, right?
Raz: Yeah, I think it was good.
I you need to be you need to be happy when you're grumpy.
Otherwise, it's just depressing.
Tobi: Thank you so much.
Have a great day.
Tobi: Bye.
Raz: You too.
Raz: Bye bye.
Tobi: Thank you for listening to the Alphalist podcast.
If you like this episode, share it with friends.
I'm sure they love it too.
Make sure to subscribe so you can hear deep insights into technical leadership and technology trends as they become available.
Also, please tell us if there is a topic you would like to hear more about or a technical leader whose brain you would like us to pick.
Alphalist is all about helping CTOs getting access to the insights they need to make the best decisions for their company.
Please send us suggestions to cto@alphalist.com.
Send me a message on LinkedIn or Twitter.
After all, the more knowledge we bring to CTOs, the more growth we see in tech.
Tobi: Or as we say on Alphalist, accumulated knowledge to accelerate growth.
See you in the next episode.
Tobi: This podcast is produced by Podstars by OMR.