1 00:00:00,000 --> 00:00:11,120 Welcome to Fork Around and Find Out, the podcast about building, running, and maintaining 2 00:00:11,120 --> 00:00:15,920 software and systems. 3 00:00:15,920 --> 00:00:22,480 Hello, and welcome back to Fork Around and Find Out. 4 00:00:22,480 --> 00:00:26,760 I am your host, Justin Garrison, and with me today is Autumn Nash, as always. 5 00:00:26,760 --> 00:00:30,360 How's it going, Autumn? 6 00:00:30,360 --> 00:00:31,360 Surprising? 7 00:00:31,360 --> 00:00:32,360 Like, I don't know. 8 00:00:32,360 --> 00:00:33,360 The silence, I know. 9 00:00:33,360 --> 00:00:34,360 I know. 10 00:00:34,360 --> 00:00:35,600 There's a lot going on. 11 00:00:35,600 --> 00:00:40,280 Your daily distraction, or your monthly distraction from all the other bad news. 12 00:00:40,280 --> 00:00:43,600 Let's talk about outages and software. 13 00:00:43,600 --> 00:00:46,040 It seems so much better right now. 14 00:00:46,040 --> 00:00:50,720 It's just a lot, okay? 15 00:00:50,720 --> 00:00:55,240 Joining us on the podcast today, we have Ellie Ford, staff software engineer at DevZero. 16 00:00:55,240 --> 00:00:56,240 Thanks for coming on, Ellie. 17 00:00:56,720 --> 00:00:57,720 Of course. 18 00:00:57,720 --> 00:00:59,320 Thank you for inviting me on. 19 00:00:59,320 --> 00:01:06,400 I absolutely invited you on because your blue sky feed was for a time phenomenal for giving 20 00:01:06,400 --> 00:01:10,640 me continued insights into what's going on in data centers and hardware, because I really 21 00:01:10,640 --> 00:01:11,640 kind of miss it. 22 00:01:11,640 --> 00:01:15,840 Like, there's a lot of things that were just like, if you stay away from it for one or 23 00:01:15,840 --> 00:01:17,760 two cycles, like, you just, it's gone. 24 00:01:17,760 --> 00:01:20,400 Like, you just, you don't know anything that's going on anymore. 25 00:01:20,400 --> 00:01:24,160 You go to the cloud for four or five years and you're like, well, this must be state 26 00:01:24,160 --> 00:01:25,160 of the art, right? 27 00:01:25,200 --> 00:01:26,200 No. 28 00:01:26,200 --> 00:01:30,680 Like this is like the amount of performance and price per watt and everything else you 29 00:01:30,680 --> 00:01:35,640 can get by just buying some hardware is really changed over over time. 30 00:01:35,640 --> 00:01:39,480 Yeah, I think like one of the things that has been the most interesting is watching 31 00:01:39,480 --> 00:01:44,640 like the blue sky, you know, the company behind blue sky, like build their own infra and build 32 00:01:44,640 --> 00:01:48,400 it for kind of, you know, a tremendous scale. 33 00:01:48,400 --> 00:01:53,320 They are one of the biggest proponents of like hosting it your own because they too have 34 00:01:53,360 --> 00:01:58,400 realized and done the math and they're like, you know, to run this on AWS and with all 35 00:01:58,400 --> 00:02:04,040 of the egress pricing and all of, like, like all of the miscellaneous Bezos tax that you 36 00:02:04,040 --> 00:02:09,560 pay for, like it is so tremendously cheap to actually just go out and buy hardware and 37 00:02:09,560 --> 00:02:13,640 depreciate it over like seven years and just like stick it in a colo, right? 38 00:02:13,640 --> 00:02:17,280 It doesn't matter if you have like remote hands and like you have to pay for like people 39 00:02:17,280 --> 00:02:18,280 to staff it. 40 00:02:18,280 --> 00:02:22,960 It's just, it is so tremendously cheaper and you get more performance and you get more 41 00:02:23,000 --> 00:02:29,000 control and kind of at the, you know, you trade that off with like you have to do more 42 00:02:29,000 --> 00:02:30,000 capacity planning. 43 00:02:30,000 --> 00:02:33,800 You have to think more through like, okay, you know, we can only handle so much load 44 00:02:33,800 --> 00:02:37,320 on this hardware, but then you kind of get into things like performance tuning and things 45 00:02:37,320 --> 00:02:40,240 like that, which is so much fun. 46 00:02:40,240 --> 00:02:46,920 I mean, I'm a big proponent of people should budget there, like plan to some capacity and 47 00:02:46,920 --> 00:02:49,080 just make a budget and say, I'm not going over this. 48 00:02:49,080 --> 00:02:55,760 And that's like the cloud in general with infinite, infinite scaling credit card bills 49 00:02:55,760 --> 00:02:59,320 is more along the lines of not infinite scaling infrastructure, but you can spend as much 50 00:02:59,320 --> 00:03:02,600 money as you want on anything and never have a plan for it. 51 00:03:02,600 --> 00:03:08,800 There was, so we've been doing a lot of Kubernetes optimization work at my work lately. 52 00:03:08,800 --> 00:03:15,960 And there was a study by DataDog, I think, that showed that I think 80% of containerized 53 00:03:15,960 --> 00:03:19,040 workloads are over provisioned, right? 54 00:03:19,040 --> 00:03:25,400 So like, and it's, it has been crazy, like doing metrics and doing like a detailed rigorous 55 00:03:25,400 --> 00:03:28,760 analysis of like how much waste is actually happening inside of Kubernetes environments. 56 00:03:28,760 --> 00:03:30,560 And it's insane. 57 00:03:30,560 --> 00:03:32,600 And you're paying for all of that too. 58 00:03:32,600 --> 00:03:33,600 Is that new though? 59 00:03:33,600 --> 00:03:34,600 Not really. 60 00:03:34,600 --> 00:03:40,280 Like we used to buy servers and they'd sit there, you know, 10% utilization maybe. 61 00:03:40,280 --> 00:03:43,440 And then we'd like, oh, we need to utilize this more or less use VMs. 62 00:03:43,440 --> 00:03:46,800 And then we had a bunch of VMs that were 10% utilized and then we put it in containers 63 00:03:46,800 --> 00:03:48,360 and those are all, you know. 64 00:03:48,360 --> 00:03:51,520 In every case, I feel like we're just like, like try to densely pack these things more 65 00:03:51,520 --> 00:03:52,520 and more. 66 00:03:52,520 --> 00:03:53,520 And at some point you hide it, right? 67 00:03:53,520 --> 00:03:56,360 We're like, VMware had a memory sharing, right? 68 00:03:56,360 --> 00:04:00,120 You're like, I got 20 VMs on this, but I'm only using the RAM from one of them because 69 00:04:00,120 --> 00:04:03,640 it's all doing the same thing. 70 00:04:03,640 --> 00:04:05,600 Is the waste, is the waste a bad thing though? 71 00:04:05,600 --> 00:04:10,400 Like, I mean, so my background is, is a SRE, right? 72 00:04:10,400 --> 00:04:18,000 So I typically look at waste as capacity rather than actual waste, which very much flies contrary 73 00:04:18,000 --> 00:04:23,320 with like what the bean counters like to think and everything like that. 74 00:04:23,320 --> 00:04:28,880 Especially once you're talking about, you know, like reducing tail latency and like 75 00:04:28,880 --> 00:04:33,640 meeting strict SLAs, it is so much harder to do that when you are on a system that is 76 00:04:33,640 --> 00:04:35,560 like fully loaded. 77 00:04:35,560 --> 00:04:43,000 And one of the things that's been really interesting is like the Linux kernel isn't really well 78 00:04:43,000 --> 00:04:47,960 designed to handle large amounts of load, right? 79 00:04:47,960 --> 00:04:51,840 You think of things like the out of memory killer, the out of memory killer wants to 80 00:04:51,840 --> 00:04:56,840 preserve kernel functionality rather than user space functionality. 81 00:04:56,840 --> 00:05:01,160 So you will have the out of memory. 82 00:05:01,160 --> 00:05:05,400 I have a four by four that says OOM on it for my, it's my, it's my prop for out of memory 83 00:05:05,400 --> 00:05:06,400 killers. 84 00:05:06,400 --> 00:05:10,160 Yeah, it's just so funny. 85 00:05:10,160 --> 00:05:14,600 Like you will have to go out and you will like to make any system functional like Facebook 86 00:05:14,600 --> 00:05:16,080 has found this out the hard way. 87 00:05:16,080 --> 00:05:20,840 You have to have an early OOM daemon that kills processes before it even starts to trigger 88 00:05:20,840 --> 00:05:25,160 the kernel out of memory reaper because it will just like lock up that whole system and 89 00:05:25,160 --> 00:05:26,720 you'll have to power cycle it. 90 00:05:26,720 --> 00:05:32,320 Yeah, the work Facebook and Meta has been doing on a user space OOM killer is fascinating 91 00:05:32,320 --> 00:05:36,520 to me because it just shows that even though the kernel has all the power, it doesn't have 92 00:05:36,520 --> 00:05:40,000 all the context and information for like what's actually important. 93 00:05:40,000 --> 00:05:43,040 And we try to do that on the operating system I work on where it's like, how do we protect 94 00:05:43,040 --> 00:05:44,480 the things that we want to protect? 95 00:05:44,480 --> 00:05:46,680 It's like, we can't just do it in kernel. 96 00:05:46,680 --> 00:05:50,080 Like we have to have something that knows what's actually going on. 97 00:05:50,080 --> 00:05:55,000 Yeah, like you have like niceness values, which are supposed to do that. 98 00:05:55,000 --> 00:05:59,520 But if you've ever looked on like any regular Linux system, you'll see that like some processes 99 00:05:59,520 --> 00:06:04,760 are set as negative 20, which are like never kill and then others are like zero and like 100 00:06:04,760 --> 00:06:07,820 no one actually uses these kernel tunables. 101 00:06:07,820 --> 00:06:13,300 So you get, you quite often get stuck in this situation where you just run into out 102 00:06:13,300 --> 00:06:15,980 of memory and you can't do anything about it, right? 103 00:06:15,980 --> 00:06:18,900 So what sort of performance tuning have you been doing on Kubernetes? 104 00:06:18,900 --> 00:06:19,900 Fine. 105 00:06:19,900 --> 00:06:25,020 You know, you're just trying to eliminate the over provisioning or is it a latency or 106 00:06:25,020 --> 00:06:26,020 what is it? 107 00:06:26,020 --> 00:06:27,820 Not necessarily latency. 108 00:06:27,820 --> 00:06:33,020 So before we started doing the cost optimization work, you know, startups, we pivot all the 109 00:06:33,020 --> 00:06:34,020 time. 110 00:06:34,020 --> 00:06:36,060 We used to do cloud development environments before that. 111 00:06:36,060 --> 00:06:42,660 And a lot of the people at Dev Zero have really like they have worked on developer environments 112 00:06:42,660 --> 00:06:43,660 at Uber. 113 00:06:43,660 --> 00:06:45,260 We have people from Facebook. 114 00:06:45,260 --> 00:06:47,020 We have people from Carta. 115 00:06:47,020 --> 00:06:49,860 A lot of people who have done these developer environments. 116 00:06:49,860 --> 00:06:54,260 And one of the things that we were trying to do is like trying to make it snappy and 117 00:06:54,260 --> 00:06:58,820 responsive and actually functional, right? 118 00:06:58,820 --> 00:07:03,060 We wanted to do everything that an EC2 virtual machine could do, but then better and on your 119 00:07:03,060 --> 00:07:07,540 own infrastructure, which is not particularly easy. 120 00:07:07,540 --> 00:07:11,100 And it was in kind of like a different vein from like what Coder and everything does because 121 00:07:11,100 --> 00:07:15,260 Coder, like to their credit, it's more or less just the Terraform wrapper. 122 00:07:15,260 --> 00:07:18,740 It just provisions that environment for you and that does some like initial setup. 123 00:07:18,740 --> 00:07:21,260 It was, we were a lot more controlled. 124 00:07:21,260 --> 00:07:29,980 There were, I introduced micro VMs into the mix, which made things chaotic and so fun. 125 00:07:29,980 --> 00:07:32,980 I had to debug multiple hypervisor bugs. 126 00:07:32,980 --> 00:07:37,180 It was so gory. 127 00:07:37,180 --> 00:07:43,100 And you know, developers, we were trying to also solve like for developer efficiency 128 00:07:43,100 --> 00:07:47,820 at the same time and respond like responsiveness and everything like that, which basically 129 00:07:47,820 --> 00:07:52,100 meant that like we took it like Ubuntu distribution. 130 00:07:52,100 --> 00:07:54,900 We pinned the kernel at a specific version. 131 00:07:54,900 --> 00:07:59,260 We made our own kernel config with our own options to like slim it down as much as possible 132 00:07:59,260 --> 00:08:01,540 to reduce startup time. 133 00:08:01,540 --> 00:08:06,420 And then we had a custom PID one, like PID one that would run before anything else in 134 00:08:06,420 --> 00:08:11,700 the container that would copy all of the contents over to a persistent block device. 135 00:08:11,700 --> 00:08:14,220 It was, it was batshit crazy. 136 00:08:14,220 --> 00:08:18,220 I think that's kind of the struggle though. 137 00:08:18,220 --> 00:08:24,020 People want to create Linux distributions that have the least amount of what you need, 138 00:08:24,020 --> 00:08:29,220 you know, to do it because for one, it makes it more secure, two, it makes it faster. 139 00:08:29,220 --> 00:08:35,100 But try to figure out exactly what you need to be usable as the hard part balancing that 140 00:08:35,100 --> 00:08:38,700 and actually being usable. 141 00:08:38,700 --> 00:08:41,100 No, absolutely. 142 00:08:41,100 --> 00:08:43,340 It's, it is so much back and forth. 143 00:08:43,340 --> 00:08:45,340 It is, it is tuning. 144 00:08:45,340 --> 00:08:46,840 It is trial and error. 145 00:08:46,840 --> 00:08:51,500 It is, it is like trying 15 different kernel configs to find out which one works, which 146 00:08:51,500 --> 00:08:52,500 one doesn't. 147 00:08:52,860 --> 00:08:59,780 It is, it is an egregious amount of time and I have so, so much newfound respect for people 148 00:08:59,780 --> 00:09:05,540 who maintain distributions after that because I was like, this is, you could probably spend 149 00:09:05,540 --> 00:09:08,820 the rest of your life just tuning the Linux kernel. 150 00:09:08,820 --> 00:09:13,500 And not just that, but like Linux users are very specific about what they want. 151 00:09:13,500 --> 00:09:14,500 You know what I mean? 152 00:09:14,500 --> 00:09:15,700 Like it's, it's its own religion. 153 00:09:15,700 --> 00:09:16,700 Like people. 154 00:09:16,700 --> 00:09:17,700 You calling me out? 155 00:09:17,700 --> 00:09:18,700 What is this? 156 00:09:18,700 --> 00:09:19,700 Yes. 157 00:09:19,700 --> 00:09:20,700 Yes, I'm calling you out, sir. 158 00:09:21,100 --> 00:09:22,100 What do you mean? 159 00:09:23,500 --> 00:09:26,700 I got a new fun Linux distro this week was it Rocknix? 160 00:09:26,700 --> 00:09:30,420 Justin, every other week, he's like, I found a new distro and this is how I'm going to 161 00:09:30,420 --> 00:09:32,180 hurt my own feelings this week. 162 00:09:32,180 --> 00:09:34,740 Like, why are you like this? 163 00:09:34,740 --> 00:09:38,620 It's like you like paint or something for you and all our listeners. 164 00:09:38,620 --> 00:09:41,460 If you haven't compiled the Linux kernel the last couple of weeks, maybe you should. 165 00:09:41,460 --> 00:09:42,460 Maybe you should try. 166 00:09:42,460 --> 00:09:43,460 You see this? 167 00:09:43,460 --> 00:09:45,820 He's like, call these Linux isn't enough. 168 00:09:45,820 --> 00:09:48,820 I have to go with her. 169 00:09:48,820 --> 00:09:49,820 It's okay. 170 00:09:49,940 --> 00:09:51,440 I'm going to in high school. 171 00:09:51,440 --> 00:09:57,580 So just started out like, yeah, choosing violence, just choosing violence. 172 00:09:57,580 --> 00:10:02,540 Like, like I was, I was sitting in like AP human geography and I was compiling Firefox 173 00:10:02,540 --> 00:10:07,100 from source and like, laptop would go to a hundred percent, fans would kick on. 174 00:10:07,100 --> 00:10:09,180 It was, it was a sight to behold. 175 00:10:09,180 --> 00:10:12,580 I feel like you were doing the right thing because I was so bored in high school. 176 00:10:12,580 --> 00:10:15,380 At least she found something that was like fun to do. 177 00:10:15,380 --> 00:10:17,020 Like I was just sitting there doodling. 178 00:10:17,020 --> 00:10:18,020 I don't know. 179 00:10:18,580 --> 00:10:23,580 I think it's like, it's also hard too because there's no like, I think a lot of the other 180 00:10:23,580 --> 00:10:30,300 open source projects that everybody use have this like one united overall project and they 181 00:10:30,300 --> 00:10:34,780 might kind of branch off, but Linux distributions are different. 182 00:10:34,780 --> 00:10:40,620 Like there are so many sources of truth and there's so many like, there's so many different 183 00:10:40,620 --> 00:10:46,100 ways where people will pick the packages they want in their distribution and a lot of open 184 00:10:46,100 --> 00:10:50,980 source and the beauty of open source is maintaining things with a community, right? 185 00:10:50,980 --> 00:10:56,460 So you're not maintaining it by yourself and kind of like having that continuity over the 186 00:10:56,460 --> 00:10:59,140 packages that are built. 187 00:10:59,140 --> 00:11:05,340 And then trying to figure out what you want to continue to have is hard. 188 00:11:05,340 --> 00:11:09,740 It's even like, like when you're building an end user application for a Linux system, 189 00:11:09,740 --> 00:11:12,740 it's like, what versions of glibc do you need to support? 190 00:11:12,740 --> 00:11:13,740 How do you want to package it? 191 00:11:13,740 --> 00:11:14,740 Do you want to use RPM? 192 00:11:14,740 --> 00:11:15,740 Do you want to use dead? 193 00:11:16,380 --> 00:11:21,780 I feel like I have PTSD this week, good lord. 194 00:11:21,780 --> 00:11:29,220 I have learned the correct way is to statically link everything to Musil Libc and ship people's 195 00:11:29,220 --> 00:11:32,700 static binaries because everyone will just figure it out if you give them a static binary 196 00:11:32,700 --> 00:11:34,020 that just works on their system. 197 00:11:34,020 --> 00:11:39,820 But Linux distribution, like okay, when you're, Linux users are a special like, you have to 198 00:11:39,820 --> 00:11:40,820 know your customer, right? 199 00:11:40,820 --> 00:11:45,300 Like when you're building anything, you have to know your customer and like something about 200 00:11:45,340 --> 00:11:50,820 developers, especially Linux developers are like, I want to have all like, it's like people 201 00:11:50,820 --> 00:11:55,860 that like, there's people that use Macs in real life or some sort of like operating system 202 00:11:55,860 --> 00:11:58,660 in real life, but they're Linux developers, right? 203 00:11:58,660 --> 00:12:02,420 And then you've got the people who have to have Linux on everything, Justin. 204 00:12:02,420 --> 00:12:07,340 And then like, like, you know what I mean, like they're different flavors of like they 205 00:12:07,340 --> 00:12:14,100 want all the knobs, or I want it to be bare metal, like, and just do only what I want 206 00:12:14,180 --> 00:12:16,620 it to do and have nothing else, you know what I mean? 207 00:12:16,620 --> 00:12:19,500 Like, so it's like, what is your distribution for? 208 00:12:19,500 --> 00:12:21,740 Is it for the people that want just peer secure? 209 00:12:21,940 --> 00:12:27,700 There's so many different levels and you're building for like, who are, who's your audience? 210 00:12:27,900 --> 00:12:30,180 You know, like, if, yeah, absolutely. 211 00:12:30,220 --> 00:12:35,420 And that's like, like, that's why if I was to tell anyone, like, like any advice 212 00:12:35,420 --> 00:12:38,860 from building cloud developer environments for like the past, like our company's been 213 00:12:38,860 --> 00:12:40,380 doing it for the past three years, right? 214 00:12:40,660 --> 00:12:43,940 If there's anything we've learned, it's like, don't do it, don't do it. 215 00:12:43,980 --> 00:12:45,620 It's so bad, so awful. 216 00:12:45,620 --> 00:12:46,620 It's kind of fun too, though. 217 00:12:48,620 --> 00:12:52,220 Like when you were talking earlier about how you like, and I put this like cloud 218 00:12:52,220 --> 00:12:55,260 vm and I did this and I was just like, it's because you like chaotic things. 219 00:12:55,260 --> 00:13:01,260 Like I think we all have a little bit of like, you just kind of like the fire a little bit. 220 00:13:01,260 --> 00:13:05,060 And then you have to figure out like how much you're like risk tolerances for, you know, 221 00:13:05,060 --> 00:13:08,740 like, but another thing that you mentioned that I thought was interesting is you 222 00:13:08,740 --> 00:13:11,380 mentioned Colos and I've talked to people at Blue Sky who do that. 223 00:13:11,380 --> 00:13:14,820 And it's just, I don't think people realize that that's the intermediate, right? 224 00:13:14,820 --> 00:13:18,140 Like you have a cloud where you completely don't know what's going on. 225 00:13:18,140 --> 00:13:19,380 You have nothing to do with it. 226 00:13:19,940 --> 00:13:24,540 You've got Colos where you buy all the hardware, you can depreciate it, you ship 227 00:13:24,540 --> 00:13:26,420 it, and it's cheaper, right? 228 00:13:26,460 --> 00:13:30,700 But you still have some kind of like, they're your VMs, but somebody else 229 00:13:30,700 --> 00:13:31,580 may be maintaining it. 230 00:13:31,580 --> 00:13:33,020 You might have to pay for hands, whatever. 231 00:13:33,340 --> 00:13:36,700 And then you have completely on-prem, you are writing, running a data center. 232 00:13:36,700 --> 00:13:41,140 And I feel like we talk about on-prem, we talk about cloud, but nobody talks 233 00:13:41,140 --> 00:13:44,860 about Colos and how, like, because we're talking about on-prem, like everyone's 234 00:13:44,860 --> 00:13:45,540 going back to on-prem. 235 00:13:45,540 --> 00:13:48,380 People are not going back to truly writing your own data centers. 236 00:13:48,380 --> 00:13:48,820 Don't lie. 237 00:13:48,820 --> 00:13:49,660 Like, you know what I mean? 238 00:13:50,140 --> 00:13:54,620 Like people keep writing these articles for spicy, like takes and I'm like, 239 00:13:54,620 --> 00:13:56,700 bullshit, like that's not what we're doing right now. 240 00:13:56,740 --> 00:14:01,020 Like people are going to the middle option of like, not today Satan. 241 00:14:01,020 --> 00:14:05,540 I don't want to go back to holding a pager myself, you know, like, but buying 242 00:14:05,540 --> 00:14:06,900 my own hardware is cheaper. 243 00:14:06,900 --> 00:14:10,300 So like, I love that you brought that up because I feel like we, like. 244 00:14:10,780 --> 00:14:14,100 People are writing a lot of articles to be spicy, right? 245 00:14:14,100 --> 00:14:17,780 And like there is value in it being cheaper, but don't lie. 246 00:14:17,780 --> 00:14:19,180 Nobody's doing that kind of work. 247 00:14:19,300 --> 00:14:22,700 And the fact that we like, and that's the wild thing, because all of a sudden we 248 00:14:22,700 --> 00:14:27,180 now need DBAs, we need like people that run data centers. 249 00:14:27,180 --> 00:14:28,860 And not everybody has that talent. 250 00:14:28,860 --> 00:14:34,020 And most of the people that do are these huge companies that do nothing, but, you 251 00:14:34,020 --> 00:14:37,740 know, run people's Colos or the people that are, you know, have these big data 252 00:14:37,740 --> 00:14:38,220 centers. 253 00:14:38,340 --> 00:14:43,100 I find that really fascinating where the, the people that are making on-prem and 254 00:14:43,100 --> 00:14:45,340 data centers sound so cool are the cloud providers. 255 00:14:46,020 --> 00:14:49,580 It's like literally Amazon's like, we're making the best, biggest, coolest 256 00:14:49,580 --> 00:14:51,380 data center ever existed. 257 00:14:51,660 --> 00:14:53,420 And everyone's like, I want one of those, right? 258 00:14:53,420 --> 00:14:54,260 And like, well, let's just use ours. 259 00:14:54,300 --> 00:14:55,380 You can just, you can just pay for it. 260 00:14:55,380 --> 00:14:58,020 It's funny because it goes back to exactly what we're talking about Linux. 261 00:14:58,020 --> 00:14:59,820 It's the different levels of abstraction. 262 00:14:59,820 --> 00:15:01,020 How many knobs do you want to press? 263 00:15:01,140 --> 00:15:03,460 How many, like how much responsibility do you want here? 264 00:15:03,460 --> 00:15:05,780 It's like Macedon versus blue sky. 265 00:15:05,780 --> 00:15:08,700 People were like, yeah, and we're going to have our own stuff and we're going to 266 00:15:08,700 --> 00:15:08,980 do it. 267 00:15:08,980 --> 00:15:10,100 And then we got halfway through. 268 00:15:10,100 --> 00:15:11,420 We're like, this is terrible. 269 00:15:13,700 --> 00:15:15,620 I don't want this much responsibility. 270 00:15:15,620 --> 00:15:16,940 I just wanted it to be fun. 271 00:15:16,940 --> 00:15:19,860 And then we were like, but we don't want to go back to the evil place and the 272 00:15:19,860 --> 00:15:20,580 bad place. 273 00:15:20,820 --> 00:15:25,100 So then they were like, okay, the middle blue sky is the Colos of like social, 274 00:15:25,140 --> 00:15:26,500 of like social media, right? 275 00:15:26,500 --> 00:15:31,460 Like you got like, we don't want to completely be responsible because we do 276 00:15:31,460 --> 00:15:32,300 that all day. 277 00:15:32,300 --> 00:15:34,420 And nobody wants to run their own PDS. 278 00:15:35,380 --> 00:15:35,740 Did it? 279 00:15:35,900 --> 00:15:36,220 Okay. 280 00:15:36,660 --> 00:15:40,180 But did you remember earlier when I said that Justin chooses violence and he 281 00:15:40,180 --> 00:15:42,540 likes to be like, am I wrong? 282 00:15:44,020 --> 00:15:45,820 Fatal masochism is a disease. 283 00:15:45,820 --> 00:15:51,220 I'm telling you, you just got to pick what level of pain that you're just like 284 00:15:51,220 --> 00:15:51,980 excited about. 285 00:15:52,060 --> 00:15:53,780 I have to know how it works and I have to do it. 286 00:15:53,780 --> 00:16:00,220 So there's also this, he made it where he couldn't even like direct message me 287 00:16:00,220 --> 00:16:03,220 and talk to me on this guy for like a month. 288 00:16:03,820 --> 00:16:04,300 Okay. 289 00:16:04,300 --> 00:16:07,700 And then I was just like, Justin, every time I tried to at you and he's like, 290 00:16:07,700 --> 00:16:08,700 no, it's fine. 291 00:16:08,700 --> 00:16:09,900 What are you talking about? 292 00:16:09,900 --> 00:16:10,500 It was a feature. 293 00:16:10,500 --> 00:16:11,980 It was not fine. 294 00:16:12,020 --> 00:16:15,140 Ali, he didn't even like figure it out for a month. 295 00:16:15,260 --> 00:16:15,820 Okay. 296 00:16:16,700 --> 00:16:17,140 Yeah. 297 00:16:17,180 --> 00:16:21,300 So whenever someone talks about on-premise, I show them the picture of the OVH data 298 00:16:21,300 --> 00:16:22,060 center fire. 299 00:16:24,460 --> 00:16:26,540 The big one that just lit on fire. 300 00:16:26,780 --> 00:16:26,980 Right. 301 00:16:26,980 --> 00:16:28,900 I'm like, you want to deal with that? 302 00:16:29,100 --> 00:16:32,100 Not just that, but now people are like, I'm running a server in my grandma's 303 00:16:32,100 --> 00:16:32,540 basement. 304 00:16:32,540 --> 00:16:34,660 Please don't let your grandma's house on fire. 305 00:16:34,660 --> 00:16:38,980 Like grandma did not ask, are you hoping grandma paid this electricity bill? 306 00:16:38,980 --> 00:16:39,660 Like goodness. 307 00:16:39,660 --> 00:16:43,180 And that's, that is one of the things that I find super fascinating, especially 308 00:16:43,180 --> 00:16:47,380 about hardware, which I do want to talk about is how much cheaper, 309 00:16:47,380 --> 00:16:50,900 and quieter, and low power these, especially like the home lab systems, 310 00:16:50,900 --> 00:16:55,340 like the N100 chips are like, they just trickle power and can run a bunch of 311 00:16:55,340 --> 00:16:55,780 stuff. 312 00:16:55,780 --> 00:16:57,300 It's just super fascinating. 313 00:16:59,860 --> 00:17:02,180 We met Matthew works for them. 314 00:17:02,620 --> 00:17:03,940 Oxide, yeah. 315 00:17:04,020 --> 00:17:06,580 Oh my God, their servers are wild. 316 00:17:06,940 --> 00:17:10,820 I have never geeked out about a server so hard. 317 00:17:11,340 --> 00:17:14,060 I mean, usually like, ew, I don't want to touch those things. 318 00:17:15,020 --> 00:17:16,740 But that one, I was like, what? 319 00:17:16,740 --> 00:17:21,140 Like, I'm not buying one of these like, like, and then the tests that they do, 320 00:17:21,140 --> 00:17:23,220 they'll like go with their little mics and you're just like, 321 00:17:25,300 --> 00:17:31,060 I'm like, I have been begging the oxide people to like give me a rack for free. 322 00:17:31,060 --> 00:17:34,420 And I will like, I will do horrendous, awful things to it. 323 00:17:34,460 --> 00:17:35,860 I want to know how you break it. 324 00:17:35,900 --> 00:17:37,500 I want to know how you put it together. 325 00:17:37,500 --> 00:17:38,740 I wouldn't like, you know what I mean? 326 00:17:38,740 --> 00:17:39,980 I'm just like, you can do what? 327 00:17:40,620 --> 00:17:43,420 Like, do you know how many hood rat things you could run on that? 328 00:17:43,420 --> 00:17:45,580 And just so much fun. 329 00:17:46,020 --> 00:17:50,460 You know, I, I've been, we've been talking about, um, like the, the poly 330 00:17:50,460 --> 00:17:57,580 keel, poly keel oxide rack, one of, one of the, uh, you know, understated 331 00:17:57,580 --> 00:17:59,060 things about polyamory. 332 00:17:59,060 --> 00:18:02,300 I know like three people that would be on this like server. 333 00:18:02,300 --> 00:18:07,580 Like, oh, I can't wait to this episode goes live. 334 00:18:07,580 --> 00:18:12,620 I'm just going to be like, I talked about you at Charlotte. 335 00:18:14,420 --> 00:18:17,260 I'm going to be like, I love you so much. 336 00:18:19,620 --> 00:18:23,540 I'm always the friend that like, I told them I want a T-shirt friend of the 337 00:18:23,540 --> 00:18:26,500 poly keel, like poly keel. 338 00:18:27,340 --> 00:18:33,900 I'm just like, you guys keep adding like more people and I just love everybody. 339 00:18:35,900 --> 00:18:36,500 Yeah. 340 00:18:36,860 --> 00:18:42,460 Like the, so the thing is, like hardware is getting more, more like efficient, 341 00:18:42,500 --> 00:18:46,100 more powerful, but then at the same time, the AI craze is happening too. 342 00:18:46,220 --> 00:18:50,340 And that's like, I don't know if you've ever seen an A 100 while it's running. 343 00:18:50,620 --> 00:18:53,900 I don't know if you've ever had to work with an A 100 while it's running, but 344 00:18:54,100 --> 00:18:56,540 it is a passively cooled 400 watt card. 345 00:18:58,780 --> 00:19:01,540 So like, that's a lot of electrons. 346 00:19:01,740 --> 00:19:02,020 Yeah. 347 00:19:02,020 --> 00:19:05,500 That like it, it gets real hot, real toasty. 348 00:19:05,780 --> 00:19:09,300 And that's kind of been one of the funny things about like, I've been working 349 00:19:09,300 --> 00:19:11,780 with NVIDIA hardware for like the past year. 350 00:19:12,220 --> 00:19:15,900 God knows how long before that, um, and like a data center capacity. 351 00:19:15,900 --> 00:19:20,020 And like, no one doesn't like, no one tells you NVIDIA cards. 352 00:19:20,300 --> 00:19:21,700 They get so hot. 353 00:19:21,740 --> 00:19:25,340 They get so hot, so quick, even when they're idle, right? 354 00:19:25,740 --> 00:19:29,980 I have had an A 100 shut down on me because it was passively cooled 355 00:19:29,980 --> 00:19:32,020 and I didn't have any airflow going over it. 356 00:19:32,460 --> 00:19:36,100 And so there are like 3D printed brackets that you will have to get and you 357 00:19:36,100 --> 00:19:41,420 will have to point a whole 120 millimeter fan just straight at the fins to 358 00:19:41,420 --> 00:19:43,900 cool it down because otherwise it will shut down. 359 00:19:44,380 --> 00:19:44,780 Right. 360 00:19:45,580 --> 00:19:46,420 So it's been funny. 361 00:19:46,420 --> 00:19:49,500 Like CPUs are so efficient now, right? 362 00:19:49,900 --> 00:19:54,060 Um, especially once you look at like the arm space, what Apple's doing with 363 00:19:54,060 --> 00:19:56,620 like their M4 chips now, I think they're on. 364 00:19:56,860 --> 00:19:57,060 Yeah. 365 00:19:57,140 --> 00:20:01,660 Um, I got a new M4 laptop and I swear to God, I've never, it was the most 366 00:20:01,660 --> 00:20:06,860 beautiful thing I've ever opened in my, like the things that you can do on 367 00:20:06,860 --> 00:20:10,460 that laptop are just should be like, it's great. 368 00:20:10,460 --> 00:20:14,260 I when the, when the first like ends, like when they were first talking 369 00:20:14,260 --> 00:20:17,260 about switching over to arm, I was so skeptical charcoal ones. 370 00:20:17,580 --> 00:20:21,460 Oh God, they're so, when they were first talking about switching over to arm, 371 00:20:21,860 --> 00:20:23,180 I was like, so skeptical. 372 00:20:23,180 --> 00:20:24,700 I was like, no way in hell. 373 00:20:24,740 --> 00:20:25,420 They pulled it off. 374 00:20:25,460 --> 00:20:26,100 They pulled it off. 375 00:20:26,140 --> 00:20:26,700 I'm sorry. 376 00:20:26,860 --> 00:20:31,460 Like even, even working at a like, like language and runtime and seeing all 377 00:20:31,460 --> 00:20:34,820 the extra stuff you had to do for those M1 M4 trips, worth it. 378 00:20:35,100 --> 00:20:42,420 Well, I'm like, I, I swear to God, like I run exclusively Max. 379 00:20:42,540 --> 00:20:44,300 I had one laptop, right? 380 00:20:44,300 --> 00:20:45,780 It was a terabyte of RAM. 381 00:20:45,780 --> 00:20:48,180 It was like, oh, it's going to be my new gaming computer. 382 00:20:48,260 --> 00:20:51,100 And I would put one here and then I had an M4 laptop. 383 00:20:51,380 --> 00:20:52,940 This fan's like kicking on. 384 00:20:52,940 --> 00:20:53,820 It's dying. 385 00:20:54,020 --> 00:20:59,380 You'd run all the same stuff on an M4 and it's like, oh, like, it's 386 00:20:59,660 --> 00:21:03,380 you're just like, how like there, it's a year difference between these two computers. 387 00:21:03,420 --> 00:21:04,420 It's insane. 388 00:21:04,780 --> 00:21:07,700 Yeah, like, like things are so efficient now. 389 00:21:07,740 --> 00:21:10,100 Well, they have to be because it's the only way we're going to be able 390 00:21:10,100 --> 00:21:13,900 to continue to this whole AI race because there's no way they can continue 391 00:21:13,900 --> 00:21:17,540 to like use this much electricity and water, you know what I mean? 392 00:21:17,540 --> 00:21:20,420 So they're going to have to figure it out and they want to put it on everything. 393 00:21:20,780 --> 00:21:24,780 One of the things that nobody really talks about, it isn't really about the power. 394 00:21:24,780 --> 00:21:28,700 I mean, like the power is obviously one of the big constraints in like growing 395 00:21:28,700 --> 00:21:31,180 from the grid that much power is a huge concern. 396 00:21:31,420 --> 00:21:37,420 And like what, what I think XAI is doing is they have gas turbines at their plant 397 00:21:37,420 --> 00:21:41,060 now, which is like spitting, like spitting emissions into the air. 398 00:21:41,060 --> 00:21:43,860 Polluting, like a small under. 399 00:21:43,940 --> 00:21:48,380 Privilege community that they just keep pretending like they're not in line. 400 00:21:48,380 --> 00:21:49,380 Oh, yeah. 401 00:21:50,260 --> 00:21:55,500 Yeah, like, like the power is one thing, but no one talks about cooling, right? 402 00:21:55,820 --> 00:21:58,900 Like, well, that's why some of them are using water, but it's crazy. 403 00:21:58,900 --> 00:22:01,180 Like, did you see the study they did in Alaska? 404 00:22:01,180 --> 00:22:03,060 They were like, well, just go to the cold. 405 00:22:03,060 --> 00:22:04,700 I'm not Alaska and good drinker. 406 00:22:04,700 --> 00:22:08,500 They're like, we'll build them there and then we'll open the windows. 407 00:22:08,500 --> 00:22:11,140 And I was like, what are you doing to the environment by changing the 408 00:22:11,140 --> 00:22:13,940 temperatures? And then they were like, we'll build one under water. 409 00:22:14,140 --> 00:22:16,580 And it's not like we're already killing fish or anything. 410 00:22:17,460 --> 00:22:20,660 Matt has been doing the open the windows for a long time now, because they have 411 00:22:20,660 --> 00:22:23,020 the, the. Well, they're trying to build them in colder places. 412 00:22:23,020 --> 00:22:26,900 Yeah, well, I think one of the things like they built these data centers 413 00:22:26,900 --> 00:22:30,020 in what used to be colder places that are now getting warmer. 414 00:22:30,540 --> 00:22:34,100 Hey, we need to move closer to the poles because it's close to it's colder there. 415 00:22:35,300 --> 00:22:38,540 Yeah, there's going to be a lot of money for the first person to make a 416 00:22:38,540 --> 00:22:40,620 colo at Antarctica, right? 417 00:22:40,820 --> 00:22:45,180 So they're already trying, but I'm just saying, like, it's really working out 418 00:22:45,180 --> 00:22:47,740 from my home value because Washington's going to be California. 419 00:22:47,740 --> 00:22:49,820 Other one day, well, everything else is hot as balls. 420 00:22:49,820 --> 00:22:51,540 So it's going to be like, great, like. 421 00:22:54,340 --> 00:22:58,340 Now, question, though, if they build a data center or region at the South Pole, 422 00:22:58,340 --> 00:23:01,580 is that, is that a West or an East eliminated? 423 00:23:01,820 --> 00:23:07,580 Can we name it Santa or like, yeah, can I be like, can North Pole be the 424 00:23:07,620 --> 00:23:09,140 like, instead of like East one? 425 00:23:09,140 --> 00:23:11,380 Can it be like North Pole one? 426 00:23:11,380 --> 00:23:16,660 Cause like with like a little like candy, like a candy cane, like emoji. 427 00:23:16,660 --> 00:23:22,300 Emoji regions, that would be like, like East, like West, like, what is it? 428 00:23:22,300 --> 00:23:24,180 West. Let's test all your Unicode. 429 00:23:24,180 --> 00:23:29,500 West should be like a little, like, West should be like a little beach, you know? 430 00:23:29,940 --> 00:23:34,180 And like, like New York East one or two or whatever. 431 00:23:35,180 --> 00:23:36,980 Send us a blue. 432 00:23:36,980 --> 00:23:39,900 What should the emojis be for each region for regions? 433 00:23:42,060 --> 00:23:44,940 Dude, we're going to have, we're going to have a West coast, 434 00:23:44,940 --> 00:23:48,900 like East coast works if you're going to be great. 435 00:23:49,020 --> 00:23:50,140 I love it. 436 00:23:50,140 --> 00:23:53,460 I don't know, but I mean, it's, it's fascinating. 437 00:23:53,460 --> 00:23:55,260 Like you mentioned water cooling, though, right? 438 00:23:55,260 --> 00:23:57,020 It's actually going further than water cooling. 439 00:23:57,020 --> 00:24:01,060 It's like full immersion, like mineral, oil, immersion, cooling. 440 00:24:01,260 --> 00:24:03,300 But they're talking about under the water. 441 00:24:03,300 --> 00:24:04,180 We're right under the sea. 442 00:24:04,180 --> 00:24:05,740 Yeah, there was that. 443 00:24:05,740 --> 00:24:09,700 I think when I was at Facebook, when I was at Disney, Pixar had 444 00:24:09,700 --> 00:24:13,180 a immersion data center, which was fascinating because you walk in there, 445 00:24:13,180 --> 00:24:14,860 it's just quiet because there's no fans. 446 00:24:14,860 --> 00:24:15,980 There's no, it's just like, what's going on? 447 00:24:15,980 --> 00:24:18,980 Like it's hot in there too, because they run at a higher ambient temperature. 448 00:24:19,180 --> 00:24:23,020 So you're in the, in the data center and it's like 85 Fahrenheit or something like that. 449 00:24:23,020 --> 00:24:25,060 You're like, it's, it is muggy in here right now. 450 00:24:25,060 --> 00:24:25,700 What is going on? 451 00:24:25,700 --> 00:24:28,660 It's like, but it was, it was super cool because it was just like, yeah, 452 00:24:28,660 --> 00:24:32,900 we can run these, you know, harder, longer and we have less breakage on things. 453 00:24:32,900 --> 00:24:35,340 It wasn't all their data centers, but it was super fascinating just seeing 454 00:24:35,340 --> 00:24:37,820 how that was playing out in like a real environment. 455 00:24:38,220 --> 00:24:43,100 Yeah, I think, I think in California, I think I remember reading some of the 456 00:24:43,100 --> 00:24:49,740 data centers are there because like they need to prioritize capacity so much. 457 00:24:49,740 --> 00:24:52,740 And you, like the only way to get higher capacity inside of a data 458 00:24:52,740 --> 00:24:54,940 center like that is to go immersion, right? 459 00:24:56,100 --> 00:25:00,740 And so I think like some of the data centers in San Jose, at least last time 460 00:25:00,740 --> 00:25:06,180 I was reading are like fully immersion as well, because it's come so far 461 00:25:06,180 --> 00:25:09,780 and like a couple of years because they want to be able to push the constraints. 462 00:25:09,780 --> 00:25:10,380 And it's amazing. 463 00:25:11,060 --> 00:25:15,460 You, like, do you remember when it was just hobbyists, like the fish tank PC 464 00:25:15,460 --> 00:25:17,860 on like Linus tech tips back in the day? 465 00:25:17,900 --> 00:25:21,060 Do you remember, I remember going to PAX a couple of years ago and people were 466 00:25:21,060 --> 00:25:25,500 walking in with their like PCs underwater, but it was to be cool. 467 00:25:25,500 --> 00:25:27,020 Like it wasn't like, you know what I mean? 468 00:25:27,020 --> 00:25:29,140 Like, no, like you need it to like run. 469 00:25:30,940 --> 00:25:36,020 The bubbles coming off of this also the dedication to pull, to like drag a whole 470 00:25:36,020 --> 00:25:39,740 ass PC down to like Seattle, like props. 471 00:25:42,140 --> 00:25:43,780 It's, it's kind of funny. 472 00:25:43,780 --> 00:25:47,580 It's like, there's like, it's very much a push and pull factor. 473 00:25:48,500 --> 00:25:50,220 I think we're going to keep with that push and pull. 474 00:25:50,220 --> 00:25:52,340 It's always, it's like, oh, cloud. 475 00:25:52,340 --> 00:25:55,700 And then it's like data centers and like, there's always a limit, right? 476 00:25:55,700 --> 00:25:58,060 You go, you find the next limit and you work around it. 477 00:25:58,620 --> 00:26:01,740 Not even just the limit though, it's whatever people want to worship that day. 478 00:26:01,900 --> 00:26:08,020 You know, religion, like, and then you're just like, there, there's a real tools 479 00:26:08,020 --> 00:26:11,660 that you could just kind of weigh your options, but then it's just like, what, 480 00:26:11,660 --> 00:26:15,100 like, what, what direction is the religion going on today? 481 00:26:15,100 --> 00:26:19,540 And people are going to find their version of social media to argue to the 482 00:26:19,540 --> 00:26:23,500 depths, even though there could be like a middle argument and there's like a good 483 00:26:23,580 --> 00:26:25,580 use case for both, but you know, keep going. 484 00:26:26,340 --> 00:26:28,860 Cause it wouldn't be fun if you weren't arguing. 485 00:26:29,500 --> 00:26:31,420 No, I, well, yeah. 486 00:26:31,580 --> 00:26:35,540 And you know, like so many push and pull factors. 487 00:26:35,540 --> 00:26:41,020 I mean, like, because the internet is growing and continuing to grow, like we 488 00:26:41,020 --> 00:26:43,940 are getting faster and faster networks, right? 489 00:26:43,980 --> 00:26:48,420 I think they are now testing like 1.6 terabit optics. 490 00:26:48,700 --> 00:26:49,500 Just wild. 491 00:26:49,500 --> 00:26:53,060 Do you remember how long it took to like, download a movie back in the day? 492 00:26:53,100 --> 00:26:54,220 Do you remember being on like. 493 00:26:54,660 --> 00:26:58,260 I mean, I think one of the things that we're trying to talk about is like the 494 00:26:58,260 --> 00:27:04,460 history of computing, like back in 2013, 2014, like it was rare to see a 40 gigabit 495 00:27:04,460 --> 00:27:05,420 network even. 496 00:27:05,740 --> 00:27:11,100 I think like at the high end, people would have 10 gig now, but like, you can 497 00:27:11,100 --> 00:27:16,300 go talk to an ISP and get a dedicated 10 gig fiber line for about a thousand 498 00:27:16,300 --> 00:27:17,780 dollars a month right now. 499 00:27:18,180 --> 00:27:18,780 I know. 500 00:27:18,780 --> 00:27:24,140 Cause I just talked to like Zeo and, and astound they're a local ISP in 501 00:27:24,140 --> 00:27:25,100 the Seattle area. 502 00:27:25,540 --> 00:27:31,220 Um, and it's like, it is, it is so cheap to get 10 gig now. 503 00:27:31,820 --> 00:27:37,060 Um, and even on the hobbyist end, you can get a hundred gig equipment for under 504 00:27:37,060 --> 00:27:38,020 $2,000. 505 00:27:38,180 --> 00:27:38,580 Right. 506 00:27:38,900 --> 00:27:43,740 I used Melanox switch will only run you about $1,500. 507 00:27:43,780 --> 00:27:44,500 Don't tell me that. 508 00:27:46,540 --> 00:27:48,060 No, no, no, poor Beth. 509 00:27:48,060 --> 00:27:50,020 She's just going to have to put up with this now. 510 00:27:50,060 --> 00:27:52,100 She'd be like, Justin, what are you doing in the garage? 511 00:27:53,100 --> 00:27:58,420 I'll send you the eBay list because, because we just built a lab off of like a 512 00:27:58,420 --> 00:28:02,260 hundred don't help this on eBay. 513 00:28:02,260 --> 00:28:03,420 Literally yesterday. 514 00:28:04,420 --> 00:28:06,340 Have you seen his garage? 515 00:28:06,340 --> 00:28:07,660 Look behind him. 516 00:28:07,660 --> 00:28:09,460 He doesn't need any help. 517 00:28:10,340 --> 00:28:16,700 So the secret is, and before the tariffs, it was like the Chinese are 518 00:28:16,700 --> 00:28:21,540 incredibly good at recycling data center equipment and, and turning it into 519 00:28:21,540 --> 00:28:25,420 hobbyist grade equipment so you can get, um, Bluefield twos. 520 00:28:25,420 --> 00:28:28,660 They're, they're insanely cool, um, network cards. 521 00:28:28,660 --> 00:28:30,100 I can get into them a little bit more. 522 00:28:30,220 --> 00:28:32,620 They're a hundred gig capable, actually 200 gig capable. 523 00:28:33,020 --> 00:28:37,540 And like, you can get them for like $300 each, which is the same. 524 00:28:37,620 --> 00:28:38,620 Are those the ones you were running? 525 00:28:38,620 --> 00:28:40,700 They have like, you can run a full operating system on them. 526 00:28:40,700 --> 00:28:40,940 Oh yeah. 527 00:28:40,940 --> 00:28:41,260 Yeah, yeah, yeah. 528 00:28:41,260 --> 00:28:43,460 I run, I ran a blue sky PBS on one of them. 529 00:28:44,060 --> 00:28:48,500 Don't give it on a network card. 530 00:28:49,420 --> 00:28:50,980 I remember, I remember you posted that. 531 00:28:51,060 --> 00:28:55,260 And I spent the entire night watching YouTube videos about these cards because 532 00:28:55,260 --> 00:28:57,780 I didn't know anything about all these different hacks that people were doing. 533 00:28:57,820 --> 00:29:00,420 I'm not going to be able to add him again for like another month. 534 00:29:00,420 --> 00:29:01,420 Cause he's going to break it. 535 00:29:01,740 --> 00:29:04,380 So, so that's funny. 536 00:29:04,380 --> 00:29:08,780 This has been a whole research article or like whole research point of research for 537 00:29:08,780 --> 00:29:09,020 me. 538 00:29:09,020 --> 00:29:13,300 There's a blog post about this, uh, at my work blog. 539 00:29:13,780 --> 00:29:17,340 Um, I was going to say, if you're not on blue sky, are you talking about this 540 00:29:17,340 --> 00:29:17,780 somewhere else? 541 00:29:17,780 --> 00:29:19,860 Cause like then we'll lose all the information. 542 00:29:20,860 --> 00:29:23,540 Come back to us and just like, don't, just ignore everybody. 543 00:29:23,580 --> 00:29:26,380 Like, don't actually talk to people. 544 00:29:26,380 --> 00:29:28,580 Just post it and then just, just post. 545 00:29:29,660 --> 00:29:31,740 Come back, just drop the mic after. 546 00:29:34,140 --> 00:29:34,620 Yeah. 547 00:29:35,020 --> 00:29:41,140 Um, it's like, NVIDIA is doing some insanely cool stuff with these DPUs. 548 00:29:41,180 --> 00:29:44,220 The blue field, two cards, I mean, $300 each. 549 00:29:44,500 --> 00:29:49,260 They have a full arm VA core on them and they're like tied to a 550 00:29:49,260 --> 00:29:51,500 Kinect X seven network adapter. 551 00:29:51,660 --> 00:29:55,700 And just like it's in a really weird way where that arm chip has like full 552 00:29:55,700 --> 00:30:00,100 control over that network adapter and create like pseudo interfaces that show 553 00:30:00,100 --> 00:30:05,060 up on the host that are actually routed through that arm core out to the 554 00:30:05,060 --> 00:30:08,260 public internet through the ethernet port on the back or technically it's an 555 00:30:08,300 --> 00:30:09,100 SMV port. 556 00:30:09,460 --> 00:30:13,060 It's going to get to a point where we have so many of these fast ships that 557 00:30:13,060 --> 00:30:15,820 they will like just be everywhere. 558 00:30:15,820 --> 00:30:17,900 Cause it seems like, you know what I mean? 559 00:30:17,900 --> 00:30:20,260 Like they just keep building them faster and faster. 560 00:30:20,260 --> 00:30:22,980 So people are going to want the new ones and it's just like, we're just 561 00:30:22,980 --> 00:30:25,180 going to end up with this crazy surplus of chips. 562 00:30:25,620 --> 00:30:29,780 Well, not only just a crazy surplus, but just so many stacked on layers of 563 00:30:29,780 --> 00:30:31,700 like abstraction and complexity. 564 00:30:31,700 --> 00:30:36,900 I mean, I really don't envy someone who has to learn all of like the basics of 565 00:30:36,900 --> 00:30:38,180 PC hardware from scratch. 566 00:30:38,220 --> 00:30:40,660 And then now you're throwing on like DPUs. 567 00:30:40,860 --> 00:30:45,380 You're throwing on all of the extra complexity of like, oh, you know, how 568 00:30:45,380 --> 00:30:48,060 do you handle a hundred gig, gigabit line rates, right? 569 00:30:48,140 --> 00:30:49,180 Accelerator cards. 570 00:30:49,260 --> 00:30:53,740 My little brother was learning how to build his own PC like two years ago and 571 00:30:53,740 --> 00:30:58,460 he was trying to add more RAM and like get a new chip because he was running 572 00:30:58,460 --> 00:30:59,180 video games. 573 00:30:59,660 --> 00:31:03,820 And I remember like, you couldn't find like the crappiest card anywhere. 574 00:31:03,900 --> 00:31:04,900 Like it was wild. 575 00:31:05,380 --> 00:31:10,300 And I'm like, what are we going to do when they've got like, just at some point 576 00:31:10,300 --> 00:31:13,340 there's going to be too many chips because they keep making new ones, you 577 00:31:13,340 --> 00:31:15,340 know, what's going to be like, I wonder how that affects. 578 00:31:15,940 --> 00:31:19,820 One of the things I was thinking of just as I speak about hardware data 579 00:31:19,820 --> 00:31:25,780 centers is just that recycle factor of like companies handing out old hardware 580 00:31:25,780 --> 00:31:29,820 to employees that are interested in the amount of things that I learned over 581 00:31:29,820 --> 00:31:34,900 the years by having free access to random compute things that I didn't know 582 00:31:34,900 --> 00:31:35,740 what to do with or not. 583 00:31:35,740 --> 00:31:39,300 Like I've, I've taken my fair share of rackmount servers home, not knowing 584 00:31:39,300 --> 00:31:43,220 that I could, if I could power or cool it, but it was still interesting to be 585 00:31:43,220 --> 00:31:44,780 able to say that you didn't burn your out. 586 00:31:44,780 --> 00:31:47,940 So in the accident, this was in a dorm room, right? 587 00:31:47,940 --> 00:31:51,980 Like this is just taking out the whole door. 588 00:31:54,060 --> 00:31:55,140 There's someone burning popcorn. 589 00:31:55,140 --> 00:31:55,740 Don't worry about it. 590 00:31:57,540 --> 00:31:57,740 Yeah. 591 00:31:57,740 --> 00:32:00,900 Like this is, I feel like we, a lot of people miss out on that, right? 592 00:32:00,900 --> 00:32:05,780 Cause so many of these like college students and early career people are only 593 00:32:05,780 --> 00:32:08,980 working in cloud providers and they don't actually get access to know how 594 00:32:08,980 --> 00:32:13,140 this stuff works at a lower level, be able to freely break something and have 595 00:32:13,180 --> 00:32:16,740 constraints around like, I'm not going to, it's not going to accidentally charge 596 00:32:16,740 --> 00:32:20,980 me $10,000 if I, you know, like for loop this Lambda function, right? 597 00:32:20,980 --> 00:32:23,940 Like at some point, there's different trade-offs though, like a lot of times 598 00:32:23,940 --> 00:32:27,140 you get extra credits and you can experiment, but it'd be, give people 599 00:32:27,140 --> 00:32:28,900 access to both, you know what I mean? 600 00:32:29,820 --> 00:32:34,540 But I mean, is, is there anything like the constraints of a, here's a box 601 00:32:34,540 --> 00:32:39,180 that runs something like here's a, here's a DPU or even a GPU or an old desktop 602 00:32:39,180 --> 00:32:42,900 or something like the amount of home labs I've built on old desktops is just 603 00:32:42,900 --> 00:32:45,780 because I was like, I always had free access to them and it was fascinating 604 00:32:45,780 --> 00:32:49,860 how much I could learn within that constraints of like, oh, what can this do? 605 00:32:50,140 --> 00:32:51,900 So it's funny that you mentioned this. 606 00:32:51,900 --> 00:32:54,420 This is kind of one of the things that I've always wanted to do. 607 00:32:54,500 --> 00:32:56,980 Um, there's this project called cloud lab. 608 00:32:57,100 --> 00:33:03,180 It's run by, I think like the university of Georgia or, or like some, maybe Clemson, 609 00:33:03,180 --> 00:33:07,300 Clemson, yeah, Clemson, the university of Utah participate. 610 00:33:07,380 --> 00:33:11,580 And they basically built this like software defined data center where you 611 00:33:11,580 --> 00:33:15,060 can rent some of this hardware for like academic research use. 612 00:33:15,420 --> 00:33:19,060 And like, I just think like that's the type of things that needs to be in the 613 00:33:19,060 --> 00:33:20,540 world more, right? 614 00:33:20,980 --> 00:33:25,740 You need to give like so many people who don't have access, who can't 615 00:33:25,780 --> 00:33:28,380 afford this hardware, because a lot of it is so expensive. 616 00:33:28,420 --> 00:33:29,740 Like you need to give them access. 617 00:33:29,740 --> 00:33:33,300 Otherwise, like, how do you expect anyone to really get knowledgeable? 618 00:33:33,420 --> 00:33:36,900 I was going to say, cause like, I mean, I do think that having a box is great, 619 00:33:36,900 --> 00:33:41,020 but sometimes you may not have the room, you may not have one stable place to 620 00:33:41,020 --> 00:33:45,140 live, like there's so many different, like, I know people who were living in 621 00:33:45,140 --> 00:33:49,100 their car and using libraries, like to do development work. 622 00:33:49,100 --> 00:33:51,380 So like, I think there's a place for everything. 623 00:33:51,420 --> 00:33:55,180 I do think something that has no limits that you can break and fix is definitely 624 00:33:55,180 --> 00:33:56,140 different than the cloud. 625 00:33:56,540 --> 00:33:57,860 Yeah, no, absolutely. 626 00:33:57,860 --> 00:34:01,740 Like I, like me personally, like I've moved so much, right? 627 00:34:01,740 --> 00:34:07,900 I, I just finished up a move from North Carolina to Seattle and I, it is, I had 628 00:34:07,900 --> 00:34:11,660 to pack up all of the servers in my house that were just sitting there and 629 00:34:11,660 --> 00:34:12,820 shipped them across the country. 630 00:34:12,820 --> 00:34:15,220 And a lot of them got broken along the way. 631 00:34:15,500 --> 00:34:22,340 And it's just like, like, it is such a privilege to have so much space and so 632 00:34:22,340 --> 00:34:26,380 much, like all of that power available in order to be able to power this equipment. 633 00:34:26,380 --> 00:34:28,340 But there are a lot of people who don't, right? 634 00:34:28,780 --> 00:34:32,940 And I definitely do think that, like, there will always be a market in making 635 00:34:32,940 --> 00:34:33,860 these things accessible. 636 00:34:34,820 --> 00:34:35,180 Yes. 637 00:34:35,620 --> 00:34:38,220 Because like, there's just surpluses, right? 638 00:34:38,540 --> 00:34:42,460 Like, because of the way that this hardware is depreciated and like the way that 639 00:34:42,500 --> 00:34:46,980 it's written off, like to a company is basically useless, probably after like 640 00:34:47,380 --> 00:34:49,300 maybe three or four years, right? 641 00:34:49,340 --> 00:34:51,260 And just the advances they're going to get new. 642 00:34:52,180 --> 00:34:57,820 Yeah, and it's, it's been so cool because you will see, like a lot of Chinese 643 00:34:57,820 --> 00:35:03,940 companies that are solely like built around recycled or like secondhand, you 644 00:35:03,940 --> 00:35:08,620 know, enterprise hardware, which is like, it's no longer in fashion mainstream and 645 00:35:08,620 --> 00:35:12,540 like current data centers, but it's like still valuable, right? 646 00:35:12,860 --> 00:35:15,700 That's been, that's been like the fascinating thing to watch about deep 647 00:35:15,700 --> 00:35:16,700 seek, right? 648 00:35:17,420 --> 00:35:21,420 They basically, you know, their principles are all correct. 649 00:35:21,460 --> 00:35:22,820 They're using the right stack. 650 00:35:22,820 --> 00:35:24,260 They're doing the right things. 651 00:35:24,260 --> 00:35:27,540 I, I am so bullish on deep seek. 652 00:35:27,660 --> 00:35:32,940 If anyone is getting to AGI, which, you know, I don't know if that's even a possibility. 653 00:35:32,940 --> 00:35:37,380 I, I remain skeptical, but if anyone is going to do it, I definitely think it's 654 00:35:37,380 --> 00:35:39,540 going to be the Chinese, right? 655 00:35:39,900 --> 00:35:43,500 I think the fact that they were willing to put that clearly opens up. 656 00:35:44,180 --> 00:35:45,060 Yeah, yeah. 657 00:35:45,260 --> 00:35:46,220 That's ballsy. 658 00:35:46,260 --> 00:35:49,900 That's ballsy, you know, saying I know that we're really, really good because 659 00:35:49,900 --> 00:35:51,460 I'll let you have the code and see all. 660 00:35:53,140 --> 00:35:55,460 That's how, that's how you disrupt markets anyway, right? 661 00:35:55,460 --> 00:35:59,700 Like you make the thing that someone else finds valuable and just run it to zero. 662 00:35:59,700 --> 00:35:59,900 Right. 663 00:35:59,900 --> 00:36:01,140 Like that's what Kubernetes exists. 664 00:36:01,420 --> 00:36:06,420 As Google said, we wanted everyone to rethink how to deploy applications and we 665 00:36:06,420 --> 00:36:07,940 thought containers were better than VMs. 666 00:36:07,940 --> 00:36:09,740 And so we wanted to ruin Amazon's business. 667 00:36:09,740 --> 00:36:10,740 So Kubernetes exists. 668 00:36:10,940 --> 00:36:14,260 I don't think it's that, but I also think it's the Kubernetes community because 669 00:36:14,260 --> 00:36:17,020 that's probably the coolest community I've ever, well, sure. 670 00:36:17,060 --> 00:36:17,260 Yeah. 671 00:36:17,260 --> 00:36:18,780 Like, like the people. 672 00:36:19,980 --> 00:36:24,420 I think that they had to do things differently because they had other open 673 00:36:24,420 --> 00:36:26,660 source containerization things that exist. 674 00:36:26,660 --> 00:36:30,860 Me, so, but even Google, like Google had, let me container that containerize that 675 00:36:30,900 --> 00:36:38,740 for you and like these, there was, there, there was Mesa's, there was no mad where 676 00:36:38,740 --> 00:36:43,940 the two that were, were around at the time, Kubernetes, um, yeah, and it's funny 677 00:36:43,940 --> 00:36:50,540 too, if people contributed so much that that is the most diverse, like people 678 00:36:50,540 --> 00:36:56,420 with pink hair, people that are brown trans with just all the things. 679 00:36:56,540 --> 00:36:59,900 And I like, like you walk into a Kubernetes conference and I'm like, I 680 00:36:59,940 --> 00:37:02,780 want to run this stuff, even if I don't need to use it because I want to 681 00:37:02,780 --> 00:37:04,020 hang out with all these people. 682 00:37:04,060 --> 00:37:09,140 Like, yeah, like, like people like Ian Coldwater are my, like, are my role. 683 00:37:09,140 --> 00:37:13,860 Dude, like you actually, like, there's so many open source projects where 684 00:37:13,860 --> 00:37:17,180 they're just like old, grouchy people and you're like, oh, I like your stuff. 685 00:37:17,180 --> 00:37:23,740 But like, y'all, like Kubernetes, you're like, oh, it's painful, but the people are 686 00:37:23,740 --> 00:37:24,180 rad. 687 00:37:25,140 --> 00:37:30,340 It's, it's kind of like the untold secret that trans people tend to make up a 688 00:37:30,340 --> 00:37:34,060 lot of the backbone of most distributed systems, which is why I think it's 689 00:37:34,060 --> 00:37:36,100 funny when they're like, oh, we don't care about them. 690 00:37:36,100 --> 00:37:38,380 And I'm like, yeah, we tell your infrastructure to like breaks. 691 00:37:39,540 --> 00:37:40,700 Don't call anybody. 692 00:37:41,380 --> 00:37:42,780 They're not coming to help you. 693 00:37:43,100 --> 00:37:47,180 It's, it's like an unspoken fact that trans people and furries tend to make up 694 00:37:47,300 --> 00:37:49,580 most of this nation's critical infrastructure. 695 00:37:49,700 --> 00:37:53,260 Dude, it's, it's like when people are like, oh, we don't like immigration. 696 00:37:53,420 --> 00:37:58,700 I'm like, you shouldn't eat the most like unseasoned, like horrible food for 697 00:37:58,700 --> 00:37:59,660 the rest of your life. 698 00:37:59,660 --> 00:38:01,100 You're not allowed to have a taco. 699 00:38:01,100 --> 00:38:01,540 Okay. 700 00:38:01,580 --> 00:38:04,700 Like I just, nothing unless it's like the, just. 701 00:38:05,420 --> 00:38:08,740 I hope you eat the most bland food for the rest of your life and also no 702 00:38:08,740 --> 00:38:09,700 infrastructure for you. 703 00:38:09,900 --> 00:38:10,140 Like. 704 00:38:11,020 --> 00:38:14,620 So, so I grew up half Japanese in Salt Lake City, Utah. 705 00:38:14,900 --> 00:38:15,420 So. 706 00:38:15,540 --> 00:38:16,140 Oh, girl. 707 00:38:20,260 --> 00:38:25,260 So, um, Mormon too, Mormon too, should, should mention Mormon. 708 00:38:25,620 --> 00:38:25,980 Right. 709 00:38:25,980 --> 00:38:31,860 So I, I, I was an assumption made Salt Lake City that you got it. 710 00:38:31,860 --> 00:38:33,460 You have to clarify assumptions. 711 00:38:34,380 --> 00:38:34,620 Yeah. 712 00:38:34,620 --> 00:38:34,820 Yeah. 713 00:38:34,820 --> 00:38:35,340 You got it. 714 00:38:35,460 --> 00:38:35,740 Yeah. 715 00:38:35,740 --> 00:38:36,700 It was Mormon. 716 00:38:36,980 --> 00:38:39,380 It is, it is, it is shocking. 717 00:38:39,820 --> 00:38:42,540 Um, and then I moved to North Carolina right after that. 718 00:38:42,540 --> 00:38:44,780 And I was like, wow, food can taste good. 719 00:38:46,420 --> 00:38:49,740 It's like, it's like the, like, I don't know. 720 00:38:49,740 --> 00:38:54,420 It's like the great, I don't know, like it gets like, it's like the universe's 721 00:38:54,420 --> 00:38:56,620 way of being like, you don't deserve good food. 722 00:38:58,180 --> 00:38:59,540 The equalizing factor. 723 00:38:59,580 --> 00:39:03,860 You know, like I grew up a jove witness and I was like, all my friends who came 724 00:39:03,860 --> 00:39:08,180 from very, very, very, very religious backgrounds are all like so much fun. 725 00:39:08,180 --> 00:39:10,140 And I was like, you can tell we didn't get to do anything when we were little. 726 00:39:15,780 --> 00:39:16,380 Yeah. 727 00:39:16,540 --> 00:39:21,220 But it's, it's funny that you mentioned Kubernetes too, because if you really 728 00:39:21,220 --> 00:39:24,980 start looking at the background on like who's pushing a lot of the needle forward 729 00:39:24,980 --> 00:39:29,580 on a lot of this Kubernetes research too, isn't really the big cloud providers. 730 00:39:29,580 --> 00:39:34,700 It isn't really like, like the giants in the U S of a lot of the stuff that I've 731 00:39:34,700 --> 00:39:41,060 been looking at it, like overlay BD, um, is, it's, it's a fascinating project. 732 00:39:41,060 --> 00:39:44,340 If you've not looked into overlay BD, they basically made like a block device, 733 00:39:44,340 --> 00:39:48,780 which remotely streams in its content from a, a Docker registry. 734 00:39:48,780 --> 00:39:52,100 Like that's, that's I think a 10 cent project. 735 00:39:52,380 --> 00:39:52,780 Right. 736 00:39:53,340 --> 00:39:54,780 Um, GCP has that, right? 737 00:39:54,780 --> 00:39:56,660 GCP has a way to, to stream. 738 00:39:56,660 --> 00:39:57,060 Yeah, yeah, yeah. 739 00:39:57,060 --> 00:39:58,420 Like East RGZ. 740 00:39:58,820 --> 00:39:59,300 Yeah. 741 00:39:59,500 --> 00:40:07,260 It's, I, I like there, I think one of the main, main tanners behind container D is 742 00:40:07,300 --> 00:40:11,220 also like these Japanese Akiha Arosuda, right? 743 00:40:11,540 --> 00:40:12,340 Is this handle? 744 00:40:12,340 --> 00:40:18,980 I think, like there are people across the ocean who are kind of leading the way on 745 00:40:18,980 --> 00:40:23,420 a lot of this Kubernetes research and it's kind of fascinating. 746 00:40:23,540 --> 00:40:27,660 If you look at it all, like just all the way, that's why like I'm just like, 747 00:40:27,700 --> 00:40:32,220 when people say that like diversity work is like some sort of like you're doing 748 00:40:32,220 --> 00:40:33,780 it to be nice and help people. 749 00:40:33,780 --> 00:40:37,220 I'm like, um, have you ever heard of the bottom line and building things? 750 00:40:37,420 --> 00:40:40,500 Like, cause like this is not charity work out here. 751 00:40:40,820 --> 00:40:43,740 Like you are not doing anything for people. 752 00:40:44,100 --> 00:40:49,740 Like you, to get the best people that are like really into whatever they're 753 00:40:49,740 --> 00:40:54,500 building and they're just doing it, you have to go everywhere and allow everyone 754 00:40:54,500 --> 00:40:58,500 there, you literally get better products and more of a return on investment. 755 00:40:58,740 --> 00:41:02,260 Some of the smartest people I've ever worked with were high school dropouts. 756 00:41:02,380 --> 00:41:04,780 And that's, that's like a bizarre factor. 757 00:41:04,820 --> 00:41:06,460 I'm a college dropout myself, right? 758 00:41:06,460 --> 00:41:07,140 I didn't get to finish. 759 00:41:07,180 --> 00:41:11,780 Some of the best people, college dropouts, theater degrees, absolutely no 760 00:41:11,780 --> 00:41:13,180 real computer site. 761 00:41:13,220 --> 00:41:18,180 Like, and it's, it makes me sad because I've ran into so many times where we're 762 00:41:18,180 --> 00:41:21,620 like, you don't have real computer science to create or like, like, and I think 763 00:41:21,620 --> 00:41:26,220 that we've got this elitist vocab, you know, that if you don't sound exactly 764 00:41:26,220 --> 00:41:31,260 the same as other people that people think that you like, or if you didn't go 765 00:41:31,260 --> 00:41:36,180 to a fancy school and I'm like, some of the best technologists I know have 766 00:41:36,180 --> 00:41:41,020 invented half of these as vocab words and like, don't have that degree. 767 00:41:41,060 --> 00:41:42,380 Like that's ridiculous. 768 00:41:42,420 --> 00:41:47,100 Like, yeah, I think that's been one of the most interesting factors. 769 00:41:47,140 --> 00:41:54,180 Like, because, because of like open source technology is it is so, so like, it 770 00:41:54,180 --> 00:41:59,100 is so easy for people from underserved communities to like break into tech by 771 00:41:59,100 --> 00:42:01,060 just contributing to open source projects. 772 00:42:01,300 --> 00:42:02,500 That's how I got my break. 773 00:42:02,580 --> 00:42:05,940 It also helps you to be able to play with technology because it's all 774 00:42:05,940 --> 00:42:09,860 there in the open and there's usually good documentation or doctors will 775 00:42:09,860 --> 00:42:10,580 have a walkthrough. 776 00:42:10,740 --> 00:42:16,420 Yeah, it's, it's like, it's like, I really admire tail scale. 777 00:42:16,460 --> 00:42:21,980 I really admire oxide, how they all have their core basically open in public 778 00:42:21,980 --> 00:42:22,900 for people to look at. 779 00:42:23,140 --> 00:42:23,300 Right. 780 00:42:23,300 --> 00:42:26,060 A lot of people don't have the ability to do internships. 781 00:42:26,060 --> 00:42:28,780 Internships are especially like unpaid ones. 782 00:42:28,780 --> 00:42:29,980 That's for rich kids. 783 00:42:30,100 --> 00:42:36,100 Like, not everybody can do, not everybody can move to Seattle or New 784 00:42:36,100 --> 00:42:41,420 York or Silicon Valley and how are you supposed to pay the extra high rent for 785 00:42:41,420 --> 00:42:44,180 three months, even if you are making some like, you know what I mean? 786 00:42:44,180 --> 00:42:48,740 Like, people don't realize how much of the traditional process of getting into 787 00:42:48,740 --> 00:42:55,020 tech is gatekeeper by like your social economic class about where you're able 788 00:42:55,020 --> 00:42:56,500 to get like your education. 789 00:42:56,500 --> 00:42:57,460 Like there's so much of it. 790 00:42:57,460 --> 00:43:01,820 And I think open source is the great equalizer and 70% of infrastructure 791 00:43:01,820 --> 00:43:02,860 comes from open source. 792 00:43:03,940 --> 00:43:06,780 And that's the other thing I was saying with like the trickle down of free 793 00:43:06,780 --> 00:43:07,260 hardware, right? 794 00:43:07,260 --> 00:43:11,020 Like, cause that goes, it ends up going far and wide just beyond the person 795 00:43:11,020 --> 00:43:14,580 that gets it, like the amount of computers that I've, I've got from companies 796 00:43:14,580 --> 00:43:17,860 and then also given away to other people that I'm like, Oh, you should get into 797 00:43:17,860 --> 00:43:17,980 this. 798 00:43:17,980 --> 00:43:22,140 I play hockey with a, with someone and they work a service job and they seem 799 00:43:22,140 --> 00:43:23,900 really kind of apps to technology. 800 00:43:23,900 --> 00:43:25,580 And I'm like, Hey, I have a spare laptop. 801 00:43:25,580 --> 00:43:27,780 Do you want it so that we can do some programming classes? 802 00:43:27,780 --> 00:43:29,900 And they're like, I've never had a laptop before. 803 00:43:29,980 --> 00:43:33,220 I'm like, I always give my laptops away because I just definitely the kid that 804 00:43:33,220 --> 00:43:33,860 didn't have one. 805 00:43:34,140 --> 00:43:34,340 Yeah. 806 00:43:34,340 --> 00:43:38,100 And it totally has a second order effect of like, if you have, you have more 807 00:43:38,100 --> 00:43:42,460 time than ability at some point, you're like, just give me access to something. 808 00:43:42,740 --> 00:43:46,540 And I will bang my head against the desk until I figure something out in, in 809 00:43:46,540 --> 00:43:49,340 for systems work, for programming, for all sorts of things. 810 00:43:50,100 --> 00:43:54,180 We can't replace the desire to learn with more training, right? 811 00:43:54,180 --> 00:43:56,700 Like the, the amount of curiosity is one thing you can't train. 812 00:43:57,380 --> 00:43:57,580 Yeah. 813 00:43:57,580 --> 00:44:00,660 And like the, the amount like, Oh, we're just going to train up a new group 814 00:44:00,660 --> 00:44:01,620 of people to do this thing. 815 00:44:01,620 --> 00:44:02,500 I'm like, no, you're not. 816 00:44:02,500 --> 00:44:04,460 Like if they're not interested, they're not going to learn it. 817 00:44:05,100 --> 00:44:08,540 Not just that, but the rabbit hole and the research that you will go and you 818 00:44:08,540 --> 00:44:12,460 will learn when you are just like, want to solve a problem. 819 00:44:13,020 --> 00:44:16,860 Like that's why the best projects come from like somebody wanting to solve 820 00:44:16,860 --> 00:44:17,580 an issue that thing. 821 00:44:17,980 --> 00:44:18,980 Or prove someone wrong. 822 00:44:19,300 --> 00:44:23,420 Oh, oh my God, my life is spike driven development. 823 00:44:23,420 --> 00:44:24,340 I swear to God. 824 00:44:24,340 --> 00:44:25,340 Like, oh my God. 825 00:44:25,660 --> 00:44:25,980 No. 826 00:44:25,980 --> 00:44:29,580 So like this has kind of been one of the most frustrating things about 827 00:44:29,580 --> 00:44:31,780 watching modern computer science education. 828 00:44:31,820 --> 00:44:34,060 Like I, I am a colleague interviewing. 829 00:44:34,100 --> 00:44:35,100 Don't forget the interviewing. 830 00:44:35,220 --> 00:44:38,580 The, the interviewing process is frustrating, but like watching people 831 00:44:38,580 --> 00:44:42,740 who are so reliant on chat, GBT and like don't want to learn anything. 832 00:44:42,740 --> 00:44:46,140 It's so frustrating because I'm really scared for future developers. 833 00:44:46,140 --> 00:44:50,020 Like I, I think we're going to make people like dumber, but I think 834 00:44:50,020 --> 00:44:51,980 that AI could be useful in certain ways. 835 00:44:51,980 --> 00:44:55,220 But I just, it's so weird to me that they're shopping it as the black 836 00:44:55,220 --> 00:44:58,260 box of box of answers instead of this new way to learn. 837 00:44:58,260 --> 00:45:02,540 Like I, it's so weird to me that that would be so like also just the way that 838 00:45:02,540 --> 00:45:09,340 we, the fact that you can have AI do process on Apple phones and everything 839 00:45:09,340 --> 00:45:10,620 is processed in chip. 840 00:45:11,180 --> 00:45:12,260 That's fricking amazing. 841 00:45:12,300 --> 00:45:12,860 You know what I mean? 842 00:45:12,860 --> 00:45:17,220 Like there's like, there's so many cool ways that we could use this to help 843 00:45:17,220 --> 00:45:21,140 people and we're just, let's make them dumber. 844 00:45:22,380 --> 00:45:23,700 It's a really weird flex. 845 00:45:24,580 --> 00:45:28,700 I have, I have such a nuanced opinion about AI because on the one hand, I 846 00:45:28,700 --> 00:45:29,860 think it's so cool. 847 00:45:29,860 --> 00:45:32,340 Like this is the next frontier of natural language processing. 848 00:45:32,340 --> 00:45:36,060 Like you can talk to a computer and like literally just plain text. 849 00:45:36,060 --> 00:45:38,100 This can be your rubber ducky. 850 00:45:38,140 --> 00:45:42,300 It can help you like when you're a neuro-spicy person and you're trying 851 00:45:42,300 --> 00:45:45,100 to figure out how to interact with ready, but like, does this sound dumb? 852 00:45:45,140 --> 00:45:46,140 How do I rewrite this? 853 00:45:46,140 --> 00:45:47,460 Like, you know what I mean? 854 00:45:47,460 --> 00:45:51,500 Like here's all my neuro-spicy ideas that I'm too excited to make into words. 855 00:45:51,500 --> 00:45:52,820 Can you help me make it into work? 856 00:45:52,820 --> 00:45:58,380 Like, you know, like a structured like paragraph from my bullet list of ideas. 857 00:45:58,380 --> 00:46:03,260 Like, yeah, but like at the same time, like I, I spent so much time around my 858 00:46:03,260 --> 00:46:06,820 friends who are still in college who are just graduating and like, like chat 859 00:46:06,820 --> 00:46:11,660 GPT has completely changed how they kids don't want to write paragraphs. 860 00:46:11,700 --> 00:46:12,020 Yeah. 861 00:46:12,540 --> 00:46:14,420 They don't, they don't actually try. 862 00:46:14,420 --> 00:46:16,300 They just feed everything into chat. 863 00:46:16,300 --> 00:46:20,420 GPT use the output, like output, turn it in, maybe change some words here and 864 00:46:20,420 --> 00:46:21,660 there so to like avoid that. 865 00:46:21,660 --> 00:46:26,260 Yeah, I believe all the M dashes, which is funny because I feel like we did. 866 00:46:26,780 --> 00:46:30,220 Remember when they told people that you can't use an ID, you have to use Vim. 867 00:46:30,900 --> 00:46:34,460 And then they were like, Oh, well, like you can't copy and paste. 868 00:46:34,460 --> 00:46:37,540 You got to make your projects for like, people have been doing this shit for 869 00:46:37,540 --> 00:46:38,260 forever, right? 870 00:46:38,260 --> 00:46:41,580 But like, I think this is just like, it gives you such a finished product. 871 00:46:42,020 --> 00:46:46,820 That's so tailored to what you want before you could go copy some code, but 872 00:46:46,820 --> 00:46:50,340 you had to make it, you had to change enough to do what you want it to. 873 00:46:50,660 --> 00:46:55,420 You know, the one, the one radicalizing thing about learning about higher 874 00:46:55,420 --> 00:46:59,900 institutions and how education is actually conducted there is it isn't like 875 00:46:59,900 --> 00:47:03,220 not only is the curriculum more advanced, the cheating is more advanced, right? 876 00:47:03,500 --> 00:47:06,420 They get so creative on cheating. 877 00:47:06,420 --> 00:47:11,020 There are whole cheating rings formed and like disbanded on a daily basis. 878 00:47:11,020 --> 00:47:12,900 There are discord servers created. 879 00:47:13,220 --> 00:47:16,900 So every now and then I want to give them a little award like that dude who did 880 00:47:16,900 --> 00:47:21,180 the Trojan horse and the like, and he like bullied the maintainer into 881 00:47:21,180 --> 00:47:22,420 making him a maintainer. 882 00:47:22,460 --> 00:47:23,940 Like, should you do that? 883 00:47:24,100 --> 00:47:25,900 No, was it creative? 884 00:47:26,020 --> 00:47:27,700 You get an A plus for creativity. 885 00:47:27,700 --> 00:47:32,100 And I want to teach this in a computer science cyber class, but that was mean. 886 00:47:36,060 --> 00:47:44,100 Yeah, it's, you know, that is one of the things that I, it just, I, I think we 887 00:47:44,300 --> 00:47:47,060 don't do a good job of making people curious. 888 00:47:47,060 --> 00:47:50,340 And that's one of the things, but in a way that is curiosity and problem 889 00:47:50,340 --> 00:47:52,300 solving, but they're using it for bad. 890 00:47:52,300 --> 00:47:53,660 It's like when you have kids, okay? 891 00:47:53,900 --> 00:47:57,540 My kids use their intelligence to do little evil things. 892 00:47:57,540 --> 00:47:59,460 And I'm like, you could have done this the right way. 893 00:47:59,820 --> 00:48:01,580 And it would have took less brain power. 894 00:48:09,100 --> 00:48:13,140 Okay, I might have been talking crap about you earlier, but that was so rude. 895 00:48:14,740 --> 00:48:19,660 Oh, would you work with people that are your friends and they know you for real? 896 00:48:19,660 --> 00:48:20,420 It's so hard. 897 00:48:23,420 --> 00:48:25,180 Also, my kids worship him. 898 00:48:25,420 --> 00:48:26,020 I know. 899 00:48:26,500 --> 00:48:29,100 I was like, damn, and he's not wrong. 900 00:48:29,100 --> 00:48:35,020 So I don't even have a rebuttal to the heart. 901 00:48:38,860 --> 00:48:42,660 What do you think is interesting in hardware in the next five years? 902 00:48:42,740 --> 00:48:46,500 I was like, all these accelerators right now are like, everyone, I feel 903 00:48:46,500 --> 00:48:50,460 like everyone's just like throwing more electrons at AI right now, but it's 904 00:48:50,460 --> 00:48:52,020 not super interesting to me. 905 00:48:53,580 --> 00:48:57,540 Yeah, I think, I think one of the most fascinating things is like, people are 906 00:48:57,540 --> 00:49:01,540 now kind of revisiting the basics when it comes to the data center technology. 907 00:49:01,820 --> 00:49:02,860 Bye, Autumn. 908 00:49:03,820 --> 00:49:08,260 Um, it, and like, there's so much talk now about disaggregation, right? 909 00:49:08,460 --> 00:49:13,420 Because now you have GPUs, which are basically their own whole, like they 910 00:49:13,500 --> 00:49:15,060 consume the whole unit. 911 00:49:15,060 --> 00:49:18,820 They are basically the majority of the power unit of a lot of these rack units. 912 00:49:19,340 --> 00:49:23,420 Um, there's a lot more talk about, um, making them into their own standalone 913 00:49:23,420 --> 00:49:26,820 rack units and connecting over CXL. 914 00:49:26,860 --> 00:49:30,820 That's the standard that's coming out, I think, and it's going GA this year. 915 00:49:31,260 --> 00:49:36,020 Um, and people, you know, putting in more memory than what like these servers 916 00:49:36,020 --> 00:49:40,500 are physically capable of, um, like dynamically allocating and swapping 917 00:49:40,500 --> 00:49:43,980 around memory between servers without ever turning them off. 918 00:49:43,980 --> 00:49:47,180 It's, there's, there's so much cool research there. 919 00:49:47,260 --> 00:49:51,180 I had a call with both MemVirge and Liquid. 920 00:49:51,220 --> 00:49:57,180 Those are two startups that are, um, very hot and playing in the disaggregation 921 00:49:57,180 --> 00:50:00,180 space and it is insane what's going on there. 922 00:50:00,540 --> 00:50:05,980 It is, I, so what they're able to do is they basically realized, 923 00:50:06,020 --> 00:50:09,980 okay, PCIe is a switched, um, switched interconnect, right? 924 00:50:09,980 --> 00:50:13,620 It is, it is, if you squint, it's like a network protocol, right? 925 00:50:13,740 --> 00:50:14,820 And so they basically made it. 926 00:50:14,820 --> 00:50:20,780 So they break out the PCIe slot from inside of the computer into a copper 927 00:50:20,780 --> 00:50:26,060 like interconnect that can go 15 meters, 10 to 15 meters. 928 00:50:26,060 --> 00:50:31,380 And so you're able to have these individual thin, thin, um, you know, 929 00:50:31,460 --> 00:50:34,020 thin racks, thin, thin rack units. 930 00:50:34,140 --> 00:50:38,300 And then you're able to have all of these copper interconnects into, um, 931 00:50:38,500 --> 00:50:42,860 like a GPU box where you have like eight GPUs all in like a four U enclosure. 932 00:50:42,860 --> 00:50:44,620 And that's all it is, right? 933 00:50:44,700 --> 00:50:48,940 There's, you know, power, there's a power distribution system that's more 934 00:50:48,940 --> 00:50:53,620 equipped to handle all of that load for the, those GPUs, you know, or, or like 935 00:50:53,620 --> 00:50:56,980 even you could do it from like another rack unit over, right? 936 00:50:58,140 --> 00:51:00,540 That's been one of the fascinating things. 937 00:51:00,700 --> 00:51:05,780 When I, when I see like that in NVME over fabric and these things that are 938 00:51:05,780 --> 00:51:10,260 just like, let's just put everything as a network protocol and, and connect it 939 00:51:10,260 --> 00:51:13,580 over, over more generic cabling, right? 940 00:51:13,580 --> 00:51:15,100 This is basically what we're doing. 941 00:51:15,500 --> 00:51:19,740 And then I, when I envision it, like initially I think that like a rack, 942 00:51:19,740 --> 00:51:24,980 an entire rack is basically going to look like what a desktop PC looks like today. 943 00:51:24,980 --> 00:51:27,980 Like if I open my desktop PC, I got the CPU on this side. 944 00:51:27,980 --> 00:51:29,340 I got the storage over here. 945 00:51:29,340 --> 00:51:31,260 I got the GPU down here, whatever else. 946 00:51:31,260 --> 00:51:35,820 And like we're just doing that at the rack scale to say, Oh, it's fast enough 947 00:51:35,820 --> 00:51:40,500 to connect these things together with the network or some fabric connectivity. 948 00:51:40,500 --> 00:51:41,340 It's close enough. 949 00:51:41,340 --> 00:51:44,580 So now we can just like, I always envision like pulling off the side of a 950 00:51:44,580 --> 00:51:48,420 rack, like a big door and it's like a big PC where I'm looking at, okay, I'm 951 00:51:48,420 --> 00:51:51,260 going to change the CPU cooler here or something like that. 952 00:51:51,500 --> 00:51:53,940 But that, that seems like where we're going, where we want these, these 953 00:51:53,940 --> 00:51:57,620 specific units that do one thing. 954 00:51:57,620 --> 00:52:01,740 And we say, Oh, we need to swap the, you know, compute Bay two or something. 955 00:52:01,740 --> 00:52:01,940 Right. 956 00:52:01,940 --> 00:52:04,140 We got like, pull it out, hot swap a CPU out. 957 00:52:04,140 --> 00:52:04,380 Okay. 958 00:52:04,380 --> 00:52:05,580 We fix that, put it back in. 959 00:52:05,860 --> 00:52:06,620 It's back into. 960 00:52:07,020 --> 00:52:07,300 Yeah. 961 00:52:07,340 --> 00:52:08,660 That's, that's where we're going. 962 00:52:08,700 --> 00:52:13,180 I like, like, you know, that saying it's turtles all the way down. 963 00:52:13,380 --> 00:52:14,740 It's computers all the way down. 964 00:52:14,860 --> 00:52:15,100 Right. 965 00:52:15,140 --> 00:52:15,420 But yeah. 966 00:52:15,420 --> 00:52:20,100 And that's the thing I have a question about is at that point, every one of 967 00:52:20,100 --> 00:52:24,620 those components has an arm chip attached to it or something that is its 968 00:52:24,620 --> 00:52:25,660 operating system. 969 00:52:25,820 --> 00:52:26,180 Mm hmm. 970 00:52:26,980 --> 00:52:29,260 What is the operating system for the rack? 971 00:52:29,540 --> 00:52:29,860 Right. 972 00:52:29,860 --> 00:52:32,580 And I don't think Kubernetes is the right abstraction for that. 973 00:52:33,860 --> 00:52:38,020 I, I think like there has been some thought there. 974 00:52:38,100 --> 00:52:42,900 I think open stack was a very interesting player in that space. 975 00:52:42,940 --> 00:52:47,060 I would probably say they were the closest to get to get right, but it was 976 00:52:47,060 --> 00:52:48,580 just too complicated. 977 00:52:49,300 --> 00:52:54,540 Um, I, I been looking at other projects, open Nebula. 978 00:52:54,540 --> 00:53:02,900 They're, uh, you know, it's like no one has really figured that out. 979 00:53:03,380 --> 00:53:03,820 Right. 980 00:53:04,140 --> 00:53:09,620 I, I think the enterprise is kind of standardizing around Kubernetes as is 981 00:53:09,620 --> 00:53:14,220 right now, but it'll kind of be very interesting to see what comes out of this. 982 00:53:14,420 --> 00:53:18,620 Um, because I don't think Kubernetes provides the lower level abstractions 983 00:53:18,620 --> 00:53:23,460 that people want to manage that hardware, uh, especially if it's a, a per 984 00:53:23,500 --> 00:53:25,500 nick operating system, right? 985 00:53:25,500 --> 00:53:25,540 Yeah. 986 00:53:25,540 --> 00:53:28,940 So, so that's actually, it's actually funny that you mentioned this. 987 00:53:28,980 --> 00:53:33,380 Um, one of my research topics at work has been the Doka platform. 988 00:53:33,780 --> 00:53:38,780 Um, basically NVIDIA built a bunch of CRDs, uh, custom resource definitions on 989 00:53:38,780 --> 00:53:42,940 top of the Kubernetes API to like declaratively manage the DPU networking 990 00:53:42,940 --> 00:53:44,260 cards that we're talking about earlier. 991 00:53:44,700 --> 00:53:46,540 And it's so weird. 992 00:53:46,540 --> 00:53:50,660 It's like, it's like, yeah, you could do this, but should you? 993 00:53:50,820 --> 00:53:51,100 Yeah. 994 00:53:51,220 --> 00:53:57,140 Um, and the answer for me was kind of like, no, this, like, and, and like the 995 00:53:57,140 --> 00:54:01,300 way that they want you to set it up and, and like the way that you want to maintain 996 00:54:01,300 --> 00:54:05,020 it, it's like, you're running three Kubernetes clusters at the same time 997 00:54:05,020 --> 00:54:06,980 there, which is insane. 998 00:54:07,140 --> 00:54:07,420 Yeah. 999 00:54:07,460 --> 00:54:07,820 Right. 1000 00:54:07,980 --> 00:54:12,820 And there are like projects like Kamadi, which is what I use to manage, um, those 1001 00:54:12,820 --> 00:54:17,020 three Kubernetes clusters, which is so cool because it lets you like, it lets you 1002 00:54:17,020 --> 00:54:21,340 declaratively create Kubernetes clusters inside of Kubernetes clusters, which 1003 00:54:21,340 --> 00:54:28,420 is, which is, you know, uh, batshit and saying, but it's like, you know, this 1004 00:54:28,420 --> 00:54:33,380 doesn't feel like what we should be doing with Kubernetes because it wasn't 1005 00:54:33,380 --> 00:54:39,300 really designed for this is it like CRDs were really meant to be kind of, if you 1006 00:54:39,300 --> 00:54:42,820 need something more than just a deployment or replica set, something 1007 00:54:42,820 --> 00:54:45,140 like that, it still should be at the application layer, right? 1008 00:54:45,140 --> 00:54:45,660 It still should be. 1009 00:54:45,660 --> 00:54:50,540 If I think of the like OSI model, right, that's still at the top of the, like 1010 00:54:50,540 --> 00:54:54,900 CRDs are not down here in like the physical sort of like TCP area. 1011 00:54:54,900 --> 00:54:57,460 This is like, no, there needs to be something else that's managing everything 1012 00:54:57,460 --> 00:54:57,940 below that. 1013 00:54:57,940 --> 00:55:01,900 And, and to some degree, that's just the operating system has traditionally done 1014 00:55:01,900 --> 00:55:02,060 that. 1015 00:55:02,060 --> 00:55:06,540 I don't know that general purpose operating systems are the right thing for that either. 1016 00:55:06,740 --> 00:55:06,940 Right. 1017 00:55:06,940 --> 00:55:09,540 That's where we just start getting into like, what is firmware? 1018 00:55:09,540 --> 00:55:12,020 It's like, well, it's like a little operating system that does one thing. 1019 00:55:12,260 --> 00:55:12,860 Basically. 1020 00:55:13,340 --> 00:55:13,700 Yeah. 1021 00:55:13,700 --> 00:55:17,140 I mean, and it's like, you could probably cobble together like all of the 1022 00:55:17,140 --> 00:55:22,100 individual parts and individual software to like make like a whole rack scale, you 1023 00:55:22,100 --> 00:55:24,020 know, computer software. 1024 00:55:24,020 --> 00:55:28,940 But it's just like, you know, I spent so much time trying to get even like 1025 00:55:28,940 --> 00:55:30,500 net box working correctly. 1026 00:55:30,500 --> 00:55:32,100 And it's just, it's a headache. 1027 00:55:32,100 --> 00:55:38,420 Like there, no one has really thought of it that way as like a rack scale computer. 1028 00:55:38,660 --> 00:55:42,300 I think the closest people would probably be Oxide, honestly. 1029 00:55:43,580 --> 00:55:47,500 Because, you know, if there was anyone that was going to figure it out, 1030 00:55:47,500 --> 00:55:49,060 I would probably assume it's them. 1031 00:55:49,780 --> 00:55:54,300 Just because they're, you know, they're building out all of the key building 1032 00:55:54,300 --> 00:55:56,980 blocks and the way that you would expect, right? 1033 00:55:57,380 --> 00:55:59,500 Well, they're doing, I think they're doing it in such a different way. 1034 00:55:59,700 --> 00:55:59,900 All right. 1035 00:55:59,900 --> 00:56:03,220 Like they're, they're just saying, like, you know what, we don't need a BIOS. 1036 00:56:03,580 --> 00:56:04,820 Like it just doesn't exist. 1037 00:56:04,820 --> 00:56:06,980 Like we don't, we're not building that kind of machine. 1038 00:56:07,220 --> 00:56:11,140 We know everything about this hardware and how to initialize it when we plug it in. 1039 00:56:11,140 --> 00:56:14,340 So we don't need that general purpose BIOS UEFI stuff. 1040 00:56:14,340 --> 00:56:19,300 And it's just more complicated than it's worth to have that interface in the system. 1041 00:56:19,300 --> 00:56:20,260 So we'll get rid of it. 1042 00:56:20,380 --> 00:56:20,660 Yeah. 1043 00:56:20,700 --> 00:56:25,540 And in the more of, the more of those layers we can remove the better in a lot 1044 00:56:25,540 --> 00:56:30,860 of these cases where it's just like, actually, what is, what is valuable in this stack? 1045 00:56:31,940 --> 00:56:32,140 Yeah. 1046 00:56:32,140 --> 00:56:34,500 Well, I mean, disaggregation is cool, right? 1047 00:56:34,500 --> 00:56:36,020 It is, it is so cool. 1048 00:56:36,020 --> 00:56:37,940 It is so, so wild. 1049 00:56:37,940 --> 00:56:44,140 But at the same time, like there are limitations due to the speed of light. 1050 00:56:44,140 --> 00:56:45,860 A lot of, a lot of times, right? 1051 00:56:46,060 --> 00:56:50,580 There is an unavoidable amount of latency involved in like converting optical signals 1052 00:56:50,580 --> 00:56:54,260 to electricity, like two copper signals, right? 1053 00:56:55,660 --> 00:57:02,780 And it's like, it's just, it's going to be really fascinating to see, like, to see 1054 00:57:02,780 --> 00:57:07,500 all of this operationalize and see like maybe like 10, 15 years from now, like 1055 00:57:07,740 --> 00:57:10,020 were we correct about disaggregation, right? 1056 00:57:10,020 --> 00:57:11,100 Were we not correct? 1057 00:57:11,740 --> 00:57:16,020 And like the answer is kind of like, you have to try and find out, right? 1058 00:57:16,260 --> 00:57:16,540 Yeah. 1059 00:57:18,180 --> 00:57:25,180 But like from an SRE standpoint, disaggregation, it makes everything a black box. 1060 00:57:25,700 --> 00:57:27,740 Means you can't trust your storage devices. 1061 00:57:27,740 --> 00:57:29,460 It can't, you can't trust your network. 1062 00:57:29,900 --> 00:57:33,340 And one of the first things you learn about distributed systems is that the 1063 00:57:33,340 --> 00:57:35,980 network is not reliable, right? 1064 00:57:36,460 --> 00:57:40,820 Like there is a reason why TCP is basically the de facto norm. 1065 00:57:40,820 --> 00:57:44,740 And it is because packets are dropped pretty frequently, right? 1066 00:57:45,180 --> 00:57:49,660 And I mean, if like it has been a long, long time since I haven't trusted 1067 00:57:49,660 --> 00:57:54,940 a front side bus between the CPU and a PCI, like that has always worked for me. 1068 00:57:54,940 --> 00:57:56,620 Or most, most of my life that has worked. 1069 00:57:56,620 --> 00:57:59,780 And I've never had to think about maybe I need to debug this layer. 1070 00:57:59,980 --> 00:58:02,460 But when you go to a distributed system or disaggregation, you're like, 1071 00:58:02,460 --> 00:58:05,580 everything is network and now I don't trust any of those buses. 1072 00:58:05,700 --> 00:58:06,180 Yes. 1073 00:58:06,740 --> 00:58:07,180 Yeah. 1074 00:58:07,220 --> 00:58:13,260 It is, it is going to be so, so wild to see like all of this productionized. 1075 00:58:13,260 --> 00:58:18,300 Like, like, what does the SRE do 10, 15 years from now? 1076 00:58:18,300 --> 00:58:22,140 Like, are we teaching people how to like attach an oscilloscope and like 1077 00:58:22,140 --> 00:58:25,180 see the actual signal coming over the wire, like trace it? 1078 00:58:25,180 --> 00:58:31,060 Like, it kind of seems like where we're headed because, you know, one of the 1079 00:58:31,060 --> 00:58:36,700 things that I, I really pride myself on is like reverse engineering and 1080 00:58:36,700 --> 00:58:40,780 knowing every single stack and every single layer of like involved in like 1081 00:58:40,780 --> 00:58:43,180 an OS call or a sys call or something like that. 1082 00:58:43,420 --> 00:58:46,100 But then what's the like through disaggregation in the mix now? 1083 00:58:46,100 --> 00:58:48,660 Like, you can't even trust like an M map, right? 1084 00:58:48,980 --> 00:58:50,380 And M maps can fail. 1085 00:58:50,380 --> 00:58:51,380 And then what does that do? 1086 00:58:51,380 --> 00:58:53,020 Like software, right? 1087 00:58:53,500 --> 00:58:57,700 Like, it's, it's funny because like all of our hardware is advancing at such 1088 00:58:57,700 --> 00:59:01,380 a fast pace, but it doesn't feel like the software is keeping up. 1089 00:59:02,260 --> 00:59:02,540 Hmm. 1090 00:59:03,020 --> 00:59:10,140 And that, that has kind of been the, the like killer, killer, sorry. 1091 00:59:10,140 --> 00:59:11,100 What's like the term here? 1092 00:59:11,100 --> 00:59:13,660 It's like nailing the coffin for a lot of things. 1093 00:59:13,660 --> 00:59:17,140 Like you think of itanium, you think of Zion, fuck me. 1094 00:59:17,940 --> 00:59:23,380 Zion fi, I think, like Intel's experiments to try and make really cool hardware. 1095 00:59:23,700 --> 00:59:24,100 Right. 1096 00:59:24,340 --> 00:59:29,660 The software just never capped up for either to, for either itanium or Zion 1097 00:59:29,660 --> 00:59:32,620 fi, which led to both of them failing. 1098 00:59:32,980 --> 00:59:33,420 Right. 1099 00:59:34,100 --> 00:59:38,980 And so, you know, that's, that's kind of going to be really interesting to see 1100 00:59:39,020 --> 00:59:43,820 with Memvirgin liquid, because they're both betting on Kubernetes as being kind 1101 00:59:43,820 --> 00:59:45,340 of like the management plan for these things. 1102 00:59:45,660 --> 00:59:46,140 Interesting. 1103 00:59:47,740 --> 00:59:53,860 So like in some ways, like Kubernetes is, is not the right fit for any of 1104 00:59:53,860 --> 00:59:58,580 these, but it is what the enterprise is kind of standardizing on, because the 1105 00:59:58,580 --> 01:00:02,140 alternative is like proprietary vendor locked software. 1106 01:00:02,380 --> 01:00:02,620 Yeah. 1107 01:00:02,860 --> 01:00:08,140 Um, you know, like ESXi, that everyone was running on ESXi for a while until 1108 01:00:08,140 --> 01:00:12,020 Broadcom bought VMware and then was like, Hey, we're going to screw up all of 1109 01:00:12,020 --> 01:00:13,260 your licensing, by the way. 1110 01:00:16,620 --> 01:00:20,820 Uh, you mentioned provisioning there for a minute and I'm kind of curious, it's 1111 01:00:20,820 --> 01:00:24,380 always been like this passion of mine throughout my entire career. 1112 01:00:24,380 --> 01:00:28,020 I was just like, I, I love the, I love pixie booting things. 1113 01:00:28,100 --> 01:00:29,900 I love provisioning automated. 1114 01:00:29,900 --> 01:00:32,700 Like I can install a hard, like an operating system. 1115 01:00:33,620 --> 01:00:35,940 What's the, do you know what state of art, like state of the art, like for 1116 01:00:35,940 --> 01:00:37,100 provisioning systems these days? 1117 01:00:37,100 --> 01:00:41,420 Like when I did it, like I literally was like cobbler back in the day and a 1118 01:00:41,420 --> 01:00:45,900 satellite foreman, um, doing that sort of like red hat stack for quite a while. 1119 01:00:45,900 --> 01:00:49,460 And then I was like switching over to like when I did a core OS, like their whole 1120 01:00:49,500 --> 01:00:52,340 like install script was like a, it was just a bash script that just like 1121 01:00:52,340 --> 01:00:54,420 D deed an image to this kind of like, that's fascinating. 1122 01:00:54,420 --> 01:00:55,020 That's all you need. 1123 01:00:55,020 --> 01:00:55,820 I was like, yeah, that's it. 1124 01:00:55,820 --> 01:00:58,900 And, and I've kind of, I've, I've moved away from a lot of stuff. 1125 01:00:58,900 --> 01:01:01,020 I'm kind of curious what is, oh, net box. 1126 01:01:01,020 --> 01:01:01,740 That's what you mentioned. 1127 01:01:01,940 --> 01:01:04,500 Well, so net box is just like the network source of truth. 1128 01:01:04,500 --> 01:01:04,820 It is. 1129 01:01:04,820 --> 01:01:05,100 Okay. 1130 01:01:05,100 --> 01:01:07,940 That's just the, not IMDB, the CMDB, right? 1131 01:01:07,940 --> 01:01:11,340 So it's the database of like what's, what's there in the network. 1132 01:01:11,700 --> 01:01:14,180 So Nix is really interesting. 1133 01:01:14,580 --> 01:01:18,580 Nix is really interesting, but so far I haven't found a single person who's 1134 01:01:18,580 --> 01:01:24,420 gotten Nix to work as like that sort of like provisioning, you know, 1135 01:01:24,460 --> 01:01:26,340 Nix as the OS or Nix. 1136 01:01:26,420 --> 01:01:26,660 Yeah. 1137 01:01:26,660 --> 01:01:27,700 Nix OS, right? 1138 01:01:28,100 --> 01:01:31,740 I haven't seen a single person who has managed to make Nix work. 1139 01:01:32,180 --> 01:01:37,140 I think the state of the art there is still like puppet chef, Ansible. 1140 01:01:37,420 --> 01:01:42,420 Um, but I genuinely don't think that we've evolved beyond puppet or chef. 1141 01:01:42,420 --> 01:01:46,300 I mean, if you go to like the big hyperscalers, yeah, they, they totally 1142 01:01:46,300 --> 01:01:49,380 have a replacement for puppet and chef and everything like that. 1143 01:01:49,380 --> 01:01:52,300 And they definitely have like integrated solutions for all of these things. 1144 01:01:52,300 --> 01:01:59,940 But, um, yeah, you know, I think the people that I know who have made 1145 01:01:59,940 --> 01:02:05,260 advancements here are like packet before they got bought out by Equinex and 1146 01:02:05,260 --> 01:02:06,420 turned it to Equinex metal. 1147 01:02:06,460 --> 01:02:07,540 And now it gets shut down. 1148 01:02:07,540 --> 01:02:07,780 Yeah. 1149 01:02:09,700 --> 01:02:14,540 You know, like there hasn't been a lot of like, and this is something that 1150 01:02:14,540 --> 01:02:17,100 OpenStack was actually trying to do as well. 1151 01:02:17,220 --> 01:02:17,420 Right. 1152 01:02:17,420 --> 01:02:17,900 Ironic. 1153 01:02:18,260 --> 01:02:18,660 Yeah. 1154 01:02:18,660 --> 01:02:18,980 Yeah. 1155 01:02:18,980 --> 01:02:23,780 It's, it's like, it was, it was just a couple of years too ahead of its time. 1156 01:02:24,180 --> 01:02:29,020 And maybe now if someone tried to do OpenStack again, we'd be like, okay, you 1157 01:02:29,020 --> 01:02:32,620 know, and that's going to be really interesting to see also from like, 1158 01:02:33,180 --> 01:02:33,900 um, what was it? 1159 01:02:33,900 --> 01:02:34,780 Cidero too. 1160 01:02:35,300 --> 01:02:35,700 Right. 1161 01:02:35,740 --> 01:02:38,260 Like how is that Talos Linux stuff going to work? 1162 01:02:38,540 --> 01:02:38,780 Yeah. 1163 01:02:38,780 --> 01:02:44,340 And, and it's been interesting on like both sides of how things get provisioned 1164 01:02:44,340 --> 01:02:47,980 because I look at people that went to the cloud, just ignored the whole 1165 01:02:47,980 --> 01:02:52,100 how to provision things and it's just like boot the generic Ubuntu and, and 1166 01:02:52,100 --> 01:02:53,580 can, and cloud init the whole thing. 1167 01:02:53,860 --> 01:02:54,220 Right. 1168 01:02:54,220 --> 01:02:56,900 And like cloud init is the config management system, which is just such 1169 01:02:56,900 --> 01:03:01,980 a, a slow, difficult path to like actually make something work. 1170 01:03:01,980 --> 01:03:04,860 And then coming from like data centers where they're like, oh, we're running 1171 01:03:05,100 --> 01:03:06,740 puppet agent or chef or something. 1172 01:03:06,740 --> 01:03:11,380 And that's, uh, it is, is a, is a lot more powerful than cloud in its, but 1173 01:03:11,380 --> 01:03:15,060 also can, can quickly become more complex and harder to maintain. 1174 01:03:15,060 --> 01:03:15,540 Yes. 1175 01:03:15,740 --> 01:03:20,020 Um, and I, I keep telling friends at work, like people at work. 1176 01:03:20,020 --> 01:03:23,180 I'm like, I'm so spoiled by Talos now because like, I don't, I don't worry 1177 01:03:23,180 --> 01:03:26,940 about either of those things where it's like, I call an API and my OS gets 1178 01:03:26,940 --> 01:03:28,980 installed and that's weird. 1179 01:03:29,140 --> 01:03:32,780 I had to do a, I told you before we started recording, I'm doing performance 1180 01:03:32,780 --> 01:03:35,860 testing on a bunch of different platforms and operating systems. 1181 01:03:36,100 --> 01:03:39,620 And like every, I have a KVM and I'm mounting like two gig of Ubuntu 1182 01:03:39,660 --> 01:03:41,340 ISOs to it, to like install Ubuntu. 1183 01:03:41,340 --> 01:03:42,820 I'm like, this is so painful. 1184 01:03:43,100 --> 01:03:47,100 Like I can't believe it's no wonder that people wanted to get away from this. 1185 01:03:47,300 --> 01:03:51,340 Uh, because my, you know, my Talos Linux is like 80 megs in it. 1186 01:03:51,500 --> 01:03:53,100 It just like one API was okay. 1187 01:03:53,100 --> 01:03:54,140 It's installed, I'm ready to go. 1188 01:03:54,140 --> 01:03:55,740 And I'm like, oh, I'm still, it's still booting. 1189 01:03:55,860 --> 01:03:57,460 Like Ubuntu is still kind of going through that. 1190 01:03:57,460 --> 01:04:03,780 And I can't believe that nothing has changed that ecosystem in drastic ways. 1191 01:04:03,780 --> 01:04:09,540 I'm, I'm using things like, um, Silverblue, Bluefin for my desktop, which is, you 1192 01:04:09,540 --> 01:04:13,940 know, imaged base sort of installs, it's still large because it's still a 1193 01:04:13,940 --> 01:04:18,500 general purpose sort of Fedora installation, but the maintenance side of it has improved. 1194 01:04:18,500 --> 01:04:21,260 And so there's all these like weird areas where we're, we're kind of making 1195 01:04:21,260 --> 01:04:26,660 changes in some places and not a lot of people are rethinking the possibilities. 1196 01:04:26,780 --> 01:04:27,220 Yeah. 1197 01:04:27,300 --> 01:04:31,420 No, so many, so many of these like second order cloud providers, they're 1198 01:04:31,420 --> 01:04:32,940 still using cloud in it, right? 1199 01:04:33,540 --> 01:04:38,580 It is all, it is basically Libvert with cloud in it and they are using like the CD 1200 01:04:38,580 --> 01:04:43,220 data store that is it that, you know, and like you look at the cloud providers, 1201 01:04:43,220 --> 01:04:45,620 they're doing things like MMDS, right? 1202 01:04:45,660 --> 01:04:52,980 With the EC2 instances where you can call, I think 196.254.196.254 and it's like 1203 01:04:52,980 --> 01:04:58,940 this API where you get all sorts of metadata about your, um, like things 1204 01:04:58,940 --> 01:05:05,580 like that are probably the future and where we need to move towards. 1205 01:05:05,620 --> 01:05:10,500 And, you know, I just think like no one has really tackled that space. 1206 01:05:10,500 --> 01:05:15,620 And it's understandable why, but it's like, at the same time, like, you know, 1207 01:05:15,820 --> 01:05:19,060 you have a feeling like someone could do this better with like Nix or like 1208 01:05:19,060 --> 01:05:20,460 Talos or something like that. 1209 01:05:20,460 --> 01:05:23,820 It's just, you know, I just, I just don't have the time to do it. 1210 01:05:25,980 --> 01:05:28,620 Well, thank you so much for taking the time for this podcast. 1211 01:05:28,620 --> 01:05:29,500 This has been a lot of fun. 1212 01:05:30,340 --> 01:05:33,700 Uh, uh, if, if, if no one has seen it, I'm going to put it in show notes. 1213 01:05:33,740 --> 01:05:37,140 Uh, le.fm at your website is something that everyone needs to go to. 1214 01:05:37,140 --> 01:05:38,500 At least once experience. 1215 01:05:40,180 --> 01:05:41,620 I'm not going to throw out any spoilers yet. 1216 01:05:41,620 --> 01:05:42,580 Just, just click the link. 1217 01:05:44,340 --> 01:05:48,260 Uh, I hope that's not mobile friendly because it's not, it's not, right. 1218 01:05:48,340 --> 01:05:48,900 This is great. 1219 01:05:48,900 --> 01:05:49,100 Yeah. 1220 01:05:49,100 --> 01:05:50,740 Not optimized for mobile whatsoever. 1221 01:05:51,300 --> 01:05:53,380 If there's not horizontal scrolling, I don't want it. 1222 01:05:56,020 --> 01:05:58,660 So I know you're, uh, I don't know if people can reach out to you 1223 01:05:58,660 --> 01:06:02,060 somewhere, or at least, uh, where you could see where people are talking about, 1224 01:06:02,580 --> 01:06:06,660 uh, you know, I lurk, I lurk, I will, I will, I will see, I will see the 1225 01:06:06,660 --> 01:06:10,860 discussions, maybe, maybe I'll pop in on whatever new blue sky account I have at 1226 01:06:10,860 --> 01:06:11,420 the time. 1227 01:06:13,300 --> 01:06:13,940 Sounds great. 1228 01:06:13,940 --> 01:06:17,500 And I look forward to learning more about, uh, DPUs and, and going down, uh, 1229 01:06:17,540 --> 01:06:20,980 rabbit holes of, of YouTube videos of, I didn't know that was possible from 1230 01:06:20,980 --> 01:06:21,620 you in the future. 1231 01:06:23,500 --> 01:06:24,180 Have a good one. 1232 01:06:24,300 --> 01:06:24,460 Yeah. 1233 01:06:24,500 --> 01:06:24,900 Thanks. 1234 01:06:25,380 --> 01:06:27,780 And everyone, thank you for listening and we will talk to you again soon. 1235 01:06:28,660 --> 01:06:40,060 Thank you for listening to this episode of fork around and find out. 1236 01:06:40,380 --> 01:06:43,900 If you like this show, please consider sharing it with a friend, a coworker, a 1237 01:06:43,900 --> 01:06:45,740 family member, or even an enemy. 1238 01:06:45,820 --> 01:06:49,700 However we get the word out about this show helps it to become sustainable for 1239 01:06:49,700 --> 01:06:50,420 the longterm. 1240 01:06:50,620 --> 01:06:56,740 If you want to sponsor this show, please go to FAFO.fm slash sponsor and reach 1241 01:06:56,740 --> 01:06:59,780 out to us there about what you're interested in sponsoring and how we can 1242 01:06:59,780 --> 01:07:00,020 help. 1243 01:07:01,340 --> 01:07:04,580 We hope your system stay available and your pagers stay quiet. 1244 01:07:05,060 --> 01:07:06,220 We'll see you again next time.