1 00:00:00,120 --> 00:00:04,470 Saying this exact, uh, thing to many, many people in Amazon. 2 00:00:04,470 --> 00:00:07,710 Over the last while, um, my excellent account 3 00:00:07,710 --> 00:00:11,220 manager has been setting me up with various leaders. 4 00:00:11,640 --> 00:00:14,490 Uh, they've been asking for documents, they've been asking for examples. 5 00:00:14,610 --> 00:00:17,400 You know, they are hungry for this stuff, so I don't, don't doubt 6 00:00:17,460 --> 00:00:21,300 that there's no desire to be the leaders or to, to really satisfy 7 00:00:21,300 --> 00:00:23,370 their customers, but, you know, it's execution we care about. 8 00:00:23,580 --> 00:00:24,990 And when it comes down to it. 9 00:00:25,830 --> 00:00:27,330 We need excellent databases. 10 00:00:27,360 --> 00:00:29,430 We need the best databases to be able to 11 00:00:29,430 --> 00:00:31,320 ship world-class product to our customers. 12 00:00:36,360 --> 00:00:38,070 Welcome to Screaming in the Cloud. 13 00:00:38,160 --> 00:00:40,800 I'm Corey Quinn, and I'm here to correct an oversight. 14 00:00:40,835 --> 00:00:44,045 Because I have known Brian Scanlon for many years, and 15 00:00:44,045 --> 00:00:46,865 somehow he has slipped through the cracks and not been 16 00:00:46,865 --> 00:00:50,705 on this show previously, so let's go ahead and fix that. 17 00:00:50,855 --> 00:00:54,215 Brian Scanlon is a senior principal engineer at Intercom, 18 00:00:54,425 --> 00:00:57,695 where he has been for damn near 11 years at this point. 19 00:00:57,725 --> 00:00:59,225 Brian, welcome to the show. 20 00:00:59,255 --> 00:00:59,925 Thanks so much, Corey. 21 00:00:59,945 --> 00:01:01,655 It's great to finally be here. 22 00:01:02,065 --> 00:01:04,495 I know it's We save the best toward the end. 23 00:01:04,525 --> 00:01:05,245 Yeah, exactly. 24 00:01:05,245 --> 00:01:07,015 Except we're not ending the show anytime soon. 25 00:01:07,015 --> 00:01:08,575 Much to various people's chagrin. 26 00:01:08,605 --> 00:01:11,725 'cause most people want me to shut the hell up a lot more than I do, 27 00:01:12,055 --> 00:01:12,445 not me. 28 00:01:12,445 --> 00:01:13,885 I hope we keep going for a long time. 29 00:01:14,640 --> 00:01:17,070 Crying Out Cloud is one of the few cloud security 30 00:01:17,070 --> 00:01:19,500 podcasts that's actually fun to listen to. 31 00:01:19,590 --> 00:01:22,500 Smart Conversations, great guests, and Zero Fluff. 32 00:01:22,590 --> 00:01:25,590 If you haven't heard of it, it's a cloud and AI security 33 00:01:25,590 --> 00:01:29,940 podcast from Wiz Run by Clouds Sec. Pros for Clouds SEC Pros. 34 00:01:30,120 --> 00:01:32,280 I was actually one of the first guests on the 35 00:01:32,280 --> 00:01:34,230 show and it's been amazing to watch it grow. 36 00:01:34,290 --> 00:01:39,270 Make sure to check them out at wiz.io/crying-out-cloud. 37 00:01:40,095 --> 00:01:41,265 So Intercom. 38 00:01:41,384 --> 00:01:43,455 I have complicated feelings about the company. 39 00:01:43,455 --> 00:01:46,005 Originally I hated the thing 'cause I'm on a website trying to 40 00:01:46,005 --> 00:01:49,185 get something done and it pops up like freaking digital clippy. 41 00:01:49,185 --> 00:01:51,104 Hey, do you want to talk to a person? 42 00:01:51,375 --> 00:01:51,975 God no. 43 00:01:51,975 --> 00:01:52,755 I'm a millennial. 44 00:01:52,755 --> 00:01:53,025 Elder. 45 00:01:53,025 --> 00:01:55,575 Millennial, but still I don't want to talk to people. 46 00:01:55,600 --> 00:01:59,055 I. But they've come back around because it turns out the issue is not intercom. 47 00:01:59,055 --> 00:02:01,185 It was bad implementations thereof. 48 00:02:01,185 --> 00:02:03,915 Maybe you don't need to pop up, talk to a human on your 49 00:02:03,915 --> 00:02:06,675 landing page at the front of it, but when you're dealing with 50 00:02:06,675 --> 00:02:09,375 a support issue and you want to tag someone in and suddenly 51 00:02:09,375 --> 00:02:12,735 you're talking to a human right there, it's transformative. 52 00:02:12,915 --> 00:02:15,920 So yeah, it turns out that anything can be dumb if you hold it wrong. 53 00:02:16,840 --> 00:02:21,520 Yeah, I used to introduce Intercom to PE people to say, uh, yeah, we're one 54 00:02:21,520 --> 00:02:25,030 of those chatbots that pop up in the right hand bottom corner of your website. 55 00:02:25,030 --> 00:02:26,485 Except we're the good one, I think. 56 00:02:27,329 --> 00:02:29,760 There are a lot of bad implementations, and there you, you 57 00:02:29,760 --> 00:02:33,090 know, people use it for outreach and uh, for marketing and 58 00:02:33,090 --> 00:02:35,940 sales purposes, and, you know, they want messages in your face. 59 00:02:35,940 --> 00:02:39,750 But, uh, I think when it comes to basic customer support and 60 00:02:39,990 --> 00:02:43,140 actually putting humans in touch with humans and increasingly 61 00:02:43,140 --> 00:02:45,810 humans in touch with AI bots, that will answer your questions. 62 00:02:45,959 --> 00:02:47,519 I think intercom's, uh, definitely got. 63 00:02:47,855 --> 00:02:50,615 Uh, really good properties for, uh, in the marketplace. 64 00:02:50,885 --> 00:02:53,825 It's especially strange in this era that we are in 65 00:02:53,825 --> 00:02:58,145 where everyone is building AI chatbots and, okay. 66 00:02:58,145 --> 00:03:01,355 I have a whole laundry list of angry opinions on it, but where I, where I 67 00:03:01,355 --> 00:03:04,595 really get annoyed is when they don't admit that they're a chatbot upfront. 68 00:03:04,715 --> 00:03:07,595 And it irks me because by the time I finally break 69 00:03:07,595 --> 00:03:09,845 down and admit I have to talk to a human being. 70 00:03:10,355 --> 00:03:11,975 I have exhausted the documentation. 71 00:03:11,975 --> 00:03:14,135 It is not going to be something simple of, 72 00:03:14,135 --> 00:03:16,894 have you tried jiggling the handle instead? 73 00:03:16,894 --> 00:03:17,584 It's okay. 74 00:03:17,584 --> 00:03:19,475 There is now a weird corner case, which 75 00:03:19,475 --> 00:03:21,635 I'm very good at, blindly stumbling into. 76 00:03:21,964 --> 00:03:25,834 Uh, I need someone to go ahead and fix a thing on the backend or let 77 00:03:25,834 --> 00:03:29,405 me know that, that I wanna simply not possible with the platform or. 78 00:03:29,700 --> 00:03:32,700 Admit that your documentation is rubbish, and that's something that AI 79 00:03:32,700 --> 00:03:37,649 historically can't do, but forcing me to go through that filter to hit 80 00:03:37,649 --> 00:03:41,880 some arbitrary target of a fewest number of customer contacts, fastest 81 00:03:41,880 --> 00:03:46,380 resolution possible, and never let them talk to a human, has been maddening. 82 00:03:46,925 --> 00:03:50,825 Where are you right now on that whole gen AI spectrum from bot to human? 83 00:03:51,395 --> 00:03:57,155 So we are very much all in on, uh, gen ai, gen AI for, uh, chat bots. 84 00:03:57,395 --> 00:04:01,985 Um, so much so that, you know, in October, November, 2022 when 85 00:04:01,985 --> 00:04:05,735 chat GBT came out, uh, we reorientated the entire company. 86 00:04:05,735 --> 00:04:07,295 We saw that it was the future. 87 00:04:07,505 --> 00:04:09,155 Uh, we had actually been building ml. 88 00:04:09,235 --> 00:04:12,325 Chatbots, uh, and had them in the marketplace for a good few 89 00:04:12,325 --> 00:04:17,095 years prior to the current explosion of, uh, ML chatbots. 90 00:04:17,425 --> 00:04:22,225 Um, but the lifts or the improvement that we saw with GPT-3 0.5, and 91 00:04:22,225 --> 00:04:25,885 that came out, that was a okay, we need to change the entire company. 92 00:04:26,185 --> 00:04:29,545 Um, and we've changed the entire company and moves strongly 93 00:04:29,545 --> 00:04:32,965 towards what we think is like the best AI chat bot in the market. 94 00:04:33,205 --> 00:04:36,115 Uh, it needs to work well with humans and 95 00:04:36,115 --> 00:04:38,425 peop there's a transition as well also. 96 00:04:38,655 --> 00:04:40,515 There's just a lot of work that's to be done to get the 97 00:04:40,515 --> 00:04:43,484 quality really great so that people don't hate using it. 98 00:04:43,635 --> 00:04:46,875 One of the interesting things that we've seen is, uh, many of 99 00:04:46,875 --> 00:04:51,104 our customers who have good knowledge bases and who get high 100 00:04:51,104 --> 00:04:53,835 resolution rates, uh, from using the chatbots is that they'll 101 00:04:53,835 --> 00:04:56,744 find that customers actually just start asking more questions. 102 00:04:56,744 --> 00:05:00,674 Because if you can get really fast answers and you don't have to go around 103 00:05:00,674 --> 00:05:04,094 documentation sites and the chat bot's actually useful people get it. 104 00:05:04,195 --> 00:05:07,825 Addicted to this, they'll, they're gonna use it all the time, and weirdly 105 00:05:07,825 --> 00:05:11,665 enough, have increased the number of conversations coming in because people. 106 00:05:12,615 --> 00:05:15,615 In some cases, and we're seeing more, more and more of this 107 00:05:15,795 --> 00:05:18,375 where people will ask more questions because they're getting good 108 00:05:18,375 --> 00:05:21,105 fast answers, uh, which is a little bit counterintuitive, but 109 00:05:21,105 --> 00:05:24,135 I think we all always need to have ways to fall back to humans. 110 00:05:24,525 --> 00:05:26,715 Uh, of course, depends on the business and 111 00:05:26,715 --> 00:05:28,725 the volume and what makes sense for them. 112 00:05:28,725 --> 00:05:31,635 But you know, I don't think humans are going away and from what we've 113 00:05:31,635 --> 00:05:34,765 seen in the market as well, even for places that have adopted and are. 114 00:05:35,550 --> 00:05:38,790 Deflecting or answering large numbers of questions. 115 00:05:39,000 --> 00:05:42,570 We see that they're not reducing their support team sizes at all. 116 00:05:42,690 --> 00:05:45,990 They're putting their people on better, better questions, higher 117 00:05:45,990 --> 00:05:49,410 quality work, or just deeper work with customers, as well as like 118 00:05:49,410 --> 00:05:51,630 feeding the bot with better documentation, that kind of thing. 119 00:05:51,630 --> 00:05:53,430 So we're, we're seeing it as like actually 120 00:05:53,430 --> 00:05:55,470 a net positive into the customer experience. 121 00:05:55,620 --> 00:05:58,890 But, uh, there's definitely lots of bad implementations out there as well. 122 00:05:59,580 --> 00:06:01,440 It feels like AI assisted. 123 00:06:01,734 --> 00:06:05,995 Uh, support is on many cases better than pure AI support, and this is somewhat 124 00:06:05,995 --> 00:06:09,385 controversial among people who want to sell bots, but I find the chat bots are 125 00:06:09,385 --> 00:06:13,734 not necessarily a great interface just due to complete lack of discoverability. 126 00:06:13,734 --> 00:06:16,104 It's, it's the Alexa problem or the Siri problem, 127 00:06:16,104 --> 00:06:17,784 or whatever robot assistant you want to do. 128 00:06:17,965 --> 00:06:20,815 You ask it a. Question today, I was getting out of the 129 00:06:20,815 --> 00:06:23,335 shower and I asked one of the bots, or out of, I asked, 130 00:06:23,335 --> 00:06:25,615 tried it with both Siri and Alexa, neither one could do it. 131 00:06:25,825 --> 00:06:28,975 What is the partial pressor of oxygen at 10,000 feet above sea level? 132 00:06:29,035 --> 00:06:30,775 And they both drop the thing completely. 133 00:06:30,805 --> 00:06:32,695 'cause I have weird shower thoughts roll with it. 134 00:06:32,965 --> 00:06:36,235 But it's the, but the question then is, okay, it can't do it. 135 00:06:36,235 --> 00:06:36,665 Of course it couldn't. 136 00:06:37,345 --> 00:06:40,705 I will never ask that question again despite the fact that maybe 137 00:06:40,825 --> 00:06:43,465 it does know how to do that, it just didn't hear me properly. 138 00:06:43,465 --> 00:06:46,165 Or in two weeks it will be able to answer that question. 139 00:06:46,345 --> 00:06:49,195 But when you get, you ask a question and it can't answer it, you kind of 140 00:06:49,195 --> 00:06:52,765 feel dumb for having thought for a second that it might have been able to. 141 00:06:53,580 --> 00:06:57,630 The problem that the Alexa group has had forever, which is, is that humans use 142 00:06:57,630 --> 00:07:02,159 something like 98% of all the features they will ever use on their Alexa device. 143 00:07:02,159 --> 00:07:05,099 Within 90 minutes of setting it up the first time, play a 144 00:07:05,099 --> 00:07:09,390 song, set a timer, turn on lights, and that's about it For most 145 00:07:09,390 --> 00:07:12,539 people, it's just because it gains a bunch of these features. 146 00:07:12,705 --> 00:07:14,385 How do you tell people about that? 147 00:07:14,474 --> 00:07:16,875 Turns out that finishing every sentence with, by the 148 00:07:16,875 --> 00:07:20,565 way, pitching something unrelated just pisses people off. 149 00:07:20,745 --> 00:07:23,775 I think there will be a change as the products get better 150 00:07:23,775 --> 00:07:27,765 and as, uh, knowledge bases get better as customers. 151 00:07:27,974 --> 00:07:30,435 Our customers and, you know, customers of other chat bots 152 00:07:30,645 --> 00:07:33,735 know how to work with them better, that the expectations of. 153 00:07:34,320 --> 00:07:37,980 Of us as consumers of users of these things will improve or change 154 00:07:37,980 --> 00:07:41,010 over time and just not assume that these things are as brain 155 00:07:41,010 --> 00:07:43,500 dead as when we started interacting with them first time around. 156 00:07:44,130 --> 00:07:46,470 So I'm curious as far as what you've been 157 00:07:46,470 --> 00:07:49,410 up to from a technical perspective lately. 158 00:07:49,890 --> 00:07:50,310 We, uh. 159 00:07:50,385 --> 00:07:52,185 We've known each other for many years. 160 00:07:52,185 --> 00:07:53,505 You have been to my house for dinner. 161 00:07:53,505 --> 00:07:55,755 You're friends with my brother, which I think most 162 00:07:55,755 --> 00:07:57,945 listeners will be shocked to realize I have one of those. 163 00:07:58,094 --> 00:07:58,605 It's true. 164 00:07:58,664 --> 00:07:59,025 I do. 165 00:07:59,085 --> 00:07:59,865 He lives in Belgium. 166 00:07:59,925 --> 00:08:00,284 Great. 167 00:08:00,555 --> 00:08:04,545 The, but what I found that was so interesting and got me talking 168 00:08:04,545 --> 00:08:07,485 to you is I was recently talking to some of the fine folks over at 169 00:08:07,485 --> 00:08:10,784 Planet Scale, and they have talked about Intercom in general, and 170 00:08:10,784 --> 00:08:14,625 you in particular as being very pleased with their database offering. 171 00:08:14,925 --> 00:08:15,224 Now. 172 00:08:15,435 --> 00:08:18,735 I have talked to people in the past where I ask them questions about that. 173 00:08:18,765 --> 00:08:20,355 The response has been a wait. 174 00:08:20,355 --> 00:08:22,095 A company's saying, what about me now? 175 00:08:22,215 --> 00:08:23,325 So, right. 176 00:08:23,565 --> 00:08:25,275 You and I go back long enough that I can 177 00:08:25,275 --> 00:08:27,705 trust you not to bullshit me on these things. 178 00:08:27,705 --> 00:08:28,545 So, okay. 179 00:08:28,965 --> 00:08:31,095 Is it as good as they're telling me it is. 180 00:08:31,095 --> 00:08:36,195 And your response to still down to, if not better, you are a big champion. 181 00:08:36,195 --> 00:08:37,605 What planet scale is up to. 182 00:08:38,145 --> 00:08:38,925 Tell me more please. 183 00:08:38,925 --> 00:08:40,425 'cause I don't, I do a lot of things here. 184 00:08:40,455 --> 00:08:42,225 Scale is generally not one of them. 185 00:08:42,225 --> 00:08:44,055 For my own personal shit posting projects, 186 00:08:44,745 --> 00:08:45,165 yeah. 187 00:08:45,600 --> 00:08:46,680 Short answer is yes. 188 00:08:46,680 --> 00:08:49,230 Plan scale is great, but I'm gonna give you a long answer as well. 189 00:08:49,290 --> 00:08:49,920 So. 190 00:08:50,265 --> 00:08:54,975 Intercom is a Ruby on Rails monolith, and we really, really like this setup. 191 00:08:55,365 --> 00:08:59,955 Uh, we deploy our Ruby on Rails app onto EC2 computers, not 192 00:08:59,955 --> 00:09:04,545 stuffed away, and Docker containers, and using incomprehensible 193 00:09:04,575 --> 00:09:07,455 three letter acronyms for different parts of, uh, the, the setup. 194 00:09:07,725 --> 00:09:08,085 And we, 195 00:09:08,265 --> 00:09:09,825 oh, you can tear down and rebuild your stack on 196 00:09:09,825 --> 00:09:12,015 top of the latest, trendy thing every 18 months. 197 00:09:12,015 --> 00:09:12,075 Huh? 198 00:09:12,375 --> 00:09:12,645 Wow. 199 00:09:12,645 --> 00:09:14,535 Almost like you're not based in Silicon Valley. 200 00:09:15,070 --> 00:09:17,200 Yeah, maybe being in Dublin has its advantages. 201 00:09:17,200 --> 00:09:21,970 So, uh, yeah, we, we run really boring infrastructure and we 202 00:09:21,970 --> 00:09:25,000 have stuck with Ruby on Rails and it's largely been great for us. 203 00:09:25,060 --> 00:09:27,460 Uh, you know, you have to do a lot of work to scale it out to 204 00:09:27,460 --> 00:09:29,870 millions of lines of code, hundreds of developers working in it. 205 00:09:30,405 --> 00:09:34,635 But you at some stage you have to connect the thing to a database and 206 00:09:34,694 --> 00:09:39,345 database scaling has been a large part of the problems, uh, but also 207 00:09:39,345 --> 00:09:42,645 the joy of scaling Intercom in the 11 years that I've been there. 208 00:09:42,704 --> 00:09:46,214 When Intercom started off, we had a very simple, nice MySQL database. 209 00:09:46,214 --> 00:09:46,224 We. 210 00:09:47,040 --> 00:09:50,160 Then unfortunately we hit product market fit, having 211 00:09:50,160 --> 00:09:53,070 explosive growth, that was, uh, very challenging. 212 00:09:53,340 --> 00:09:55,590 Um, and even though we were based in RDS, you know, 213 00:09:55,590 --> 00:09:57,930 we were cloud first at the start from the very start. 214 00:09:58,230 --> 00:10:01,170 But, uh, native RDS couldn't deal or deal 215 00:10:01,170 --> 00:10:02,910 easily with, with what we were doing with it. 216 00:10:03,090 --> 00:10:06,510 When Aurora came along, we, uh, it honestly was a game 217 00:10:06,510 --> 00:10:09,330 changer for us and we jumped in very aggressively. 218 00:10:09,480 --> 00:10:10,980 We got to work with the Aurora team. 219 00:10:10,980 --> 00:10:13,350 We had some of the biggest tables on Aurora at the time. 220 00:10:13,620 --> 00:10:15,510 They would do all sorts of custom work for us. 221 00:10:15,820 --> 00:10:21,640 It was good fun, and just the Aurora architecture itself, the split between Com 222 00:10:21,790 --> 00:10:26,260 compute and storage, and how low latency the read replicas were and everything. 223 00:10:26,530 --> 00:10:28,330 This stuff just bought us years and years and 224 00:10:28,330 --> 00:10:31,030 years of scalability up to a certain point. 225 00:10:31,030 --> 00:10:34,420 And then we started having to do things like sharding some of our data. 226 00:10:34,760 --> 00:10:37,880 We had tables that were so large we couldn't mutate, we couldn't add new 227 00:10:37,880 --> 00:10:42,890 columns or, or, uh, do kind of other MI database migrations on this data 228 00:10:42,890 --> 00:10:46,430 because it was changing so fast, uh, without taking a lot of downtime. 229 00:10:46,580 --> 00:10:47,780 So we had to take action. 230 00:10:47,780 --> 00:10:49,970 And so we built our own kind of database charting 231 00:10:49,970 --> 00:10:52,760 system, again, built on top of Aurora, choosing. 232 00:10:52,835 --> 00:10:55,505 Technologies that we were very comfortable with. 233 00:10:55,865 --> 00:10:58,415 Um, and this bought us again, like years of scalability. 234 00:10:58,445 --> 00:11:02,015 It bought us the ability to be able to do database migrations on our 235 00:11:02,015 --> 00:11:06,905 tables, and life was good for another few years, but over time these 236 00:11:06,905 --> 00:11:11,045 like different sharding patterns and and way we were kind of using 237 00:11:11,045 --> 00:11:16,475 Aurora meant that we had 13 clusters connected to one application and. 238 00:11:17,075 --> 00:11:20,525 You get into these unfortunate situations where AWS will say, Hey, we 239 00:11:20,525 --> 00:11:23,825 gotta patch out and you really need to apply it to all of your clusters. 240 00:11:23,975 --> 00:11:24,485 Yes, we are. 241 00:11:24,485 --> 00:11:27,815 Downtime will be at some point during this broad window 242 00:11:27,815 --> 00:11:31,115 that is inconvenient for you and non-deterministic. 243 00:11:31,145 --> 00:11:35,375 So you've gotta be able to build a graceful degradation mode into 244 00:11:35,375 --> 00:11:38,495 your app from the, if you're using this technology, 'cause you've, 245 00:11:38,555 --> 00:11:41,855 one of the things you lose by a, with a managed database offering. 246 00:11:42,265 --> 00:11:46,675 Is the ability to be very granular around when and what gets applied where. 247 00:11:47,125 --> 00:11:47,455 Yeah. 248 00:11:47,455 --> 00:11:51,775 And even some of the upgrades that we would do, most of 249 00:11:51,775 --> 00:11:54,475 the time the cluster would come back in a minute or two. 250 00:11:54,505 --> 00:11:57,385 You know, not, not bad, but occasionally we'd have, you 251 00:11:57,385 --> 00:11:59,605 know, something would get stuck inside of a queue inside 252 00:11:59,605 --> 00:12:03,565 of Amazon or something, and it would be 20 minutes and. 253 00:12:04,064 --> 00:12:06,615 This kind of stuff starts to wear you down, especially 254 00:12:06,615 --> 00:12:08,655 when you've got so many clusters connected to your app 255 00:12:08,655 --> 00:12:10,875 and you are highly faced, highly fit, highly critical. 256 00:12:10,875 --> 00:12:14,505 As far as customer facing stuff, you're on the front of everyone's website. 257 00:12:14,564 --> 00:12:17,535 It, this is one of those areas where mistakes will show. 258 00:12:17,685 --> 00:12:18,375 Absolutely. 259 00:12:18,375 --> 00:12:19,360 And you know, we have. 260 00:12:20,280 --> 00:12:25,260 Loads of customers, uh, who have up to thousands of people whose 261 00:12:25,260 --> 00:12:29,910 job it is to be using intercom all day, replying to their customers. 262 00:12:30,209 --> 00:12:34,620 And it's, it's definitely no fun when they have their entire 263 00:12:34,620 --> 00:12:38,520 teams unavailable, not able to do their job because we. 264 00:12:39,145 --> 00:12:42,025 We're twiddling our tongues, waiting for an Aurora upgrade to complete. 265 00:12:42,295 --> 00:12:44,605 Well just do it outside of core business hours. 266 00:12:44,665 --> 00:12:45,715 I'm in San Francisco. 267 00:12:45,715 --> 00:12:46,795 Lots of people here use it. 268 00:12:46,795 --> 00:12:47,545 You're in Dublin. 269 00:12:47,545 --> 00:12:48,775 Lots of people there use it. 270 00:12:48,775 --> 00:12:51,564 You have customers in Australia, Japan, India, around the world. 271 00:12:51,564 --> 00:12:52,885 It is core business hours. 272 00:12:53,005 --> 00:12:54,954 Somewhere for someone. 273 00:12:54,954 --> 00:12:58,975 At any given point of the day, there is no, and now it is nighttime. 274 00:12:58,975 --> 00:13:00,270 The servers can take a nap now. 275 00:13:00,860 --> 00:13:01,460 Absolutely. 276 00:13:01,730 --> 00:13:04,520 This is not the DMV in the US or the social security 277 00:13:04,520 --> 00:13:07,250 website in the United States, which still blows my mind. 278 00:13:07,370 --> 00:13:10,790 It has a six hour maintenance window every night, like the last 279 00:13:10,790 --> 00:13:13,580 person outta the office, like turns off the mainframe or something. 280 00:13:13,670 --> 00:13:15,530 Probably some legacy batch job or whatnot. 281 00:13:15,530 --> 00:13:17,300 And there's good reason for it, but it feels 282 00:13:17,300 --> 00:13:19,160 like that the servers keep banker's hours. 283 00:13:19,550 --> 00:13:23,240 So we were aware of VI Tess and Planet Scale. 284 00:13:23,240 --> 00:13:29,870 So Vi Tess, uh, I'll like the brief introduction to VI Tess is it's a MySQL. 285 00:13:30,275 --> 00:13:36,155 Wrapper as such, or system that came out of YouTube about 12, 13 years ago. 286 00:13:36,155 --> 00:13:37,415 It's an open source project. 287 00:13:38,115 --> 00:13:41,715 It's got other large SaaS, B2B SaaS providers, 288 00:13:41,715 --> 00:13:44,085 like people like Slack, HubSpot, and 289 00:13:44,145 --> 00:13:46,635 the Slack folks have been a huge advocate of this. 290 00:13:46,660 --> 00:13:49,845 It it makes sense since they've, they given talks about this where 291 00:13:49,845 --> 00:13:53,865 effectively all, all of Slack is basically a giant MySQL database. 292 00:13:53,895 --> 00:13:57,075 Sharded heavily obviously, but yeah, it is. 293 00:13:57,075 --> 00:14:00,915 Every message is a line in a database, so yeah. 294 00:14:00,960 --> 00:14:04,800 Having the database work and not take a nap for a 20 minute upgrade 295 00:14:04,800 --> 00:14:08,700 at random times is kind of high on their list of mu it must do this. 296 00:14:08,760 --> 00:14:09,030 Yeah. 297 00:14:09,030 --> 00:14:11,010 And it's, you know, it's not just downtime, it's. 298 00:14:11,505 --> 00:14:14,055 We, we need to be able to do things like 299 00:14:14,355 --> 00:14:16,335 have we, we have to share a lot of data. 300 00:14:16,335 --> 00:14:18,375 And our, our customer's data is extremely charitable. 301 00:14:18,375 --> 00:14:20,535 We have, we're a multi-tenant application. 302 00:14:20,535 --> 00:14:23,895 We have lots and lots of our customer's data and they don't need to 303 00:14:24,105 --> 00:14:27,735 join across their, like, different workspaces or different customers. 304 00:14:27,944 --> 00:14:29,805 So we have very, very charitable data. 305 00:14:30,105 --> 00:14:31,275 Um, the other thing we do is. 306 00:14:31,590 --> 00:14:34,710 We struggle with connection pool management. 307 00:14:34,950 --> 00:14:38,430 We have hundreds of thousands of ruby on rails processes that all need 308 00:14:38,430 --> 00:14:41,820 to connect to databases that can only take 16,000 connections at most. 309 00:14:42,240 --> 00:14:47,850 And, uh, so we have to run a, a layer of proxy sql, uh, connection 310 00:14:47,850 --> 00:14:52,020 proxies in between our application and our Aurora database. 311 00:14:52,200 --> 00:14:55,230 Sometimes the proxy SQL layer goes wrong, and it's just 312 00:14:55,230 --> 00:14:57,450 another layer of complication that we don't want to think of. 313 00:14:57,600 --> 00:15:00,600 So we're aware of a test and it became increasingly. 314 00:15:01,230 --> 00:15:04,830 Clear that planet scale were the way to get Vitas and we have 315 00:15:04,830 --> 00:15:08,310 no interest in as well, or like we'd rather avoid running our 316 00:15:08,310 --> 00:15:10,860 own infrastructure or running our own high level services. 317 00:15:11,100 --> 00:15:14,580 If we can pay somebody to run a database for us, we will absolutely do that. 318 00:15:14,700 --> 00:15:16,800 Now I, I'm going to stop you there because 319 00:15:16,800 --> 00:15:19,440 historically that has been the entire. 320 00:15:19,795 --> 00:15:22,885 Rallying cry of cloud where Oh great. 321 00:15:22,885 --> 00:15:24,625 You don't wanna run servers yourself. 322 00:15:24,625 --> 00:15:28,195 To a point where some people have now gone so far around the bend that we view 323 00:15:28,315 --> 00:15:32,275 running servers in data centers as being a skillset set the ancients possessed, 324 00:15:32,275 --> 00:15:36,685 but has since been lost to modern humanity outside of three hyperscalers. 325 00:15:37,405 --> 00:15:40,975 No, but those folks have been with a default 326 00:15:40,975 --> 00:15:43,855 go-to for a lot of things for a lot of years. 327 00:15:44,410 --> 00:15:47,230 You are looking, but your answer was not to go 328 00:15:47,230 --> 00:15:48,910 and yell at the Aurora team to make it better. 329 00:15:48,910 --> 00:15:50,230 It was to look somewhere else. 330 00:15:50,320 --> 00:15:52,900 You know, we did talk to the Aurora team about the problems 331 00:15:52,900 --> 00:15:56,710 and certainly Amazon are going in the right direction with the 332 00:15:56,710 --> 00:16:00,940 likes of Aurora Limitless, which does have like native sharding. 333 00:16:01,545 --> 00:16:04,814 Um, it's ex, it is a Postgres setup, but it's, 334 00:16:04,814 --> 00:16:06,464 they're, they're thinking about it in the right way. 335 00:16:06,464 --> 00:16:08,775 But, and they do have things like RDS proxy, 336 00:16:08,775 --> 00:16:10,365 which could do some of these proxy things. 337 00:16:10,574 --> 00:16:13,665 So they do have these building blocks and they, some of the problems 338 00:16:13,665 --> 00:16:17,655 we could solve or maybe swap out with some Amazon managed services. 339 00:16:17,865 --> 00:16:19,365 But really we were looking for something, 340 00:16:20,055 --> 00:16:23,204 something a bit bigger and better and where. 341 00:16:23,710 --> 00:16:26,320 Actually serving queries, actually serving customers. 342 00:16:26,320 --> 00:16:30,280 Our customers is the problem of the provider that we have. 343 00:16:30,520 --> 00:16:34,660 We don't just want to be getting a proxy service from one part of 344 00:16:34,660 --> 00:16:39,430 the company and limited insights or no ability to go in and help us 345 00:16:39,430 --> 00:16:42,760 out with bad queries or give us insights into what's going wrong. 346 00:16:42,760 --> 00:16:43,660 You know, we wanna. 347 00:16:43,980 --> 00:16:47,850 We, we really need somebody who's like a partner who can go deeper 348 00:16:47,850 --> 00:16:51,990 into our problems and share our problems and not just be hands 349 00:16:51,990 --> 00:16:56,760 off with them, which, you know, at Amazon due to scale and due 350 00:16:56,760 --> 00:17:00,810 to the way they treat due security and a bunch of other reasons, 351 00:17:01,470 --> 00:17:05,069 uh, they don't act that way in their day-to-day operations. 352 00:17:05,069 --> 00:17:06,630 You can convince them eventually kind of to 353 00:17:06,630 --> 00:17:08,670 get into certain things, but it's certainly. 354 00:17:09,115 --> 00:17:13,915 They don't have one small solution that it fixes all the problems that we 355 00:17:13,915 --> 00:17:16,615 want, like connection, pooling, sharding, fast failovers and everything. 356 00:17:16,795 --> 00:17:19,974 They're kind of just vending, a bunch of building blocks and maybe it's just 357 00:17:20,035 --> 00:17:24,240 because they're two pizza teams the whole way down and but planet scale are I. 358 00:17:25,020 --> 00:17:29,520 A good, healthy, up and coming company who we liked the look of. 359 00:17:29,520 --> 00:17:32,310 We liked the way that they were talking about providing 360 00:17:32,310 --> 00:17:35,460 managed VI tests in their into companies like us. 361 00:17:35,700 --> 00:17:38,520 I think what we liked about Planet Scale was like they were clearly 362 00:17:38,520 --> 00:17:42,030 building for companies like us using a technology built for. 363 00:17:42,449 --> 00:17:43,080 Exactly. 364 00:17:43,080 --> 00:17:44,100 Customers like us. 365 00:17:44,399 --> 00:17:49,620 Um, and you know, the, the kind of way like the, they 366 00:17:49,620 --> 00:17:52,770 kind of like a one stop shop white glove service. 367 00:17:53,070 --> 00:17:55,770 You just show up, send your queries at their database and 368 00:17:55,770 --> 00:17:58,709 they'll do the rest as opposed to you need to assemble 369 00:17:58,709 --> 00:18:01,590 a, a. A variety of building blocks and hope for the best. 370 00:18:01,649 --> 00:18:03,659 The one challenge I see coming outta the planet scale 371 00:18:03,659 --> 00:18:05,370 folks, they have, they have amazing talent there. 372 00:18:05,370 --> 00:18:07,649 Richard Crowley I've known for years is phenomenal. 373 00:18:07,649 --> 00:18:10,139 Sam Lambert is the CEO and he is there. 374 00:18:10,199 --> 00:18:10,770 Kidding, kidding. 375 00:18:10,949 --> 00:18:13,949 Uh, they have a bunch of terrific folk working there, but I find that the 376 00:18:13,949 --> 00:18:17,399 way that their position of the stories they tell are aligned perfectly. 377 00:18:17,429 --> 00:18:20,459 For folks like you, you are deep in the weeds. 378 00:18:20,459 --> 00:18:21,600 You know this stuff cold. 379 00:18:21,600 --> 00:18:24,870 You have been running hyperscale systems for many years. 380 00:18:25,650 --> 00:18:26,400 Terrific. 381 00:18:26,640 --> 00:18:29,220 There are a lot more people that look like me, by which I 382 00:18:29,220 --> 00:18:32,430 mean dumb out in the universe than there are people like you. 383 00:18:32,430 --> 00:18:36,420 So making it a a broader mass market appeal seems like it's not the story 384 00:18:36,420 --> 00:18:40,050 they're telling at the moment, which is kind of a shame because based on 385 00:18:40,050 --> 00:18:43,350 the stories I've had with you and others at the conversations around this. 386 00:18:43,710 --> 00:18:46,860 They're, they're solving a problem that meets an awful lot of people. 387 00:18:47,250 --> 00:18:49,350 It a awful lot of people's problems. 388 00:18:49,560 --> 00:18:52,800 Uh, I will also say that this reinforces a belief I've had 389 00:18:52,800 --> 00:18:56,370 for many years, which is as val, as things move up the stack, 390 00:18:56,670 --> 00:18:59,850 the value and the margins increase by being able to do it. 391 00:19:00,120 --> 00:19:02,969 Amazon has got the low level infrastructure stuff on lock. 392 00:19:02,969 --> 00:19:06,360 No one is gonna build a better VM platform than they're doing. 393 00:19:06,629 --> 00:19:08,879 Uh, their reliability is untouchable. 394 00:19:08,969 --> 00:19:11,370 They have all kinds of great baseline foundational 395 00:19:11,370 --> 00:19:14,010 services, but every time they try to move up the stack 396 00:19:14,010 --> 00:19:17,129 into applications or things a little further up the chain. 397 00:19:17,770 --> 00:19:19,300 They fail miserably. 398 00:19:19,330 --> 00:19:23,980 They've never yet built a good user interface on anything Amazon has ever done. 399 00:19:23,980 --> 00:19:27,190 We all learned to use their website, not because it's good, because we 400 00:19:27,190 --> 00:19:30,460 have to, and what we're seeing with things like planet Scale is they're 401 00:19:30,460 --> 00:19:33,610 now the, the rest of the industry is starting to erode some of those 402 00:19:33,610 --> 00:19:37,930 things and come further down faster than Amazon is able to go up the stack. 403 00:19:38,170 --> 00:19:40,240 I, it's not just things like planet Scale. 404 00:19:40,240 --> 00:19:43,240 We see it with Snowflake, Databricks, uh, a whole bunch 405 00:19:43,240 --> 00:19:45,639 of other folks out there that are doing these things. 406 00:19:45,639 --> 00:19:48,129 People are using Confluence, uh, sorry, confluent 407 00:19:48,430 --> 00:19:51,520 instead of running their own Kafka clusters or MSK. 408 00:19:51,790 --> 00:19:55,720 It's, it's, those companies are eroding AWS, they're charging 409 00:19:55,720 --> 00:19:59,470 more in some cases, but delivering vastly superior value. 410 00:19:59,889 --> 00:20:01,985 And this tells me in the future. 411 00:20:02,675 --> 00:20:05,314 Unless, so, they're gonna come out with something I can't foresee. 412 00:20:05,614 --> 00:20:10,235 Amazon is going to become the equivalent of the layer one backbone providers. 413 00:20:10,534 --> 00:20:11,675 The, they're gonna be next. 414 00:20:11,705 --> 00:20:15,995 Like if NTT goes down, the internet isn't working so well today and we're all 415 00:20:15,995 --> 00:20:18,844 having a bad time, but most people don't know what the hell that company is. 416 00:20:19,530 --> 00:20:21,810 Everything and the, all the value rides on top of them, 417 00:20:21,899 --> 00:20:24,030 and I think that's Amazon's future given their course. 418 00:20:24,149 --> 00:20:24,330 Yeah. 419 00:20:24,330 --> 00:20:27,270 We've seen this isn't a pattern that is, we've only 420 00:20:27,270 --> 00:20:29,490 seen with, with the move from Aurora to Planet Scale. 421 00:20:29,610 --> 00:20:32,940 We had the exact same with a move from Redshift to Snowflake. 422 00:20:33,750 --> 00:20:36,720 Uh, and again, we worked closely with Amazon. 423 00:20:36,720 --> 00:20:39,480 We tried to resolve our Redshift stability problems with them. 424 00:20:39,899 --> 00:20:41,850 They gave us a bunch of things to do, but ultimately. 425 00:20:42,584 --> 00:20:46,425 When we were moved to Snowflake, not only was the technology 426 00:20:46,425 --> 00:20:49,725 just that bit better, they were just able to ship for 427 00:20:49,725 --> 00:20:53,415 us or a bit more responsive on solving for our needs. 428 00:20:53,564 --> 00:20:56,205 But that bit hungrier of where we didn't feel like we 429 00:20:56,205 --> 00:20:59,264 were just one out of a million customers for Redshift. 430 00:20:59,594 --> 00:21:04,544 Um, with Snowflake, we got stuff turned around quickly and the thing has been. 431 00:21:05,100 --> 00:21:07,800 Pretty awesome as well and just kind of left Amazon behind. 432 00:21:08,070 --> 00:21:13,020 I think once things are business critical for us, uh, and 433 00:21:13,020 --> 00:21:16,060 they're higher, higher level applications, uh, I think it's. 434 00:21:16,980 --> 00:21:19,710 We're at a point now where we'd be considering taking 435 00:21:19,710 --> 00:21:22,290 it off Amazon rather than trying to fix it on Amazon. 436 00:21:22,770 --> 00:21:24,960 Something that could become important could be, 437 00:21:25,080 --> 00:21:29,040 say, DDoS has became really problematic for us. 438 00:21:29,159 --> 00:21:30,420 At the moment, we just use waf. 439 00:21:30,450 --> 00:21:31,710 You know, we use the Amazon stack. 440 00:21:31,710 --> 00:21:32,310 It's fine. 441 00:21:32,310 --> 00:21:32,909 It's not that. 442 00:21:33,125 --> 00:21:34,024 Big a deal for us. 443 00:21:34,415 --> 00:21:38,344 But if we really had to nail the DDoS problem, I'd probably go to 444 00:21:38,344 --> 00:21:41,975 CloudFlare and wouldn't, wouldn't stick around with Amazon for too long. 445 00:21:42,245 --> 00:21:46,594 Kind of on the understanding that I think Amazon probably do a reasonably 446 00:21:46,594 --> 00:21:49,655 good job and will, you know, they'll take support tickets and whatnot. 447 00:21:49,895 --> 00:21:52,715 You, if you're going to go a DDoS, you definitely need to talk to your provider. 448 00:21:52,715 --> 00:21:54,485 There's no real way around that. 449 00:21:54,514 --> 00:21:54,695 Yeah. 450 00:21:54,695 --> 00:21:58,294 And their team is excellent, but the, the customer touch 451 00:21:58,294 --> 00:22:02,615 points, they're, they're not really, I'm sorry, but they aren't. 452 00:22:02,725 --> 00:22:07,915 And, you know, AWS support, it can be tough to get listened to at times. 453 00:22:07,915 --> 00:22:10,135 Like I've done On-Call, a lot of on-call and 454 00:22:10,225 --> 00:22:13,465 opened a lot of, uh, issues with AWS support. 455 00:22:13,705 --> 00:22:16,885 And even just knowing how to open a case, it's like it's pretty difficult. 456 00:22:17,125 --> 00:22:19,495 Whereas if I'm opening a support case with some 457 00:22:19,495 --> 00:22:22,584 of our providers, like Open AI or incident.io or. 458 00:22:22,675 --> 00:22:25,345 Or Snowflake or, or or planet scale. 459 00:22:25,465 --> 00:22:27,625 Very often it's little more than a message in a 460 00:22:27,625 --> 00:22:30,325 Slack channel and all of their automation kicks in. 461 00:22:30,325 --> 00:22:32,665 You get routed to the right person very, very quickly and 462 00:22:32,665 --> 00:22:35,034 they're able to tell us very quickly if it's our problem or their 463 00:22:35,034 --> 00:22:37,945 problem, as opposed to what Amazon of where they're gonna try and 464 00:22:37,945 --> 00:22:41,185 catch you out, asking you like which region your problem is in. 465 00:22:41,185 --> 00:22:43,465 And uh, that can be frustrating at times again. 466 00:22:43,920 --> 00:22:45,690 It's a problem of scale and I kind of get it. 467 00:22:46,020 --> 00:22:48,690 Um, but the experience is way more tailored to 468 00:22:48,690 --> 00:22:52,140 our needs from smaller, hungrier c uh, companies. 469 00:22:52,140 --> 00:22:53,010 And our experience. 470 00:22:53,100 --> 00:22:56,610 This episode is sponsored by my own company, the 471 00:22:56,610 --> 00:22:59,890 Duck Bill Group, having trouble with your AWS bill. 472 00:23:00,245 --> 00:23:03,185 Perhaps it's time to renegotiate a contract with them. 473 00:23:03,395 --> 00:23:05,495 Maybe you're just wondering how to predict 474 00:23:05,495 --> 00:23:08,584 what's going on in the wide world of AWS. 475 00:23:08,735 --> 00:23:11,705 Well, that's where the Duck Bill group comes in to help. 476 00:23:12,125 --> 00:23:14,375 Remember, you can't duck the duck bill. 477 00:23:14,375 --> 00:23:16,475 Bill, which I am reliably informed by my 478 00:23:16,475 --> 00:23:19,145 business partner is absolutely not our motto. 479 00:23:20,280 --> 00:23:22,620 I, I think that you're right and it's kind of sad. 480 00:23:22,710 --> 00:23:27,660 It, it also, if I'm reading trends, it feels like Amazon is moving away 481 00:23:27,660 --> 00:23:30,990 on the AWS side, at least from product led growth and speaking explicitly 482 00:23:30,990 --> 00:23:35,010 to large enterprises and okay, maybe it's the right answer for them. 483 00:23:35,010 --> 00:23:37,680 Lord knows they have better strategic insight into their customers and 484 00:23:37,680 --> 00:23:40,380 their needs and their growth patterns than I do sitting in the cheap seats. 485 00:23:40,770 --> 00:23:44,010 But what attracted me to it was the fact that I could get started with these 486 00:23:44,010 --> 00:23:46,950 things for pennies and so much of what they're coming out with these days. 487 00:23:47,360 --> 00:23:49,070 A prerequisite enterprise support, which 488 00:23:49,070 --> 00:23:52,400 starts at $180,000 a year and ends nowhere. 489 00:23:52,400 --> 00:23:53,120 It never ends. 490 00:23:53,120 --> 00:23:56,300 It grows as an unbounded growth problem like an AWS bill itself, 491 00:23:56,660 --> 00:23:59,990 and that is that that rules out a lot of things that I'd want 492 00:23:59,990 --> 00:24:03,080 to kick the tires on unless I start taking hostages again. 493 00:24:03,380 --> 00:24:06,500 You know, the fire hose of AWS. 494 00:24:06,985 --> 00:24:09,775 Updates and launches and stuff. 495 00:24:10,075 --> 00:24:12,895 I think the hit rate for me of where I see something that 496 00:24:12,895 --> 00:24:15,505 I'm actually gonna try out or where, where I'm thinking, 497 00:24:15,685 --> 00:24:18,385 Hey, they're nailing this, they're solving our problems. 498 00:24:18,685 --> 00:24:20,905 Um, I think that's gone down over the years. 499 00:24:21,025 --> 00:24:22,975 I'm sure they're crying into their money. 500 00:24:23,665 --> 00:24:25,285 Uh, I'm sure they're, they're, they've 501 00:24:25,285 --> 00:24:26,905 got some pretty good businesses out there. 502 00:24:27,030 --> 00:24:30,745 But, uh, for the kind of mid-range cost. 503 00:24:31,199 --> 00:24:34,169 Like tech first company, it seems like they're, they're not the 504 00:24:34,169 --> 00:24:36,870 lead, the leaders that they used to be because I think with the 505 00:24:36,870 --> 00:24:39,689 likes of Aurora and Redshift, maybe they had like early mover 506 00:24:39,689 --> 00:24:42,689 advantage 'cause they had obviously access to cloud services. 507 00:24:43,350 --> 00:24:46,260 Before the cloud existed, uh, they were able to build like 508 00:24:46,320 --> 00:24:50,340 really great, um, cloud specific services on top of that. 509 00:24:50,490 --> 00:24:53,639 But I think they've been outpaced by hungrier competitors at this point. 510 00:24:53,639 --> 00:24:55,379 And, you know, it's good for us. 511 00:24:55,379 --> 00:24:58,710 We're able to take advantage of these and so I'm kind of happy to do that. 512 00:24:58,710 --> 00:25:02,430 But, um, I think it's, I'm kind of quietly sad for, for Amazon as well. 513 00:25:02,430 --> 00:25:02,460 I. 514 00:25:03,990 --> 00:25:04,650 I am too. 515 00:25:05,220 --> 00:25:07,200 Everyone seems to think I have an axe to grind against 516 00:25:07,200 --> 00:25:09,840 AWS but it comes from being close to them for so long. 517 00:25:09,840 --> 00:25:11,520 I, I don't hate the company. 518 00:25:11,520 --> 00:25:14,550 If I did what I did for a company I hated, that's a pathology 519 00:25:14,550 --> 00:25:17,010 and I need a diagnosis and probably a restraining order. 520 00:25:17,340 --> 00:25:18,690 Uh, it's, I like what they do. 521 00:25:18,690 --> 00:25:20,130 I want them to be better than they are. 522 00:25:20,130 --> 00:25:22,350 I want the offerings to improve over time. 523 00:25:22,680 --> 00:25:25,200 I just, I don't see, that's the direction it's going in the way 524 00:25:25,200 --> 00:25:28,155 that it once was, and it brings me no joy whatsoever to say that. 525 00:25:28,995 --> 00:25:31,604 Um, I mean, one of the good things about Amazon is that 526 00:25:31,935 --> 00:25:34,209 they do want to hear this stuff like saying this exact. 527 00:25:34,965 --> 00:25:37,845 Uh, thing to many, many people in Amazon. 528 00:25:37,875 --> 00:25:41,115 Over the last while, um, my excellent account 529 00:25:41,115 --> 00:25:44,655 manager has been setting me up with various leaders. 530 00:25:45,045 --> 00:25:47,895 Uh, they've been asking for documents, they've been asking for examples. 531 00:25:48,045 --> 00:25:50,805 You know, they're hungry for this stuff, so I don't, don't doubt 532 00:25:50,865 --> 00:25:54,705 that there's no desire to be the leaders or to, to really satisfy 533 00:25:54,705 --> 00:25:56,775 their customers, but, you know, it's execution we care about. 534 00:25:57,015 --> 00:25:58,255 And when it comes down to it. 535 00:25:59,280 --> 00:26:00,750 We need excellent databases. 536 00:26:00,780 --> 00:26:02,850 We need the best databases to be able to 537 00:26:02,850 --> 00:26:04,770 ship world-class product to our customers. 538 00:26:05,040 --> 00:26:06,330 And I think that that's important. 539 00:26:06,570 --> 00:26:09,150 It's, it, it's the, it's the need the customers have. 540 00:26:09,150 --> 00:26:10,830 And if the cloud provider won't give it to 541 00:26:10,830 --> 00:26:12,390 them, they will find ways to meet that need. 542 00:26:12,390 --> 00:26:13,200 It's what they do. 543 00:26:13,650 --> 00:26:16,380 Uh, a last topic that I want to get into, it's been a recurring 544 00:26:16,380 --> 00:26:18,990 theme throughout the years on this show, which is where does the. 545 00:26:19,300 --> 00:26:23,260 Generation come from because people like you and people like me who came up 546 00:26:23,260 --> 00:26:27,070 being, you know, support folk in uh, the early days back when this was all 547 00:26:27,070 --> 00:26:30,220 an open field and no one really knew how computers were supposed to work. 548 00:26:30,250 --> 00:26:33,520 Not that we do now, but we lie to ourselves and would, we gathered 549 00:26:33,520 --> 00:26:36,129 experience and came from those places to where we are now. 550 00:26:36,430 --> 00:26:38,379 That door has been firmly shut. 551 00:26:38,440 --> 00:26:40,899 That is not a path that. 552 00:26:41,475 --> 00:26:43,065 Is open, at least for me. 553 00:26:43,695 --> 00:26:44,504 Where do you come from? 554 00:26:44,534 --> 00:26:46,575 How did you get to the place that you are now? 555 00:26:46,754 --> 00:26:50,595 Yeah, so the fun part about my career, I've had just so much 556 00:26:50,595 --> 00:26:55,220 luck and fortune and random timing things, uh, that have. 557 00:26:55,650 --> 00:26:57,180 It worked out reasonably well. 558 00:26:57,180 --> 00:26:59,250 I don't think my career has been too bad to date, but 559 00:26:59,250 --> 00:27:04,830 it all started in 1997 or so when I went to university. 560 00:27:04,860 --> 00:27:09,390 And, uh, we had a, what was then called a networking society, 561 00:27:09,570 --> 00:27:13,050 which was basically a bunch of students running a few Unix boxes. 562 00:27:13,230 --> 00:27:16,140 And this was kind of in the pre-social media age where. 563 00:27:16,760 --> 00:27:20,990 We didn't have WhatsApp for, even Facebook or anything to talk to each other. 564 00:27:21,230 --> 00:27:25,070 So the obvious thing that we did back then was we had a 565 00:27:25,070 --> 00:27:29,540 large proportion of the people in our university log on to a 566 00:27:29,540 --> 00:27:33,320 Unix shell, uh, on a bunch of servers run by, uh, students. 567 00:27:33,905 --> 00:27:37,415 Uh, who, who, and we all, we struggled to keep these things online. 568 00:27:37,415 --> 00:27:41,675 It was pretty tough running these kind of Solaris uh, servers. 569 00:27:41,675 --> 00:27:43,805 And we had like instant messaging. 570 00:27:43,805 --> 00:27:45,695 We had these wrappers around, right? 571 00:27:46,205 --> 00:27:49,085 If you're old enough, you might remember, right? 572 00:27:49,175 --> 00:27:51,754 And we had like really healthy news groups and IRC and stuff like that. 573 00:27:51,754 --> 00:27:53,315 So we had this like super, super. 574 00:27:53,815 --> 00:27:57,805 Awesome community of people who are partially of people who are interested in 575 00:27:57,955 --> 00:28:01,825 doing cool stuff with tech, learning about Unix, learning about networking. 576 00:28:01,885 --> 00:28:05,665 And we had a lot of users and just like we were the largest society 577 00:28:05,665 --> 00:28:09,205 on campus, we two great parties and it was pretty, pretty cool. 578 00:28:09,385 --> 00:28:14,695 Um, but totally co uh, totally coincidentally, or like through Fortune. 579 00:28:15,385 --> 00:28:18,445 Uh, there were also some people who've ended up being like, really? 580 00:28:19,080 --> 00:28:24,090 Notable in the tech community since the likes of say, John Looney, Tanya Riley, 581 00:28:24,120 --> 00:28:29,460 Colin McCarthy, and the list goes on of people who kind of started off their 582 00:28:29,460 --> 00:28:34,380 careers in technology just tinkering around on these Unix boxes back in college. 583 00:28:34,710 --> 00:28:35,760 Um, and. 584 00:28:36,270 --> 00:28:41,610 My career largely up until maybe when I joined Intercom, um, it was all 585 00:28:41,610 --> 00:28:46,020 about getting doors open by knowing people, uh, through that community, 586 00:28:46,169 --> 00:28:50,850 staying in touch with them, you know, doing, um, doing things, whether 587 00:28:50,850 --> 00:28:54,240 it was our local Linux user group or different activities like that. 588 00:28:54,450 --> 00:28:59,610 Um, but really having, uh, good fortune from meeting a bunch of early, I guess, 589 00:28:59,669 --> 00:29:04,350 Unix tinkerers or CISA bins back in the day who all then kind of grew into, uh. 590 00:29:04,765 --> 00:29:06,535 Working in various places in the industry 591 00:29:06,535 --> 00:29:10,345 and, um, but my own, uh, where I went after. 592 00:29:10,860 --> 00:29:14,429 Tinkering around Unison College was into Solaris technical support. 593 00:29:14,669 --> 00:29:17,340 Then that moved into like real cis admin work. 594 00:29:17,639 --> 00:29:23,550 Then later into like running, uh, building out nationwide broadband networks in, 595 00:29:23,580 --> 00:29:27,750 um, in, in Ireland and connecting every school in the state and building out ISP 596 00:29:27,750 --> 00:29:31,679 services and, and all the, it was like a mix of classic system in, and a bunch 597 00:29:31,679 --> 00:29:36,240 of automation increasingly automated as, as things got better in, in technology. 598 00:29:36,690 --> 00:29:39,060 Then for a while as well, I was in this small 599 00:29:39,060 --> 00:29:42,420 bookseller called Amazon prior to, um, Intercom as well. 600 00:29:42,660 --> 00:29:46,140 Um, but I guess I had like a bit of a classic, well, what I consider to 601 00:29:46,140 --> 00:29:51,240 be a classic mo move kind of up the stack from cis admin help desk to 602 00:29:51,270 --> 00:29:54,300 writing more software, maybe a bit of management leadership, and then 603 00:29:54,300 --> 00:29:57,900 ultimately into the kind of tech leadership area that I'm in at the moment. 604 00:29:59,340 --> 00:30:00,480 Like, where do people come from? 605 00:30:00,480 --> 00:30:04,530 I mean, like, we certainly don't have the pipeline of lots of people 606 00:30:04,530 --> 00:30:07,650 sitting around like building ISPs or building hosting providers. 607 00:30:07,740 --> 00:30:09,630 I think these are all like really solve problems. 608 00:30:09,630 --> 00:30:13,020 You don't have that kind of tinkering or just hands-on work 609 00:30:13,020 --> 00:30:16,770 that you need to, to build the, to build and use these services. 610 00:30:16,770 --> 00:30:19,110 And so that, that kind of gateway. 611 00:30:19,500 --> 00:30:23,610 Into, uh, running services, infrastructure services, networking and all 612 00:30:23,610 --> 00:30:27,899 that, uh, doesn't, it's isn't obviously there as much at the moment, I think. 613 00:30:27,990 --> 00:30:30,570 No, that's the problem is the, the provider, the cloud providers 614 00:30:30,570 --> 00:30:34,320 abstracted so much of that away that I know a lot of folks 615 00:30:34,320 --> 00:30:37,830 at hyperscale born in the cloud environments like Intercom. 616 00:30:37,950 --> 00:30:39,870 I'll even ask you this, this is not. 617 00:30:39,930 --> 00:30:43,140 Necessarily, please stab coworkers in the back, but 618 00:30:43,140 --> 00:30:46,470 looking at your, at your technical team across the board, 619 00:30:46,800 --> 00:30:49,590 what is the depth of networking knowledge at Intercom? 620 00:30:49,890 --> 00:30:51,420 Oh, I think I am the networking team. 621 00:30:53,490 --> 00:30:54,330 And you're no slouching it. 622 00:30:54,330 --> 00:30:56,880 I wanna be very clear, but I gave a keynote at Nano about 623 00:30:56,880 --> 00:31:00,660 this last year where this is a, this is a perennial problem. 624 00:31:00,720 --> 00:31:03,930 I was talking to folks at AWS about this, where a lot of your 625 00:31:03,930 --> 00:31:06,815 customers do not have a deep bench of networking knowledge and. 626 00:31:07,180 --> 00:31:09,970 They make the very reasonable response of, well, that's not true. 627 00:31:09,970 --> 00:31:12,010 We were talking to a customer this morning and they 628 00:31:12,010 --> 00:31:15,220 had, they were as good at this as we are by easily. 629 00:31:15,460 --> 00:31:16,030 I'm like, great. 630 00:31:16,180 --> 00:31:19,780 Just outta curiosity, what sector was that customer in? 631 00:31:19,870 --> 00:31:20,710 Oh, they're a telco. 632 00:31:20,710 --> 00:31:20,980 Why? 633 00:31:21,195 --> 00:31:21,284 Hmm. 634 00:31:22,215 --> 00:31:23,655 Wonder if that has anything to do with it. 635 00:31:23,804 --> 00:31:23,925 Hmm. 636 00:31:23,985 --> 00:31:24,615 Imagine that. 637 00:31:24,794 --> 00:31:27,885 But these board in the cloud, standup companies don't do networking 638 00:31:27,885 --> 00:31:30,314 because you don't need to know networking until suddenly. 639 00:31:30,314 --> 00:31:34,304 You very much need to know networking, but you can go an entire career 640 00:31:34,335 --> 00:31:36,675 weaseling your way between the cracks without having to pick it up. 641 00:31:37,020 --> 00:31:39,300 You know, some of our recent hires, we've, we've we're 642 00:31:39,300 --> 00:31:42,419 fortunate enough to be close enough to a fairly large Amazon 643 00:31:42,419 --> 00:31:45,750 office, and we've hired a good few people from Amazon support. 644 00:31:46,080 --> 00:31:48,419 Um, so maybe it's shifted up the stack. 645 00:31:48,510 --> 00:31:51,719 You know, it's no longer people who are building ISPs or hosting providers. 646 00:31:51,719 --> 00:31:54,574 It's a. People who work for larger providers do in 647 00:31:54,574 --> 00:31:57,094 kind of entry level tech roles or support type roles. 648 00:31:57,304 --> 00:32:00,365 So there's, there's something of maybe le looks like the old pipelines. 649 00:32:00,365 --> 00:32:03,334 It's not the exact same, definitely different shape of people. 650 00:32:03,334 --> 00:32:06,544 Um, with, uh, they tend to be actually better at coding 651 00:32:06,544 --> 00:32:09,935 than I was back in, but when I was at their uh, level. 652 00:32:09,935 --> 00:32:11,135 But, um. 653 00:32:12,235 --> 00:32:16,285 The, it does seem like it's, uh, with AI coming in as 654 00:32:16,285 --> 00:32:18,895 well, it seems like there's gonna be a good bit of change. 655 00:32:19,254 --> 00:32:21,565 Uh, so like where people, like what skills 656 00:32:21,565 --> 00:32:25,195 people use and grow and need in their careers. 657 00:32:25,495 --> 00:32:28,975 Um, there, you know, there's concerns at the moment that the 658 00:32:28,975 --> 00:32:33,145 use of AI and engineering and to write code, we'll, like remove 659 00:32:33,145 --> 00:32:36,865 the need for junior engineers or like, just we'll, we'll, uh, 660 00:32:36,895 --> 00:32:40,105 maxim or benefit largely senior engineers or people who can. 661 00:32:40,935 --> 00:32:42,165 Guides the agent. 662 00:32:42,735 --> 00:32:44,655 Uh, m uh, LLM. 663 00:32:45,389 --> 00:32:50,340 Coding tools rather than, you know, working your way up by working, 664 00:32:50,370 --> 00:32:53,669 by working on small problems and building and shipping things. 665 00:32:53,970 --> 00:32:57,690 Um, so I think in tech, you know, there's a lot of change. 666 00:32:57,899 --> 00:33:01,200 Certainly there's many entry paths, uh, like the one that I take, 667 00:33:01,200 --> 00:33:04,169 which I think are gone, um, since some kind of replacements, but. 668 00:33:04,580 --> 00:33:07,250 Do worry about, like especially in areas like networking 669 00:33:07,250 --> 00:33:10,159 and uh, low level Unix and stuff that we're not seeing 670 00:33:10,159 --> 00:33:12,980 the kind of depth or knowledge that, uh, we used to have. 671 00:33:12,980 --> 00:33:15,064 And I don't think I'm just being bitter and old about that. 672 00:33:15,169 --> 00:33:16,820 I think it is pretty useful stuff to know. 673 00:33:17,419 --> 00:33:17,780 Oh it is. 674 00:33:18,050 --> 00:33:19,909 Uh, one thing I wanna point out, 'cause this is a recurring theme 675 00:33:19,909 --> 00:33:22,939 that I see a lot where you mentioned a few extraordinary names of 676 00:33:22,939 --> 00:33:25,550 people who are terrific and in the space that have been formative 677 00:33:25,550 --> 00:33:28,250 influences, would it surprise you to know that when I've spoken to 678 00:33:28,280 --> 00:33:31,010 multiple of those people, your name comes up in the same context? 679 00:33:31,865 --> 00:33:34,115 People don't realize that we all learn from each other. 680 00:33:34,235 --> 00:33:37,175 It's, it's one of those things where, oh yeah, those people are smart. 681 00:33:37,175 --> 00:33:38,615 I'm just an idiot sitting here. 682 00:33:38,615 --> 00:33:40,685 It, it's a common pattern and I think we 683 00:33:40,685 --> 00:33:44,315 internalize it pretty well, but it shows there. 684 00:33:44,345 --> 00:33:46,925 There's one other aspect I want to get into about Intercom. 685 00:33:47,105 --> 00:33:49,145 I was going to mention earlier, but we, we 686 00:33:49,145 --> 00:33:50,495 got, we've done this conversational path. 687 00:33:50,705 --> 00:33:52,175 It's one of those interesting things about it. 688 00:33:52,505 --> 00:33:56,475 Uh, I. Wound up focusing my skillset, which is not that dissimilar 689 00:33:56,475 --> 00:34:00,495 to yours on AWS bills, because I wanted a specific expensive 690 00:34:00,495 --> 00:34:03,314 problem eight years ago when I was getting started down this path. 691 00:34:03,554 --> 00:34:04,155 Nine years now. 692 00:34:04,155 --> 00:34:04,635 My God. 693 00:34:05,024 --> 00:34:08,264 And what the reason I did it was I was down 694 00:34:08,264 --> 00:34:10,725 to this or IAM like did I know a lot about? 695 00:34:10,725 --> 00:34:11,835 I am at the time, no. 696 00:34:11,835 --> 00:34:13,815 But I didn't know that much about AWS Bills either. 697 00:34:14,165 --> 00:34:16,775 Turns out when you focus on things, you could pick up a lot. 698 00:34:17,105 --> 00:34:19,235 But the reason I went with bills is because 699 00:34:19,235 --> 00:34:21,695 there is never a 2:00 AM billing emergency. 700 00:34:21,875 --> 00:34:25,445 I've had enough horrifying on-call experiences in my career that 701 00:34:25,445 --> 00:34:30,335 I am effectively done with it inter all companies across the board 702 00:34:30,364 --> 00:34:33,290 have on-call because they need this stuff to work in various ways. 703 00:34:33,310 --> 00:34:36,154 And you don't have every team have representatives at 704 00:34:36,215 --> 00:34:39,185 every hour around the clock in a follow the sun rotation. 705 00:34:39,935 --> 00:34:42,455 Intercom takes a unique approach to this, to my understanding. 706 00:34:42,605 --> 00:34:43,205 Tell me about it. 707 00:34:43,325 --> 00:34:46,895 Yeah, so this is one of the things that I'm most proudest 708 00:34:46,895 --> 00:34:50,585 of at Intercom, and to be clear, it's not all my work and. 709 00:34:51,765 --> 00:34:55,725 Arguably, I didn't initiate it, but I was a big influence on it, and I've 710 00:34:55,725 --> 00:34:58,665 certainly spent a lot of time, a lot of time running about it, and more 711 00:34:58,665 --> 00:35:01,335 importantly, talking about it in public and taking load of credit for it. 712 00:35:01,695 --> 00:35:05,685 But we have an on-call system where we use volunteers 713 00:35:05,685 --> 00:35:08,595 rather than conscripts, and this means that we. 714 00:35:09,545 --> 00:35:13,925 Put people on call out of office hours, um, not because 715 00:35:13,925 --> 00:35:18,425 they happen to be on a certain team or are on a a, a rota or 716 00:35:18,875 --> 00:35:22,115 know something about maybe networks or systems or anything. 717 00:35:22,385 --> 00:35:27,485 Uh, we ask for people to volunteer to join this rotation. 718 00:35:27,665 --> 00:35:30,485 And so we generally have about six or seven people, uh, in 719 00:35:30,485 --> 00:35:34,715 this rotation, and we compensate them for their time on call. 720 00:35:34,715 --> 00:35:37,925 So the way we do it is it's, you're on call for a week. 721 00:35:38,715 --> 00:35:40,515 Uh, not in in office hours. 722 00:35:40,515 --> 00:35:44,205 The teams who own the alarms that are firing will get those alarms at that time. 723 00:35:44,415 --> 00:35:46,845 But outside of office hours, if you're on call in 724 00:35:46,845 --> 00:35:49,425 this volunteer team, you get the page for that. 725 00:35:49,785 --> 00:35:53,055 Um, and, but of course you can't just say these things like, let's 726 00:35:53,055 --> 00:35:56,985 have a volunteer based on call and hope that it works out for the best. 727 00:35:57,165 --> 00:35:59,775 We have to put in place a bunch of things, uh, 728 00:35:59,835 --> 00:36:02,115 both on the technical and social side of things. 729 00:36:02,360 --> 00:36:05,180 To make sure that this thing was sustainable, that people would 730 00:36:05,180 --> 00:36:07,850 feel like it, the work was valued and not just because of the 731 00:36:07,850 --> 00:36:10,160 compensation, but that the work was rewarding and you might 732 00:36:10,160 --> 00:36:13,160 actually learn something and maybe even enjoy doing on-call work. 733 00:36:13,580 --> 00:36:17,150 So we insisted on all teams writing runbooks 734 00:36:17,150 --> 00:36:19,490 for every alarm that can page somebody. 735 00:36:19,730 --> 00:36:22,640 Most importantly, we treat every single page like 736 00:36:22,640 --> 00:36:25,610 a heart attack, kind of using charity majors. 737 00:36:25,670 --> 00:36:26,120 Uh. 738 00:36:26,335 --> 00:36:27,205 Quote here. 739 00:36:27,475 --> 00:36:31,855 Um, and so this means that say the next morning after a page goes off 740 00:36:32,125 --> 00:36:35,935 in the middle of the night or whatever, our teams take it seriously. 741 00:36:35,965 --> 00:36:38,635 In fact, they take it more seriously than as if they 742 00:36:38,635 --> 00:36:41,065 had paged somebody in their own team out of bed. 743 00:36:41,245 --> 00:36:44,755 When you're paying, paging somebody you don't know or who's remote from you. 744 00:36:45,510 --> 00:36:48,600 At a bed in the middle of night because you're, you set up a bad alarm 745 00:36:48,600 --> 00:36:51,360 or because your thing fell over, you feel you're not guiltier about that. 746 00:36:51,630 --> 00:36:54,180 Oh, whenever I page anyone I start the call, I'm sorry to 747 00:36:54,180 --> 00:36:57,240 wake you, but because it's just a little, a little politeness 748 00:36:57,240 --> 00:37:00,060 and courtesy can move mountains, but please continue 749 00:37:00,330 --> 00:37:01,680 without too much effort. 750 00:37:01,920 --> 00:37:06,210 We just got excellent buy-in from the teams who own these different areas of the 751 00:37:06,210 --> 00:37:09,750 product and you know, could be building a lot and a lot of stuff can go wrong. 752 00:37:09,990 --> 00:37:14,700 But we were able to hold a high bar for pages being. 753 00:37:15,004 --> 00:37:17,375 Actually something that a human needs to do and then 754 00:37:17,555 --> 00:37:20,765 giving that person the tools to actually fix the problem. 755 00:37:20,975 --> 00:37:23,915 We have some technical reasons why this stuff is easier for us than it 756 00:37:23,915 --> 00:37:27,484 is for compared to other companies, such as having a large Ruby on Rails 757 00:37:27,484 --> 00:37:30,305 monolith, as opposed to every single team having their own bespoke tech stack. 758 00:37:30,605 --> 00:37:34,325 So that stuff helps us, but it's more the culture and 759 00:37:34,504 --> 00:37:39,634 how we also reward and, uh, give shout outs to people. 760 00:37:39,845 --> 00:37:41,825 You know, everybody from the CEO down. 761 00:37:42,260 --> 00:37:47,135 At Christmas whenever Addie kind of time, we always make sure to, uh, not just. 762 00:37:47,820 --> 00:37:51,150 Pay the people the money for the time that they spend on call out of hours. 763 00:37:51,360 --> 00:37:54,870 But it's recognized socially and in also in 764 00:37:54,870 --> 00:37:56,790 things like promotions and things like that. 765 00:37:57,000 --> 00:37:59,760 It's something that's really, uh, valued in, in the organization. 766 00:38:00,030 --> 00:38:02,370 So we've had this in place now for seven or eight years. 767 00:38:02,370 --> 00:38:05,310 It's hard to remember exactly how long it's been sustainable. 768 00:38:05,760 --> 00:38:07,590 One of the biggest problems we've had is so many people 769 00:38:07,590 --> 00:38:11,220 want to join us, um, that pe so people actually like it. 770 00:38:11,670 --> 00:38:13,650 And we've, we've also built. 771 00:38:14,250 --> 00:38:18,330 People, uh, like we've, we've made people better operators. 772 00:38:18,330 --> 00:38:20,910 We've made people actually enjoy and learn and 773 00:38:20,910 --> 00:38:22,620 learn more about what happens in the company. 774 00:38:22,859 --> 00:38:26,400 Um, and it's been actually a, a great long-term recruitment for my own kind of 775 00:38:26,400 --> 00:38:29,790 infrastructure oriented teams where people get a taste of this kind of work. 776 00:38:30,150 --> 00:38:33,270 Um, they might just be a product engineer from some random part 777 00:38:33,270 --> 00:38:35,670 of the business, but then when they see this work and they see. 778 00:38:35,715 --> 00:38:37,845 They actually see what's going on under the hood. 779 00:38:38,145 --> 00:38:40,635 Uh, they, they ask to join our team full time. 780 00:38:40,815 --> 00:38:42,855 Um, there's other stuff we have to do as well. 781 00:38:42,855 --> 00:38:45,975 There you have to have a way for the person 782 00:38:45,975 --> 00:38:49,665 to, uh, who's on call to bring in an expert. 783 00:38:49,695 --> 00:38:52,125 Like, so We have an Instant Commander program as well, 784 00:38:52,125 --> 00:38:54,615 and there's support there so that people don't feel like 785 00:38:54,615 --> 00:38:57,135 they're isolated on their own outta the PagerDuty playbook. 786 00:38:57,315 --> 00:38:57,675 Yeah. 787 00:38:57,675 --> 00:39:01,785 And, uh, when it comes down to it as well, look, not everybody can 788 00:39:01,785 --> 00:39:04,425 solve every kind of problem and we'll just go to the back phone. 789 00:39:04,425 --> 00:39:07,905 We'll page in as many people as we need to to solve a problem, which 790 00:39:07,905 --> 00:39:11,205 is, even if you had 10 people on call, you might need to do that anyway. 791 00:39:11,415 --> 00:39:13,065 And so, uh, this has been great. 792 00:39:13,065 --> 00:39:15,405 I think having a single person on call for a business 793 00:39:15,405 --> 00:39:18,675 decide of intercom, it's, it can be challenging at times. 794 00:39:18,765 --> 00:39:21,405 Um, but we've never been at the point of where we've decided 795 00:39:21,705 --> 00:39:24,900 or been at any risk of things falling apart or having to inf. 796 00:39:25,504 --> 00:39:28,384 Put multiple teams and lots of people on call, keeping 797 00:39:28,384 --> 00:39:32,345 things down to one person on call ruins fewer lives. 798 00:39:32,674 --> 00:39:36,395 Uh, we all get a better quality of life and, uh, doing 799 00:39:36,395 --> 00:39:38,855 this sustainably gives us something that, uh, we can 800 00:39:39,064 --> 00:39:42,064 really feel like we're making a difference in our work. 801 00:39:42,335 --> 00:39:45,214 And that the work just isn't feeding the robots. 802 00:39:45,214 --> 00:39:46,174 It's like high quality. 803 00:39:46,174 --> 00:39:46,625 It's good. 804 00:39:46,625 --> 00:39:48,589 We're learning and setting. 805 00:39:52,080 --> 00:39:54,450 Not just tolerating low quality alarms and stuff like that. 806 00:39:54,750 --> 00:39:56,070 Yeah, that's the important part. 807 00:39:56,070 --> 00:39:58,440 It's if it wakes you up, you're empowered to fix it 808 00:39:58,440 --> 00:40:00,900 or turn off the alarm or just thresholds or something. 809 00:40:00,900 --> 00:40:03,180 It's, it's the human element of it. 810 00:40:03,180 --> 00:40:05,550 It's the fact that this is a, you are 811 00:40:05,550 --> 00:40:07,620 compensated for doing it as a volunteer thing. 812 00:40:07,620 --> 00:40:10,050 It's not as part of your job responsibilities. 813 00:40:10,200 --> 00:40:10,560 Yes. 814 00:40:10,560 --> 00:40:12,240 I know you have a six month old who's having 815 00:40:12,240 --> 00:40:15,120 trouble sleeping, get up anyway, it none of that. 816 00:40:15,120 --> 00:40:17,250 It's a human approach to it, and that is 817 00:40:17,250 --> 00:40:19,320 something this industry has lacked historically. 818 00:40:20,070 --> 00:40:20,280 Yeah. 819 00:40:20,285 --> 00:40:20,305 And. 820 00:40:21,049 --> 00:40:21,830 I've been. 821 00:40:22,770 --> 00:40:25,950 Spreading the good word about this, uh, trying to influence other 822 00:40:25,950 --> 00:40:29,310 places to, to, to improve things and not just accept the status quo. 823 00:40:29,339 --> 00:40:31,980 The interesting part has been having conversations with co different 824 00:40:31,980 --> 00:40:34,859 people from different companies who, who are interested in doing 825 00:40:34,859 --> 00:40:38,700 this, but they have all sorts of other issues, like whether it's. 826 00:40:38,910 --> 00:40:42,660 Many, many tech stack or different compliance approaches or just 827 00:40:42,660 --> 00:40:45,569 other socio tech technical problems that can make it difficult. 828 00:40:45,690 --> 00:40:48,870 I think we we're probably on easy mode in intercom. 829 00:40:48,870 --> 00:40:52,649 We, there's a lot of, we did design it for our culture and our technology stack. 830 00:40:52,950 --> 00:40:55,259 Uh, not everyone can do it so easily, but I 831 00:40:55,259 --> 00:40:57,960 would encourage everybody to like not accept. 832 00:40:58,120 --> 00:41:00,190 Again, like you said, status quo around. 833 00:41:00,460 --> 00:41:02,920 Um, just because you're on a certain team, you 834 00:41:02,920 --> 00:41:05,140 need to carry a pager and be always on call. 835 00:41:05,290 --> 00:41:08,230 I think being on call a lot does reduce the quality 836 00:41:08,230 --> 00:41:09,880 of your life, even if you're not being paged. 837 00:41:10,180 --> 00:41:13,120 Um, and so being deliberate about that as well as recognizing the work, 838 00:41:13,120 --> 00:41:16,810 I think it's very important and it just gives you a great story that 839 00:41:16,900 --> 00:41:19,029 shows you that you actually care about the people who work for the 840 00:41:19,029 --> 00:41:22,685 company, uh, as opposed to just being part of some machine, uh, that. 841 00:41:23,355 --> 00:41:25,484 Needs to satisfy the, the computers, 842 00:41:26,774 --> 00:41:27,765 which is important. 843 00:41:27,855 --> 00:41:31,725 There's a human piece of it, and that's the, that's the thing that gets lost. 844 00:41:31,725 --> 00:41:33,105 It's not just a technical problem. 845 00:41:35,145 --> 00:41:38,294 I, I wanna thank you for taking the time to speak with me about all of this. 846 00:41:38,384 --> 00:41:41,129 If people wanna learn more, where's the best place for 'em to find you? 847 00:41:42,660 --> 00:41:46,365 I am kind of on X Twitter, but not really anymore. 848 00:41:46,365 --> 00:41:49,335 I mean, I, I'm not so good luck. 849 00:41:50,145 --> 00:41:51,435 I'm sort of on there. 850 00:41:51,435 --> 00:41:55,634 I mean, you can type in Brian Scanlon and I, I repost work stuff, I guess. 851 00:41:55,904 --> 00:41:58,904 Um, I'm on blue Sky, but not as much as I was on Twitter. 852 00:41:58,965 --> 00:42:00,795 Again, it's like you can type my name, which 853 00:42:00,795 --> 00:42:03,075 is probably a healthy thing, but yeah, I hear you. 854 00:42:03,285 --> 00:42:04,154 Yeah, I don't know. 855 00:42:04,154 --> 00:42:05,745 I'm kind of sad about those things. 856 00:42:05,745 --> 00:42:07,140 I mean, I'm on LinkedIn, but. 857 00:42:07,630 --> 00:42:08,470 Who uses LinkedIn? 858 00:42:09,310 --> 00:42:09,790 Oh God. 859 00:42:09,790 --> 00:42:14,170 I maintain that LinkedIn remains the world's largest porn site because 860 00:42:14,170 --> 00:42:17,740 it's where business people go to pleasure themselves on the internet. 861 00:42:17,890 --> 00:42:22,030 That is the best description I've got of it, and I have no tolerance for it. 862 00:42:22,450 --> 00:42:23,320 So I don't know. 863 00:42:23,320 --> 00:42:24,820 Maybe the best place to find me will be if 864 00:42:24,820 --> 00:42:26,440 you set up a Unix server and we all just. 865 00:42:27,029 --> 00:42:29,129 Log on and use right to talk to each other. 866 00:42:29,460 --> 00:42:30,600 I figured you'd put up a personal website. 867 00:42:30,600 --> 00:42:31,830 Just have an intercom chat with me. 868 00:42:31,830 --> 00:42:33,629 Box in the corner that pops up. 869 00:42:33,629 --> 00:42:36,149 'cause you know it's not like you're doing anything else these days, right? 870 00:42:36,330 --> 00:42:36,720 That works. 871 00:42:36,720 --> 00:42:38,069 You can find me on intercom.com. 872 00:42:38,069 --> 00:42:39,660 I am the other person on the side. 873 00:42:40,529 --> 00:42:41,009 There we go. 874 00:42:41,009 --> 00:42:41,339 There we go. 875 00:42:41,370 --> 00:42:44,339 I will include links to all of this in the show notes. 876 00:42:44,549 --> 00:42:46,980 Brian, thank you so much for taking the time to speak with me. 877 00:42:46,980 --> 00:42:47,700 I appreciate it. 878 00:42:47,930 --> 00:42:48,380 That's been great. 879 00:42:48,380 --> 00:42:49,010 We should do it again, 880 00:42:49,550 --> 00:42:49,970 we should. 881 00:42:50,390 --> 00:42:53,720 Brian Scanlon, senior principal engineer at Intercom. 882 00:42:53,960 --> 00:42:57,050 I'm cloud economist Corey Quinn, and this is Screaming In the Cloud. 883 00:42:57,320 --> 00:42:59,660 If you've enjoyed this podcast, please leave a five 884 00:42:59,660 --> 00:43:01,790 star review on your podcast platform of choice. 885 00:43:01,820 --> 00:43:04,610 Whereas if you've hated this podcast, please leave a five star 886 00:43:04,610 --> 00:43:07,400 review on your podcast platform of choice along with an angry, 887 00:43:07,400 --> 00:43:10,425 insulting comment, but that that platform will not be one of 888 00:43:10,425 --> 00:43:12,920 Amazon's 'cause That's way too far up the stack for them to do well.