The Tortoise and the Agent

Hello, viewer listener. Welcome back to Fawthor, your source for deep, nuanced technical conversations. We have another honeycomb guest today or guest, I should say. We... We're continuing the honeycomb track here where we have Fred and Reid today.

How are you both doing?

Pretty good. How are you?

Good. Good. Fred, how are... How about yourself?

Quite great. I'm, I have a cold, but I'm doing alright. I'm doing alright.

Awesome. And as a reminder for viewers and listeners, we do have supporter edition in this episode. So if you want to become a supporter and get access to high quality audio and extended content, please go over to fallthrough.fm/subscribe. Oh, Let's get into it. So, Fred, Reid, I'll let you both introduce yourselves naturally, whoever wants to go 1st.

Sure. I'll I'll go. I've been... I'm a senior engineering manager at Honeycomb. I manage both site reliability engineering and the team that Fred is now on, which is Team Tenet, deals with Honeycomb Private Cloud.

I started off as an SRE about 4 years ago when the team at the time was just Fred and 1 other person, Ian. And yeah, it's been a fun ride since then.

And I'm Fred. I'm now Principal SRE, but no longer in the SRE team. I was the 1st person hired to be SRE as sort of a kickstarted team within the organization. And over the last year, moved over to the tenant team for the private cloud product as it has a big area of covering the entire product. So it's a small team.

But yeah, that's what I'm doing

right now.

I'm still getting my hands into the SRE back channels because I can't let that go fully.

Yeah. And you never I never kicked you out. SRE is like having you around.

Well, it makes sense, right? If you move from SRE, Honeycomb, I remember, was was SaaS to launch. And then now you have to bring your own cloud, so that makes sense. You're still doing the operational SRE style work, just different focus.

Yep. There's a

big effort in place to make sure that we don't have the private cloud product be a poison pill for SaaS. Like SaaS moves fast, ships 100 commits a day or something like that. And a single tenant product ships ideally once a week. And you don't want 1 to slow down the other. And so there's a lot of coordination work to make both development workflows work at the same time.

Yeah. That makes sense. I know that Charity introduced us by way of your embracing the code review bottleneck blog, Fred. And that's how we, you know, that's... I like to give our listeners and viewers understanding of how we got here today.

Right? So that's how we got here today. We were introduced on email and we're... I guess we should start there. So you wrote this blog, it looks like back in July 2026.

So, like, 5 years ago in terms of AI time. Right? What prompted this blog post?

I'm 1 of the, I guess, biggest, not naysayers of AI, but 1 of the people who dislikes it the most. I'm a big, big nerd about cognitive systems engineering, and I want to be in a position to not be the human in the loop where the role of the human in the loop is to say yes, no, and then get blamed when something goes wrong. I wanna be involved in the development of things and whatnot. And I've been very, very vocal about what I think a good workflow would be like in terms of supporting people rather than substituting them. And the blog post came because when we started the Tenet team, we knew we had to rewrite essentially the whole deployment mechanism for SaaS Honeycomb into Greenfield.

And AI was already good enough at that point to be doing a fairly decent job. And there was an agreement on the team that it would make sense to use it for a lot of work. So for maybe close to 6 to 9 months, we experimented a lot with deployment models. We did what we call falsework, which is when you build a wooden structure to put all the stones of an arch bridge until it can hold itself, and then you can take away the wooden structure. It's a bit different of a prototype, but we essentially vibe coded a lot of slop to just go like, oh, we have to grab an image and publish it.

We don't have a release mechanism. Screw this. We're just downloading it, sending it to Docker daemon, uploading it to the other account, and we pretend that releases work. And we did this for every single component until we had a proof of concept that worked end to end, and then we're able to strip it. And a bit later, AI got a bit better.

I think there was a big stepwise improvement around November, December 2025. And we also stopped being so greenfield. And the small team we were on, I think 4 people, had to come up with a convention about how we would try to drive AI even harder. During that time, management at Honeycomb, upper management at Honeycomb, had sent a message going like, we expect that by same time next year, you're all going to be twice as impactful because you're using AI, which frustrated me a lot because I was like, not that much of my workflow as a senior SRE is about generating code. It's not necessarily possible for me to do this.

But I still played in with this. And our team has an interesting mix of people who were very into it, like, yeah, let's generate everything. I was a bit more resistant. And that how do we work to still do code reviews and lean into it, as mentioned in the blog post, was the sort of team level compromise that made everyone sort of go like, yeah, that's an interesting experiment. And we ran with it.

And Reid, were you on the other side of this? Or were you in agreement with Fred? Or where did you stand on this side?

Oh, it's... I mean, in total agreement with Fred. So since I was managing 2 teams, I was kinda like, you know, whatever approach we wanna take, I'm good with. Whatever works for the team, whatever works for this new service area, I really didn't wanna get in the way. I think, basically, the team just took a look at what tools were available, the quality of LLM tools at the time, and just kind of decided what makes sense.

And I think 1 of the things we already had a feeling about was LLM seemed to be pretty good at greenfield. Like, they... If you just say, hey. Get me this thing. Make a thing that does this.

Like, you know, it's not gonna look... It's not gonna be perfect. I mean, like Fred said, it it was very intentionally meant to be thrown away, but it was perfect for the just I needed to fill these these requirements. And once it's done that, like, then we can figure out if we actually wanna replace it or do... Or create something better.

So, yeah, I was very onboard, and then it does create a good a good story because not a lot of teams have, 1, that much greenfield stuff to do 2, like, a brand new team, a brand new service area. 3, not much old stuff to maintain. So it was, like, very... There's a lot of there are lot of advantages to being experiment here that I think other teams in our org don't get because, you know, even if we try to make a new product and we kind of change the approach and deprecate some products, it's not like those things go away, but this team was very much able to just, like, let's just hit the ground running and make the most pragmatic decisions possible and kind of just try stuff and see what works.

What do you see broadly at Honeycomb? Like, is is this vibe coding opinion learnings? Like, is it well accepted in the company? Like, are there people that are pro and against, or where does it kind of fall at Honeycomb for you all?

There's definitely a mix of pro and against, but I would say that largely everyone... I mean, for better or for worse is, like, on board with this is changing the industry, and we kind of are coming along with it. And there are some people that aren't down for that, and that's that's fine. I would say nobody is, like, not using it. I think everybody, to some extent, is using it.

I think it's more uneven between teams. Like, it's kinda funny. Our storage team has a reputation for being... For, like, not using AI in some way because I think they're kind of very vocal critics. But they are vocal critics because they are heavy users of it because they...

You know, they're 1 of the key teams in the way of our storage engine, and they have a very critical workload. So every time I get something wrong or if they merge something that they didn't fully understand, if it affects performance, that hits every single customer all at once, and it can be very, very bad. And they use it in a very different way from a team that's just that's doing, say, more design or or front end components, which is still still absolutely critical. But it's just a a different way of working when there's, say, product facing greenfield work or maybe refactoring work or, like, deep subsystem database work.

We had Doug on, and I think this... That that episode ships before this episode or shipped already.

It shipped already.

Yeah. Shipped already.

And it's good that your answer and Doug's answer are basically the same, which is great. Nice.

Nice.

Basically, you know, he he gave the same point on it where he was like, hey, we're we're we're on the storage team. It's a critical component. We don't just want to let AI do whatever, but we also want to see how we can use AI to increase velocity and and whatever. So it was good to to hear that. Fred, I I heard you had something to say there.

Sorry.

Yeah. It's been 1 of the interesting things going on internally. Honeycomb is a very, I would discursive culture of people can talk it out, and there's been long threads about whether AI works for you or not. It has been 1 of the points of contrast that was very interesting. The storage team were not necessarily perceived as heavy users because they didn't ship a lot with AI, but they experimented with it a whole lot.

And they just had very, very high barriers of what they would use for it and whatnot and had tools to introspect, like what in the prompt gives them something a bit better, a bit worse? Are there things structurally in their area of code that it can change that gives them better results? And so they were using it a whole lot and complaining loudly that it was not good enough for them until models got to some point a bit better. The parts of the organization that didn't have the same, I would say, structure to their work and code and environment found it a bit harder to see why they were not getting the same benefits at the same time. And it took a while, but in the organization, I think we ended up having a sort of understanding that there are substrates of different speeds at which you can go.

And there's a sort of idea that if you're lower down on the platform, you have to provide stability for people to build all your stuff. And if you're shipping as fast at the lowest trade as you do at the top level, the top level can't necessarily keep up. They have to adjust to whatever you provide to them. The abstractions have to have some level of stability. And so there is a shift that you can see between teas that ship new products where it is closer to vibe coding.

I think we do it in a way where people still try to read some of the stuff they do because they're going to be on call for it. And the time to figure out how it works is not when it's on fire. It's a bit before that. But teams that work closer to greenfield or closer to VIBE coding, the teams that are lower down have an approach that I would say is more about how can you yeah, there's going be cogeneration, but how can you use the more adversarial mechanisms to expand what you cover and increase quality of what you do? Or how do you use it to hammer on an optimization rather than shipping a roadmap or whatever?

Yeah. It makes me realize Yeah. That team team Tenant, the 1 that is responsible for Honeycomb Private Cloud, is actually kind of in an interesting place because it is normally in what would be a... I think the term is called, like, a pace layer. I think it's kind of gaining traction in the industry, but infrastructure would normally be at a much slower pace layer.

Like, you don't always change the way you fundamentally use Kubernetes. Like, you're not changing that every week, hopefully. But on Team Tenant, we don't change the core Kubernetes stuff, but we are messing with infrastructure in a way that just is not for SRE or core or storage. And so we can really kind of throw stuff against the wall and see what works because nobody relies on it in the same way. Like, people aren't taking Honeycomb Private Cloud internally as an API that they're then building on top of.

We're kind of kind of funneling all of the changes and wrapping them up and then distributing them to customers. But all that kind of mechanisms for how that works and kind of the differences between SaaS and, say, a customer deployment can all be... I mean, like you said, we kind of vibe coded a lot of it at 1st.

A great example of it was, I think we had public blog posts about that. We migrated from an EC2 Confluent Kafka to open source Kafka in Kubernetes that was self hosted. And doing that in the SaaS product, the way our SaaS is set up is there is a Honeycomb instance in EU. There's a Honeycomb instance in The US. The prod version sends its traffic to a dog food copy that monitors production.

We use ourselves to monitor ourselves. And then dog food traffic goes to what we call a Kibble instance, which is the 3rd 1 behind it that lets you monitor the other 1 in case it breaks. And the Kimmel 1 sends its traffic back to production in a loop. And so we have 6 Honeycomb instances, but they're kind of pre production and kind of production at the same time because if they go down, you don't know how well production is doing. And so when it came to migrating Kafka, doing the migration and your experimentation in 1 of these clusters always has the potential to mess with production.

But because we are able to boot Honeycomb in a small box, we were able to just go like, screw this. I'm just switching Kafka. When it works, then we can hand it off to the team back on the platform. In December last year, we had a major outage where I think some of our Kafka clusters became corrupted and we needed to do emergency transformations that were really, really, I'll say, critical in a short amount of time. And we were able to boot these fake environments for people on these platform teams to experiment without risking anything else that was already almost on fire.

So yeah, it creates that necessary factor, I think, to go fast with AI, which is you need to have environment. If your environment is not forgiving, then any variation in output and God knows that AI does a lot of variations in its output can be a disaster. And providing that safer environment lets you experiment until you find that stability and can deliver faster. It's been 1 of the systemic elements that had been missing to some teams here, but I think are missing to a lot of teams out there, which is if you don't have that forgiving environment, you're always going to be very nervous in using it and cautious in what you do.

I like that. It's something that's so simple and yet a lot of teams, I think, struggle with it. Having a place to dog food your own product in a safe way and like make architectural changes is a thing that we often over... I mean, we've always overlooked the sort of SRE side of of things. Right?

As as an industry, it was very much like, I made a program. Go figure out how to run it in production efficiently. It's like, well, you kinda have to think about that a little bit earlier. And that's kinda what Honeycomb Honeycomb's messaging has been, at least in the observability 2.0 sort of way for a while. Right?

Where it's like, hey. Shift left your your way of thinking about production and then observability so that the developers are understanding how this stuff impacts production. And like, Honeycomb's a pioneer in that in that area, but a lot of companies and people out there still aren't there yet. Right? They still throw things over the wall or they use AI to, like, increase velocity to the point where now your architecture is changing every day or you're rewriting something every day.

And it's like, dude, I can't build on this. Right? Like like to your point, like both of you both of you mentioned this. You you don't... You're not giving me stable APIs or things to build upon.

So as a customer now, what do I do? Right? Because now I can't really trust your platform enough to build on it, or I have to, like, submit to the LLMs and constantly change my stuff to keep up with your stuff. And, like, both of those kinda suck, right, in in their own ways. And having the ability to challenge assumptions, like, I mean, I'm at Oxide, and our whole company exists because we challenge the status quo.

Right? So, like, if we don't have a place to keep challenging the status quo, that kinda sucks. So I like hearing that you all are are doing that. My my sort of question and point of where I was going with this little monologue here is you have this sort of social technical aspect to it. Honeycomb is really big on the social technical aspect.

Right? You you... When we talked in in, like, before the show, I know you mentioned it. You mentioned it here. I don't know if it's Reed or Fred.

I don't remember who mentioned it. But the in production is like the worst time to learn about architectural changes that the LLM has made. Right? Like... Or to your point, changing Kafka in your SaaS environment is not really a good idea when you have all these users, like, relying on it.

So what's what's sort of your approach to that where... How how do you safely do this then? O... Outside of just having a safe environment, when you say, yes, I want this architecture, how do you, like, promote it or or migrate to it?

Well, I guess I'll I'll let Fred answer since he's been doing this type of work for far longer than I have. But when I think of that, it's it's not that, you know, migrating Kafka in production or in critical environments isn't possible. It's just the cost of experimenting is very high. And, I mean, you know, 10, 15, 20 years ago, we've been doing complicated database migrations for a long time, and it manages... You know, it's a pretty well known area.

You just, like, create some test environments and try it out and create a playbook and run it exactly. But for some of these things, it allows us to go a lot faster. But I'm very curious to hear what Fred says about it.

Yeah.

The reason to me that Prada is the worst time to learn about an architecture change is not because it's more difficult there. It's because the context of doing it during an incident means you don't have people around. You are under stress. You are under time pressures, which are not good conditions to do exploration and experimentation and learn. Ideally, you've paid these dues ahead of time so that when it breaks in production, you have some mental model of how things are working regardless of how good or bad it is.

But you're not starting from 0 and going like, what the hell is this signal? You have some sort of idea of where to look, where to start. AI can help you, but having that ability, I think, that goes back to being in a loop. If you are using AI in a way where you go, like, what is going on? Okay, what do I do?

You're just in that passive role of saying yes and no. And that makes it very, very difficult to 2nd guess what is going on. Build your own theory. You are getting anchored by what is happening. The way I like to think about doing these operations is that I'm going make a guess 1st.

And sometimes I will have an agent on this. I go like, search what you think it is. I'm going to focus on my way and do my own track. And if there's a sort of agreement, it's like, oh, that's probably a good signal. If there's a wild disagreement, now I have 2 theories to look into, mine and the machines.

And that gives you more breadth into what you're looking into. You have a richer model. It's like having someone to help you do this. We've had so many incidents where whoever were the 1st responders are blocked, somebody else is new comes in and goes like, oh, you look at the signal a different way, you get a different point of view, and then it unblocks everything. So that richness and diversity of approaches for me is always critical.

I like everyone to have a basic understanding of how they're working at things, how they will improvise with things ahead of time. And so there's a whole backstory of how we've set up operations at Honeycomb. We have very few runbooks except for the trickiest migrations and things like that because we want people to get good at improvising. It's like a muscle. It's a skill you develop.

You have the tooling to investigate. And in the past, if we had minor incidents, some senior people who have seen them before know exactly what is going on. If it's not major downtime, you're going to let the people who are answering and responding right now figure it out on their own. You're not necessarily going to jump in and fix the problem faster because the act of figuring it out is good skill when they find bigger challenges down the road.

Hello, viewerlistener. Thank you so much for listening to this episode of Fallthrough. You are actually listening to the regular expanded edition with more conversation and more content. So if you wanna hear more from Fred and Reid, then you should become a subscriber or a supporter. You should become a supporter if you're not 1 already by heading over to fallthrough.fm/subscribe and signing up today.

Not only do you get access to expanded editions of fall through, you also get an extra bonus conversation slash discussion in our sister show continue co proposals edition, and you get access to higher quality audio only editions. So, again, if you wanna become a supporter, head on over to fallthrough.fm, start to subscribe, and sign up today. But for now, let's get back to the show.

So going back to how do you deal with that preparation ahead of time, I think there's been a lot of discussions about we're gaining a lot of productivity by shipping faster with these things. And by default, you're going to get the pressures that are in place driving you to ship faster are going to create you to keep shipping faster and faster and faster. But there's a need in some places to go, we're going to use some of that game time and use it to learn ahead of time. And embracing the code bottleneck aspect for me was 1 of the interesting ones there. You can see code review on a very technical layer of, is the code correct?

If it's not correct, fix it. And so long as there are few bugs, I'm very happy with that. The social technical point of view, to bring it back, is like, well, what we're building is not just the product and the code. It's building the team that maintains it and steers it and knows what happens. And so the code review has a benefit that's not measurable in code.

It's measurable in the impact it has on people, in sharing context, in knowing what is going on. And when we were vibing or vibe coding hard, it would start being a point where we would almost rubber stamp each other's poor requests, especially in the falsework area of just do throw away code. But the same engineer would do the entire deployment mechanism. The same engineer would handle all the upgrade mechanisms. The same engineer would do this other part.

And then we'd be in that situation where the context of where we're at, what needs to be done next would be located in a single person. And if they were out, you couldn't progress. So the code review bottleneck of going like, it's blocking because it's important. What can we take in there and optimize for it by not by automating the work, by moving it earlier in the workflow became our plan centric approach of, if everyone agrees where we're going, then yeah, can dispatch agents. And code reviewing is going be faster, but everyone on the team has an idea of what we're going to do about it.

And that builds that awareness. And if something breaks, it's like, oh, yeah, I remember this other engineer worked on that. Let me see what shift. And everything is a bit faster by doing that.

I have a lot to to to ask about this this topic.

It's endless rabbit holes.

Yeah. Yeah. It's it's weird because I see, like, we use LLMs here for the podcast. Like, Chris does a really good job of using LLMs to, like, gather some content and editing flows and publishing flow. Like, really good job there.

And and my teams at Oxide, like, I say my teams, but, like, you know, the the teams at Oxide all have different ways that they use LLMs. And I'm seeing this not even just in Oxide, but but even in the broader field. Like, it's very easy for teams to kind of... Let's say you have a team of 5 people. Well, that team sort of morphs into 5 teams of 1 because they're using AI.

Right? They have their own AI sort of agents or whatever they're they're managing. So now you have 5 people all working on their own sort of solo team with AI and not always sharing context with 1 another. And that's where I think you get into this rubber stamp flow. So you're saying like the code review is sort of your synchronization point between humans where you're sharing that context.

Is there like, how has that like worked out for you then?

I I... I'd say, like, clearest indication I have of this is we would have engineers sometimes work across team boundaries, and they would send large ish pull requests that wouldn't get reviewed for a long time. And you have 2 reactions when that happens. 1 of them is going like, oh, we need to make reviewing faster because we have all these people producing and things are not happening. And then when you talk to the people who are not reviewing things, they're gonna say things like, yeah, we are already saturated on our end.

We have priorities. We have projects. We had never heard of this coming in, and so we don't even know what to look for at 1st. And you have to block because you are synchronizing every bit that failed before then in terms of what I would call inefficient planning. You had multiple teams working on distinct stuff, not telling ahead of time.

You don't know when it's coming. You don't know what's happening. And then you're looking at a whole project and you go like, oh, this approach is not quite ideal. If you had told me ahead of time, it would have worked better. And that was the same thing for me like 10, 15 years ago in companies without It just took longer to get there.

And so when you see the code review bottleneck happening there, you can go like, oh, my reviewers are slow. I need to get rid of my reviewers. I like to think back a bit more as like, we're blocking because this matters. Where could we move the synchronization points to be earlier or more frequent so that it is not a bottleneck, but we still have the synchronization? Like, in removing it, are we losing something really important?

And in some cases, can go like, you know what, there's a security layer in there. It's going to gain us a lot of time. Remove it. We're not losing anything. But I like to ask a question like, what are we getting for free in doing this that has a function?

Code review was never necessarily meant to be like fix your planning. But that's the function it plays in organizations where you have too much siloing.

And

I like to ask the questions like, yeah, what if instead of doing this, we just ask the questions earlier? So when it comes back, there's already someone assigned to that PR. And the rubber stamp doesn't feel like a rubber stamp, and it doesn't feel like you're just throwing, spring cleaning on their doorstep or something. Just like they knew it was coming. They know what's going to be in there.

You have synchronized and aligned priorities. And then it's no longer a bottleneck. It's synchronization.

Yeah. It feels like the 2 important touch points for software development are, 1st of all, code review, because that's when stuff makes it into production. And the 2nd 1 is incidents. And you can't... I I do love it when people say, oh, we should, like, eliminate...

Know, we wanna lower the number of incidents. We should eliminate incidents. Like, you... Unless you're doing everything perfectly, which is not ever going to happen, there will be incidents. And an incident isn't, a discrete class of thing.

It's just like, oh, something is going wrong. We have to slam the button. We need some help. It could be something emergent from your systems. It could be an actual bad cone change.

It could be something completely unexpected. It could be a data center getting hit by a meteor. It's... You can't eliminate incidents. You can only eliminate people reporting them to you.

And I think the thing you should be looking for when you have, say, a large class of incidents is, like, are we having the right conversations? Like, before this happened, what what led to this scenario? What were the people's mental models about how things works? And code review can be an important part of that. But like Fred said, the point at which it gets to the code review or PR, you should probably already be at the point where everybody is kinda like, oh, yeah.

Yeah. Yeah. I know this PR. I know why it's here. Sure.

Rubber stamp. I mean, maybe not every team should work like that. That isn't the be all end all for all for all teams, but it does make sense that you probably could have had that conversation sooner rather than at the point... At the last possible point before it could merge to production.

But, you know, that... That's part of the problem is you you don't wanna index on either side. Right? You don't wanna stay sort of stagnant and and, like, you know, static. And you you don't wanna be too dynamic and fluid where, like, week by week, the entire team is changing how they work.

You wanted this... You want this balance in between where you are spending time for experimentation to see like what else is out there, how can we improve, but also have some core set of stability in your team and how you work. And like I find that to be the most difficult thing to balance because there's always people that are going to fall on the fence of, I know this tool, I know this way of working, it works. Like, if it's not broke, don't fix it. But it's like, but we could improve.

And I personally don't even know where to begin with some of that stuff. Right? Like, I don't know how to go to different teams or even different customers and, like, not just present them with something, but also have them walk away changing their mind. Right? Changing the way they think a little bit and saying, hey, can I even be open to trying something new?

And it doesn't have to be and, but it doesn't have to... It could be whatever. It's just approaching that is really, really tough. And I think both of you are, from what I've gathered from this conversation and our previous conversation, open minded about these things, but also realize like we got we got to do. Right?

Like, we got work to get done also. So maybe in your own words, where have you kind of struggled with that balance? Like, you know, any any examples or or ways to... If people are listening to this, any advice for them if they're struggling with this stuff?

I'm gonna use the the example from SRE. If you have multiple failures happening at the same time, which 1 do you work on 1st? When I joined Honeycomb, we were right at the size where we no longer had the ability to review every single alarm or warning to fix them all. And the early mindset was like, Okay, we're in an area where now it's like firefighting with climate change. You're not gonna fight all the fires.

You're gonna pick the bigger fire, you're gonna control that 1 and let the other stuff burn. And 1 of the exercises there, I'm putting a link that's going to be in the show notes, I'm assuming, and Levison on severity is an exercise that the Ascertain team ran multiple times that was called, who do we kill in space to save on Earth? And the idea comes from Nancy Levison from Engineering a Safer World. She has this not this exercise, but this list of requirements NASA has. And 1 of them is like, there's a top level priority of the mission.

It's like nobody dies on Earth. And a layer below is like nobody dies in space. And there's a few other ones. But what it means is that because everyone has agreed ahead of time that nobody dies on Earth is more important, if there is a chance that through the mission they could save the astronauts in a way that would harm people and hurt, they don't do it. You choose to kill the people in space to save the people on Earth.

And so we ran this exercise with what are the core services or the key priorities that we have, the things that we absolutely cannot cancel because they are core to what we need to deliver. And then you get to that list, and the lower you are, the harder it is because you sort of go like, well, all of these things somewhat matter sometimes, but they don't always. But in my mind, whenever we've had these sorts of issues of it's really difficult to slow down to prioritize something else, it's because we are not necessarily clear enough on our priorities. And so the trade offs that we're doing of prioritizing going a bit faster are very difficult to make. So having these sessions of going, what do we value right now?

What do we need to prioritize? Where can we sacrifice a bit of things? And openly discussing that is a great way to go, you know what? We think we're in a situation like the cartoon of we have square wheels on the cart. And so we need to do urgent things, but we need to take some time to work on the wheels we have to go faster.

And it means that you are going to agree with other stakeholders, which might be a product team, might be a different team, to say, we're going to be less ambitious on the roadmap because we think we can speed it up by doing investments like this. And we've done that with also scaling projects. Every year, SRE guides an exercise with all the teams about, what's the do you have that's breaking right now? What do you think is going break in the next 6 months? And what is fine to ignore?

And by knowing which teams are already seeing smoke come out of whatever they're building, the product organization can adjust to other fine projects that doesn't tackle them much or is a lift for multiple teams at the same time. But there's this roadmap negotiation where engineering is able to push back by going like, here's what we need to make sure that we're still productive next year. That's what comes to mind. It's not necessarily going to work with the mindset of the customer. But if you're already in the situation of talking with someone like, Okay, what's really important here?

How do we preserve that? It's a great way to get started in terms of not getting lost into, you know, the tactical and still thinking of strategic here and there.

And I would add on to that. I mean, maybe for any managers, but also people who aren't managers, like, I would say the value of that conversation of the who do we kill in space was also just in the conversation itself because we had our director at the time attend that conversation, and it was a very... I don't know if it was enlightening for them, but it was very much like they put their thumb on the scale, and then the engineers feel like they know, oh, like, management agrees with these priorities. Like, the amount of comfort you get in that when you have to make a hard call in an incident is... It's great when you know you are doing right by the organization by making a certain choice.

And it's... Can be very unnerving if you're in an incident, and it's like, okay. Hypothetical scenario, you can either turn off traffic for all customers for 10 minutes, and it will save the platform. Or you can go in a degraded state, and there'll be, like, lost data for the next 6 hours. Like, what does your organization value?

Which is better? Like, some might say, just shut it off. We'll eat the downtime. It'll be okay. Some might say, you know, we can we can be degraded.

That's fine. Customers have buffers on their end. We have whatever it is, but the worst situation to be in is 1 where it's unclear, and you have to make that call on the fly at 2AM with 3 other people who are also sleep deprived. And so that's kind of where I lean on classical... I don't know.

More like classic management theory, like, what are the priorities? What are the values? Have you talked about it?

Instill priorities and values and principles into your team and, like, let them act according to them as best as possible. Like, and then, you know, back them when they do that. Right? Like

Yeah. Yeah. And it's part of, like, where you start having carrots and sticks. And this is where I think it happened a lot with AI migrations. You have usually an easy pattern to go, oh, I want people to use AI more.

So I'm going to start not threatening to punish or proposing incentives to make it easier for them to adopt it. But if you have changed nothing of the decision space you're working on, you are just adding variables to a tough situation. What clarifies the situation is going like, here's all the competing we have. If I want people to adopt this new technology, what are things I can remove from their plate cognitively in terms of decisions to make that makes it comfortable for them to align what I want. So rather than adding incentives or penalties and things like that, which complexifies the problem space, there are things where we can tell them, you know what?

Don't worry about this. Right now, I think this is your priority. Do it. And we've had this, for example, on token spend, as an example. Rather than having token maxing with a chart of am I using enough or monitoring the price to make sure people use it reasonably, it's a sort of like experiment with it.

Go figure it out. And if there's a problem, we're going tell you and we're going to adjust. And this is a kind of really, really interesting way of doing it because instead of providing visibility and incentive to do more, to do less, it's just like experiment without worrying and we're going to tell you this price. And that simplifies the decision space. And it's 1 of the things that these exercises of who do you kill in space, who do you save on Earth also does.

It's like by having very clear priorities when you are in that situation, you don't have to think very hard about it. You know what needs to happen. And organizationally, think this is what can help the most is at some point, go like, everyone did what they could individually. So keep doing what you're doing. Now we're spinning up a team that's going to work on building better tooling for AI, and you can use that tooling when it's ready.

It's no longer like, you're familiar, but we've done what we could at an individual level. We need to think at a system level. Some people now own the system level. Keep experimenting the way you did, but, you know, it removes at some point, like, it's no longer fully my responsibility to be more productive at some point. It becomes also the system's responsibility.

These initiatives, I think, can do a lot of good.

I think, at a level, this all exposes how bad we've been as an industry at doing proper risk management as well. Because that's like a huge part of this, right? It's like, I think a lot of the token maxing or the overuse of AI comes from us not understanding the risk that came with doing that and kind of maximizing on those things. And, like, what... How could this detriment our teams or our projects or the projects we have running?

So I think, like, that's that's a really important aspect of this as well that I think we haven't figured out yet, but I hope that we do in the near future. Because I think once we do actually start doing real risk management around this, I think the amount of stress and pressure that management puts on people to, like, use AI a lot will start to dissipate some. So I think for right now, the way that AI has been sold, those risks are very, they're very opaque. They're just kind of not there. They're not really being brought up.

And, except for, like, maybe some engineers and some organizations are like, hey. Like, there's some trouble here. We should probably do something about this. But certainly, I don't think it's getting to the higher levels of management quite yet. And if it is, I don't think it's being contextualized for them properly.

So at least for me, I hope 1 of the things we get out of all of this stuff that we're doing is much better risk management, I guess, muscles or much better risk management apparatuses.

And kind of alluding to what Fred said, it is kind of related to risk is you can make an AI mandate, you can say, hey. You all have to use AI. You can set that expectation. But then you need... You also do need to listen when...

Say, you make an AI mandate, but don't change anything about the rest of the way the decisions are made or the way you do risk management or anything like that. Like, so you're beholden to the old ways of working, but you just want to change it from the top. So then you better do a good job of listening to people when they say, like, oh, this this part isn't working. Like, my job is incredibly tough because I'm maintaining all the things we've done in the past, and I'm trying to use this new thing that doesn't really fit super well on top of this stuff. So I don't know what you want me to do here.

Like, do you want me to just... What is the what is the high level goal? And that's where I kind of lean back on classic management theory stuff where, like, you just need to set clear... Well, you don't need to just set clear expectations. But if you're setting expectations, which is very key determinant of employee satisfaction and, like, whether people will stay or or leave, do you listen to them when they say, like, hey.

We are taking gigantic risks.

Do you

know that? Are you... Do you understand what this means? And management has a responsibility to... I mean, I would say ICs have a responsibility to say that we have those risks, but it is up to management to listen and say, okay.

Yes. We accept that risk. Let's go ahead and just do it. That is completely fine. And then to back it up when it actually comes to pass.

Like, we will accept more incidents, more error rates, more stuff like that. Because you can't... You simply can't say, we'll just go faster and hope that everything will go fine and work the exact same way we've been doing before with the same bar of quality. We're just not gonna read any of the code and not gonna Yeah. Change anything else.

I think that's where companies are stuck right now and why people have said, oh, the problem isn't generating code anymore. It's code review. Oh, the problem isn't code review. It's CI. Like, you keep shifting where the problem actually is because your mandates, you proverbially, not not you, you know, Fred Reed and Honeycomb.

You keep shifting the the mandates and you just say, hey, like, use AI, be faster. But then you still require 2 approvals plus a cool down period to merge PRs. So now you're just you're just getting a bottleneck of of code that is waiting for code review, but your code review process never changed to account for it. And then now you change that code review process and you're like, wait. Our CI process now is is the bottleneck, and here's why.

And just giving those mandates isn't enough. You have to, like, abide by these principles and be open to change in other parts of the stack. And, Rita, I saw you grabbed a book on on your shelf there. I was curious as to, yeah, where you're going. Yeah.

It was just... It was kinda what I said about... I don't know if you've heard of that kind of cringey management concept. Well, it's not that cringey. I kind of hedge these things because I have a complicated relationship with measurement theory, but it's the Gallup people who did StrengthsFinder.

But I just did a huge study of people back in the nineties and just, like, what is the number 1 determinant of employee satisfaction? And it's like, do you know what is expected of you at work? Like, are your expectations clear, and are they, like, at all achievable? And that it just reduces people's stress to know what is expected of them. And I think, like, that's what...

Why I was saying, like, the key thing is to set those expectations, make sure they're realistic, and then listen if they tell you that it's that it's not. Because, again, you can say the mandate, like, you have to do everything differently, but then you don't change anything else about the work. The expectations are actually... You're... You are giving a mixed message.

Even if you are like, oh, but I put out the 5 page document that says exactly what you should be doing. Like, okay. Let's just add it on to everything that they already feel was their job.

That's where there's something, I think, funny with everyone having AI mandates all of a sudden. Is that... Because all organizations have AI mandates because they need competitive, to because it comes from investors, because it comes from whatever, you can start shopping for a bunch of different tools with greater budgets if it promises to have AI involved in it. And I think it's 1 of the not hidden subverteve aspects of it. It's like, yeah, you want to do, I don't know, better tool for some task that sucked before.

You can pick a better workflow with a better tool. And if it has an AI feature, you're going get some of the AI budgets to it. It's a way for a team to empower themselves to get nicer tools in some ways sometimes. I think for us, Minar's promise to have better integration with agents was 1 of the things, and it helped for real in terms of being able to have all sorts of integrations. But you know, if you feel you're getting better tooling at the base and it also has the AI features and you're sort of double winning on that.

Totally. This is probably also... Well, definitely a change in calculus in terms of build versus buy nowadays. Like, I... We learned that our incident management tool was getting deprecated.

It's called Jelly. They were acquired by PagerDuty, and now PagerDuty is kinda Jelly

was so good. Yeah.

Yeah. Yep. It's... I spent, I don't know, a month, like, vibe coding a replacement, and it worked pretty well. But then when it actually came time to be like, okay.

So if we start using this, I either have to pass it to my team and say, hey. Here's this vibe coded ball of code that you now need to maintain, or I need to personally maintain it. And, honestly, I was like, incident. Io looks great. Everybody loves it.

Let's just use that. And I... But I have the feeling that that calculus in, say, like, 2 years might be might be a bit different.

And even in terms of, like, the incident investigation feature, the analysis bit is 1 where I think the stuff Jelly did, nobody else does because what most people do is to please generate the action items for me. And Jelly was really about do the thematic analysis to get insights out of that. And it's the 1 aspect that I ended up having a vibe coding tool do for me. And it's all local use, so there's no concerns about the data privacy things about it. But it's 1 of these where, probably for cheaper than the value of the contract, this part of the stack that I really wanted can be done only for me.

And then other people, if they want to nerd out about incidents the way I do, can use it. But it's 1 of the parts where in the pre AI world, I would have had to buy something or just use a Google Doc or something like that because I wouldn't have had the time to build a tool like that. And it becomes very interesting for this where I don't actually give a about the code. There's no risk with it, really. It's running on my machine.

But I care a lot about the workflow, and I can tailor it to what I want. And this is where I think the biggest shift for a lot of AI things are to be in the industry. It's that sort of tailor made, this is my workflow, I know what I want, just build a thing. Even if it's not perfect, who cares? Because I know exactly what I want to happen in this case, and it doesn't have to serve anybody else right.

I think this was in the show notes, but that's kind of why I made the... What we call the internal AWS platform, which is basically just a vibe coded place for people to put their vibe coded stuff that is internal. It does not touch customer data. But we noticed that people were creating cloud artifacts and Slack bots and, like, launching them wherever they had access, which was not a Here's great a link

to my app, http:onehundred72. Whatever, yeah, whatever IP.

In cases, like, some of them would be like, I'm trying this vendor. It's like, has it gone through vendor reviews? Like, no, but I just need to share this thing. It's like, no. We need to do a vendor We have a process with this.

So I... So Reid,

you built, like, a a place for all this stuff is what you're saying? Sorry.

Basically, yeah. I I mean, I I vibe coded the whole thing. I put it in a Sony AVios account. It it... Vibe deploys.

What what it does is, like, you you wanna host, say... Actually, N8N was 1 of the earlier ones. Like, our RevOps people wanted to do some automations with... To be honest, I don't even know what it does, but it's apparently very useful. Would have been a pain to put it on a normal infrastructure, but it doesn't need to touch any of it.

So you create a directory. You point your agent at that directory and say, oh, I want to run an app here. It reads the README in the repo, and then it creates a deploy dot m d file. And then whenever you merge a pull request that does not require any approval, Bedrock goes in and reads your deploy dot m d and does the the deploy for you. And it is way better than...

It's... I will say it's not perfect. Like, if you can get in weird, like, CloudFormation loops or if you're using Terraform, it gets into weird... It can get into a weird state. But, like, just funneling all that into 1 place that security can look at, that we can audit everything all at once, that peep...

That people who are actually not extremely software engineering oriented can manage their own stuff. It's been an improvement. If I had more time, I might wanna make it a little bit nicer to use, but for now, it's been it's been very helpful.

And it plays on 2 principles we had mentioned. Right? It is a safer space for you to try a thing and break things, and it's not gonna impact them. And it removes the complexity of choosing where it goes. Your garbage goes in there.

That's a garbage cloud, if you want to call it that. That's not the way we call it. That's the way I like to call things like that. But you can put whatever you want there, and it's almost never going to be a problem for anybody else. And so it's safer for the organization to ship and play with a lot of experiments like that without delays.

And it's like a sign of a great platform. This is sort of like, if it's something that's going to be great, it doesn't matter if it sucks. It still provides value. And so that's when you know it's really worth doing. And what Reid mentioned, it doesn't have to be perfect.

It's been vied, but it's provided tremendous leverage within the organization to be able to do that.

The difference was with my webcoded incident tool, that thing does have to work. We do have to be able to coordinate and communicate. And then to that, I I don't personally want to be paged when my incident management tool that I've I've coded is not working for our engineers. And that was kinda kinda the cut off for me, especially because I'm a manager, and I I don't I don't really wanna do that.

That makes sense. I mean, I see a lot of companies struggling struggling with this today of, like, where to put the artifacts or the temporary static sites or whatever it is. And some people put it on, like, Vercel or some people put it on a local VM that they have somewhere and they give you a link to it. And, like, I like having a single place to put all these things. And I agree with your approach to, like, Vibe put something up and, like, make it sort of the dumping ground, but not in a in a bad way, but just in a in a way of just consolidating.

And it it kind of scares me what people are doing because they're just... They don't know. Right? You have these like non software focused people saying, hey, I can now vibe code a cost, the total cost analysis thing for my sales pipelines. But then you don't know if you're putting customer data in there.

Maybe sometimes you are. And then you're you're hosting it somewhere, but you don't even know where you're hosting it. It's probably public assess... Publicly accessible. And now you've just created all of these avenues of risk and for what?

And, like, we have we have some things at Oxide where we spin up these vibe coded apps, and we basically have just been telling people, like, here's how you put this behind Google auth so that only Oxide people can, like see this thing because we have no guarantees of whether or not you're going put customer data or sensitive data in there because you don't have guarantees, right? Like, no, nobody knows. And I think that every organization should at least think about that stuff and find some way of of making them more secure.

Yeah. And if you can't... And if you don't have the capacity to build a platform or to... Even to buy a tool that does that... I forget...

I know there's some out there that are internal platform tools, but that... Even that does require organizational change. And the big you are, you'll have to be like, okay. You need to stop doing whatever it is with claw artifacts, and you'll need to learn this new workflow and this new tool, and you'll need to get access to it. You know?

You do need to do something, and the scale can be from, please just put it in this repo to, yeah, here's the golden path for all of the... Whatever it is you wanna do. It sounds like... I like Oxide's approach, especially giving just, like, please put it behind auth. Here's some guidelines.

That sounds very pragmatic.

Yeah. And we have... Like, our product is a computer that you put VMs on. So we have 1 of those computers in our office that's behind VPN. So, like, that's fine too.

Right? Like, put it behind a VPN, and you have to you have to use VPN to access that that machine. Fine. Luckily, I couldn't imagine if we were, like, a company that had just a bunch of cloud accounts and, like, everyone's spinning up a cloud account ad hoc to, like, put all these things in. That would...

I would I would just cringe if I had to do that. No. We're insinuating that you are such that that company, but, like, there's companies that sprawl so far, and it's really bad.

I mean, we do have what we call sandboxes, and occasionally, we do find out that there's some critical infrastructure in a sandbox. But I think people have been moving away from that. That's good.

And I and I agree with your earlier point of, like, the calculus regarding build versus buy is 1 way today. And I think it is gonna change, especially in 2 years. I think these purpose built workflows that you can vibe code apps for will get more beneficial, and we'll start to see some of these applications really not being worth buying them. And I don't know if something like linear goes into that category or not. I don't know.

I don't know what the future holds, but you can certainly see that a lot of the SaaS applications that are out there are generic because they have to work across a wide variety of users. And then when you start getting to the edges of those and you say, hey. Like, I need this feature request to do this this nicety. And they say no because the, like, the 95% of the rest of the customers don't want it. That's where that starts to to break down.

And and vibe coding your own sort of workflow replacement seems attractive.

And I think that gives a little of I forgot that I just did that. I did that to PagerDuty before they had their PagerDuty bot. Like, I love that that Linear, you could just say, hey, Linear. Create this ticket. And Yeah.

Scheduling overwrites in PagerDuty was a pain, so I made a little, like, PagerDuty bot that used my personal credentials in my sandbox, and people can tag in Slack and say, hey. I need coverage for this shift. Put this person on, and it would just do it. And at the time, PagerDuty couldn't do that. Thankfully, I think I can start winding it down now that we're moving to incidents.io, but it...

I was just agreeing with you that, like, the the personal software thing is definitely gonna be real or organizational software in this case.

And it's it's to your point earlier about, okay, I've I've made some organizational software, and now I wanted to make it a thing that we actually use and rely on. That's the decision that I think companies have to come to grips with because it's either, hey, Reid, you made this thing, you are responsible for it, and you're on call for it. Or hey, Reid, you made this thing, go hire a team to, like, you know, manage it full time now or whatever. And both of those have different costs. Right?

I think there's a And that's

It changes the calculus. Sorry.

There's a fundamental question to ask about these as well is that when you spot people doing workarounds like that, it represents a need that is not being met by what the organization provides. And it's 1 of the things that you see classically with people who have platform teams, hopefully 1 platform team, some places have multiple platform teams. And people start working around the platform, and they go to 0.333 party service. And when you start looking into it, almost every time what you see is like, well, they needed new features on the platform team, but the platform team is central and understaffed or has too busy of a roadmap. They're not able to shift the stuff that's needed, and so people start working around them.

And so the set of tools of people spinning up things like that are also a very good signal of their stuff we need to provide our people that they have to build on their own to work around. And it's a power to be able to do that, but ideally you have the, I don't know, that flexibility to be able to go, there's something good there. We're going to adopt it. We're going to support it. But also, what else are people working around in a way that's not necessarily as structured right now?

And you can optimize your workflows very, very well by studying how people already do the work before choosing what to automate.

Yep. And it's it's hard to find that stuff afterwards because people are often silent about it because they're just silently working around it. And then you, like, you meet up with your colleagues at your off-site and then you watch them do something on their laptop. You're like, wait a minute. You've been doing it this way the whole time?

Like, why didn't you say something? It's like, well, it just kinda worked. And I don't know how to best surface those conversations. Maybe to to your points earlier, having this be an open forum where people can give feedback and, like, periodically challenging the status quo, maybe that's the avenue for this. I don't know if a lot of companies are set up to do that effectively, but, yeah, Honeycomb seem to have some level of that, which I think is good.

I tend to agree with... I don't know. What's his name? I probably have his book over here somewhere. Whoever it is that says that migrations are a great thing for organizations to get good at, like moving tools, moving...

Like, migrating Kafka, migrating your internal database, migrating, like, CI providers. Like, it just unearths all those nasty little nasty little things and gives you another chance. You can swander it for sure if you want, but it gives you another chance to kind of expose those weird workflows and weird place... Places that people will use the tools. And the important thing is to not say like, oh, you're doing it wrong.

Like, wow. That's so bad. That's terrible for you to be doing that. It's like, okay. What led them to do this?

And there's probably other people doing that. And... Well, 1st of all, is it actually a problem? Like, it might not be. But if it is, then you should think about what led them to do that because fundamentally, unless you have some serious cultural issues, everybody wants to be doing a good job and is doing their best to do a good job at your at your company.

It reminds me of the whole, like, implement an interface 3 times to really stress test the implement... The interface itself. Like, similar vibes there. Right? Like, you...

By by not being tied to a tool and being able to, like, migrate across tools, you really find out what you're actually using with this workflow.

Yeah. Exactly.

Okay. What else I... Anything that I missed to talk to you all about based on our previous conversations? Because I only took... You know, I took pen and paper notes, so I couldn't write everything down.

I just remembered it was Will Larson. Will Larson is the 1 that loves to talk about migrations.

He's the staff engineer author. Right? I think that... That's the book I think he wrote.

Yep. And Elegant Puzzle. Yep. Elegant Puzzle. Yeah.

Yeah.

I think I have Elegant Puzzle on my shelf. I haven't read it yet. But staff engineer, I read it. It was good.

Yeah. Fred, I think you're muted.

Yep. We mentioned briefly about how things keep up with stuff. I think the blog post wasn't out at the time, but that's a recent 1 about how, for at least our tenant team, we have to essentially integrate the work output of like 80 engineers, and there's like 3 or 4 of us. And it's a little bit of the story about how we used to do this by hand by looking at all the pull requests until we sort of sat down and figured out

Is this the drink from

the fire hose blog post?

Yes. Okay. Yes. We sat down and figured out, oh, we mostly care when these files change inputs, outputs, the stuff that breaks interfaces and Terraform modules, health charts, the values that have that represent a migration. When an application goes from environment to environment over the course of weeks, you go like, well, that's a new app that didn't exist.

And so there's been a whole set of work to restructure how we do to make it easy to analyze. And once it was easy to analyze, only once it was easy to analyze, you put an agent on it. And then you did the 1st few migrations by hand, kept traces, and eventually we just trained the agents to mostly do automated upgrades of the cluster by structuring the information flows properly. The key point of view there is you can either make a puzzle solving machine or restructure the puzzle until it's no longer a puzzle. And I very much like that aspect of restructure until it's not a complex thing anymore, or at least it is, like, well aligned to be trivial to do, even if it has complexity baked in the structure.

Yeah. That makes sense. Imagine we have an approach of, like, alright, do you want to solve especially when people think, like, I'm gonna solve this problem. I'm gonna, like, blow through all the process to just fix this thing. But by doing that, you kind of, like, are sidestepping the organizational gains you could get from changing the system to be a world in which, like, say you just ignore the code review process and you, like, learn that you can escape your privileges in GitHub and just ship stuff to prod, and therefore, can ship faster.

But you've also probably made the organization slightly worse because maybe no 1 knows that you're doing that. You increase the risk of incidents. But what you could do is raise conversations about, like, I mean, the code review bottleneck in Fred's case, or you could say, like, what is the actual purpose that code review solve... Serves in our org? And do we need...

Does it need to change? Can we change these processes? Is there... If there's a compliance requirement? What is the actual wording of the compliance requirement?

That's what I think is interesting about SOC 2. It's like, okay. What are your processes? Are you actually following those processes? And that's kind of mostly what SOC 2 cares about rather than a specific model.

Yeah. That that that is that is like... I feel I feel like everybody that's listening to this is is thinking to themselves like, I'm guilty guilty of this. Right? Like...

Because it... It's true. Because at some point, you're gonna butt heads with the process, and you're gonna have to get something done on a different timeline or on an accelerated timeline usually. So you circumvent the process and, like... Or you tap that person on the shoulder that you know can can, like, unblock your thing, but it's just that person, not anyone else on their team.

So, like, you know what I mean? You you you sort of find these these areas where it's like, I know I know all the levers and whatever to pull to get my stuff moving forward. But if someone else were trying to do it, they wouldn't know those levers and it would take them, I don't know, twice as long or something. And that is a big glaring problem that organizations should talk about.

Yeah. Maybe the reason that I find that palatable is because I'm, like, I'm completely comfortable with the fact that the way I do anything is the way I do everything and that I'm... Well, comfortable, and it it also kinda sucks because I'm like, ugh. What what would be the ideal way to deal with this issue? Oh, no.

Like, am I am I a single point of failure here? And I just, through compulsion, have to think about, like, what would I want someone else to do in this scenario? And it's exhausting, but also I think it has led

to It's

necessary. Yeah. Yeah. Necessary.

For the viewer listener, we'll put this in the show notes. But if you're searching the blog post that we're referring to, it's how Honeycomb Private Cloud drinks from the fire hose. You can search that in your favorite search engine, and you go find it there by Fred from October 5. I haven't read it yet, but I do see the paragraph at the end or near the end. If we're stuck making puzzles, we can make make a puzzle solving machine.

Very... I like it. Very good way of thinking about it.

I guess I wanna also say that sometimes you do just need to blow up the process and go and show that things are possible. Like, that is a very... An actual valuable thing because sometimes people are... We're all just so stuck in our ways of, you know, say, like, I adopted AI. I tried it out, and I have been sitting on my patterns for 6 months, and I need to update them.

And someone is like, oh, you should just blow those up and try something new. And sometimes that's that's quite useful.

Similarly, like, talking out loud about the patterns you've developed, I think, is important. Because I because I I I do this too, or I'm guilty of this too, I should say. Like, there would be someone at Oxide that's like, hey. I did x y z. And I'm like, oh, I've been doing that for, like, 6 months.

But I never told anybody. Right? So, like, of the problem was me. I know... I wasn't spreading that knowledge.

And it could be, like, how you use AI agents or whatever it is, but you sometimes look at something that you're, like, your colleague says, and you're like, wait a minute. You just found out about this? How? Like, this has been a thing for x months. And I think just having those conversations and learning out loud and in public is important.

And I think AI AI makes it easier to do that, but it also makes us easier to silo. Right? Like... So I I don't know. It depend...

And it all depends on your organization's culture. Right? If you have an area where your colleagues can go freely, just talk about what they're doing and and showcase their things, You should use that to... You should use some of that time to say, hey, like, I've been doing x x for the past, you know, y months. Here's how it how it works.

And then you get a lot of people that are like, oh, wow. That actually solves my workflow problem. Thanks. I'm gonna try that. And you...

You'd be surprised at the at the response you get when you just share and and what you're learning out loud. Cool. Anything that we left unsaid? I know we're running up against time here.

That's the thing I've kind of been thinking about over the course of this episode. And is that, like, Fred, a long time ago, you wrote this article about queues, and you had this, like, plumbing analogy to it. And I feel like that's the same thing we're running up against now with a lot of these agents and things where it's like, oh, with, like, agents, we've, like, installed, a very oversized sink. Or with a lot of the stuff we do, it's like, oh, you have this really site... Lighter side sink, but you have not fixed the underlying plumbing so that can actually drain the water away.

So when we, like, take buckets and we, like, bail out the water every once in a while, we build all these extra processes around it. And at some point, you have to go back and just, like, look at the whole built system and be like, okay. Let's refake configure the system so that everything drained properly so that we have, you know, proper slides drains and the proper slides rest of the, you know, wastewater system that can carry everything out at the rate that, you know, we're pouring things in. So I think like that, there... There's a bit of, like, the the things from the old, I think, are relevant to the now again in a lot of ways.

Yeah. No. There... There's... I think in software, we have a tendency to reinvent from 1st principles in places where there's decades of history that shows what works well and what doesn't work well.

And we would save so much time by studying more of what other people do or did in the past. Like the question about we're chasing and we're just smashing bottlenecks 1 after each other. And I haven't seen a lot of discourse in the industry about what's the most amount of change a user can take before they just don't try your features anymore. Think if you're shipping features faster than users can try them, you're not doing important work. You're not going use them.

Like, that's your bottleneck at the end as a user. So if that's the thing, like, and you're able to establish that, you don't need to go any faster than this.

Yeah. And and to,

yeah, think about the holistic point of view of the system, probably is good. Yeah. Rete, you can take it over.

I was just gonna I was just gonna say that, like, a lesson that some companies have forgotten is that, like, from product thinking is, like, users don't care if your app uses AI or not. What they actually care about is does this solve my problem? Like, I am not enticed as someone who is responsible for some budget of some software by saying, like, oh, your your product's AI enabled. Like, okay. Cool.

Like, I don't personally care how linear does that at linear create this ticket for me. Like, what... If that was just a a shell script. I don't I don't I don't care. But it does make my life much easier.

And I think there's a lot of forgetfulness about how, again, in the product thinking lens, like, customers don't care that your cell phone has this cool new modem. Like, great. Can I make calls? Awesome. That's 90% of what I use it for.

Well, not anymore. But I guess I'm disagreeing with Fred.

But, yeah. So to your point, like, we kind of trick ourselves and say we're solving this from 1st principles without realizing that decades of experience and thought and trial and error went into these things already. Like, you know, the no new problem, no novel problems thought. But yeah, just giving... And you can do this so easily with LLMs.

Just say, Hey, like what has existed to solve these problems before? How would it impact me today? What sort of 1st principles can I start building upon? And you'll be surprised on the results there.

Like the change in risk profiles, for example, of what we're doing, it's like there's industries that are dedicated to studying how risk works. And they've tried a lot of things that go like, this actually doesn't work as a way to track risk. And this is what we're trying to do when building products for, approaches that other industries tried 40 years ago and gave up on because they fail at scale. And we're just reinventing them because it has AI this time around. I think that's...

We could learn without experiment on some of these things and just skip ahead, in front of competition.

Yeah. Reading old books is good. I'm always reminded of Waltzing with Bears, which is written by the same people that wrote PeopleWear. And the thing about waltzing with bears, I think, like, the the the main summary of it is, like, you need to actually talk about your risk in ways that other people can, like, actually, wrestle with and actually, like, counter without be... Basically saying, like, you're wrong.

Like, 1 of the things is, like, oh, you should talk about how confident you are in something being shipped at a certain point in terms of, like, percentages. Because if you're, I'm 80% sure we can get this done in 2 weeks, then someone can be, like, well, I'm actually 40% sure. You can have a conversation. Whereas if you say, like, I think this will get shipped in 2 weeks, now someone basically just say, you're wrong. I don't think it's gonna get shipped in 2 weeks.

And it's, like, a much more abrasive thing to do. And I think if we started doing a lot more of that and, as you said, borrowing from other industries that have done these sorts of things and kind of dealt with the painful stuff, we would be able to move a lot faster. Like, another thing you you mentioned, the no 1 can die on Earth. In the medical... The testing field and medical device field, there's a there's a number called, acceptable deaths.

And people are, like, look at this and are like, oh my god. That's terrible. It's like, you know, but you have to have that because any type of new innovation is probably gonna kill some people. And it's like your goal is to save as many people as you can and to figure out what that trade off is. And if you're just saying no no deaths are acceptable, well, then you're not gonna be able to, like, make vaccines or things like that, and a lot more people wind up dying.

The closest thing we have to that in terms of policy is gonna be SLOs and SLAs, where you eventually... That was part of the challenge of getting people to adopt is No, you're going to tolerate some errors, and we're going to agree what that amount is, and we're going to decide what it is. And sometimes it will go up, sometimes it will go down. But yeah, those were discussions other fields were having. I think there's a lot of things other fields can learn from us because we can experiment so much faster than they do because our stuff is usually not life critical.

It can be. But there's a lot we can learn from them as well. Yeah. Awesome.

Well, listen. Reid, Fred, I appreciate both of you coming on, chatting with us. It's it's nice to have, like... Especially, this is, I think, the 1st time we had guests from the same company in 2 different episodes. This is pretty...

This is new to us. This is a

novelty I wanna go listen to wanna go listen to Doug's episode. That sounds interesting.

Yeah. Know. That was a that was a great episode too.

Yeah. Just hope I don't get put on a pit by Reed for this episode or something.

No. You'll you'll be good. You'll be good.

If anything, you have this

as evidence in your, in your lawsuit or whatever.

But also technically, Fred technically, Fred outranks me, I think, as a principal now.

That... That's a possible... Yeah. Honeycomb has a thing where principals technically move elsewhere in the organizational structure. The org chart hasn't been updated.

I don't mind.

Yeah. I get you. Cool. Well, thank you both for coming on. I'll read...

I guess I'll read the outro then because, you know, Chris Chris is looking at me like, you have to do the outros. I I see it on your face. This episode of fall through was brought to you by Matthew Sarbeth. He's the host. The author is produced by Chris Brando, our beats barber by Break Master Cylinder.

And I have to say this every time, but we love receiving real mail or you can talk to us on social media.

But if you

wanna send us real mail, you can do that because we have a physical address. Who our

guests are and who your co host was.

Oh, yeah. You're right. Our guests today were Reid Savage and Fred Hebert, both from Honeycomb. And our co host, my co host, was Chris Brenda. Hooray.

You did it. I'm getting better at the at the intros outros. You know, we'll figure it out. We'll figure it all out. It's okay.

We'll we'll get we'll get there. You see these eyes? I'm tired.

Listen. I've done the intro and outro for, like, 90 episodes.

So... I know. You're you're you're... I am getting there. Don't worry.

I... You know, baby didn't really cooperate much last night. It's fine.