D2DO317: Building the Operational Reality for Sandboxed Agents in Production (Sponsored)

Welcome to day 2 DevOps where the dev oops is in the details. I'm Ned Belavant and I'm joined by my collaborative co host, Kyler Middleton.

Oh, I love it. Hey, Ned. Today, we're discussing how to run agents at scale, enterprise scale, which includes, you know, orchestration, scheduling, costing, observability, about 15 other things that I can't think of right now. That's a lot of things to worry about. And today, we'll hear from solo.io and how they can worry about it for you.

Let's do it.

Michael and Nathan, welcome to day 2 DevOps. We're really excited to have you here and to talk about all things agentic on Kubernetes and making sure we make good use of our resources. I know... I think we're all familiar with the idea of running agents locally at this point. Most of us have an agent that we're running with our IDE.

Maybe we even spun up an instance of OpenClaw and experimented with that a little bit. I know I have with varying results. How... When you think about agentic workloads, how is it different than, you know, running it with an IDE or OpenClub? Michael, like what's the key difference there running agents in like an or orchestrated fashion?

So I think conceptually, it's the same. Right? Like, you're interacting with an agent, that agent is interacting with an LLM, the LLM is then interacting with an MCP server and everything else that you have going on. But the core difference fundamentally is ultimately where you're running it. Right?

So like, we're talking about scale, we're talking about is this thing secure from like a sandbox standpoint, but ultimately where it's running and how it can be scaled out. Because when I'm running an agent locally, right, like I use a lot of open code for example, that's just a terminal running locally. Right? So like, my session is local, my state is local. If that agent goes away and dies in a fire or my computer crashes, I have no way to get back to that.

Right? Like I'm not sending my open code sessions out to some Postgres DB. Whereas, when you're running them in a true orchestrated runtime, you have all that covered for you. You get to pick up where you left off. You have sessions that are stored in a database.

You can then pull traces and logs and metrics to ensure that your agents are working as they should be. And you also have the ability to secure them and again, an isolated sandbox style environment. Much easier than just again, running it locally.

Before we talk about running agents at scale, we're we're gonna get there eventually. Mhmm. But Nathan, let's stick with an individual instance. Just just 1 agent for now. What would you say are the main challenges of running that agent remotely instead of having it in your local desktop or or laptop?

I mean, blast pattern. Think the blast radius is a is a 1st fear of mine. Like what what can it get up to? So I I have I have 3 things that I I think about at all times with agents locally or even remote. Forget forget production.

We'll get to that. And it's scope, attribution, and optimization. So the scope, what could it do? And we've seen great things in the media of what they can do. And they're really good at finding ways you haven't thought of yet to get that done.

And then you have attribution. You're gonna miss something. Now I need to know what did it do. And also how much tokens did it spend doing it for that specific task and that specific agent. That's a that's a very unique thing to agentic workloads.

And the 3rd one's just optimization. This stuff is so new. It doesn't have the maturity of a lot of things that we have in other industries. There's always things that can be done better. So I frame everything around those 3 things, that scope attribution and optimization.

And there's gonna be hiccups and mistakes along the way. But as long as we don't lose 1 of those areas of focus, I think we can continually improve as we like to say in DevOps and get to that more stable place. But yeah, once it leaves your machine, I mean, what can it do? I've heard nightmare stories of just things running overnight and then by the morning like, oh 0, it did what? It subscribed to what service on Amazon?

We hear about that stuff like all the time.

Mhmm. I was feeling really good about moving it off my computer because I... You know, I've got a lot of important stuff here. And then I thought, oh, well, we'll just... We'll put it in prod.

And then I thought, oh, that's that's actually much worse. No. Now, I I don't like that at all. What are we thinking these agents are are doing? Are are these agents like, are these chatbots?

Are these monitoring or infrastructure? Or does it even really matter?

That's a good 1. Like a lot of people are just using them for the management scope of just like, hey, tell me how things are running. That's a good, hopefully, not too invasive pattern that you can you can kinda look at. Like, just take a look at all the logs. Like, what should I care about right now?

Because it's so good at looking at massive amounts of data and seeing trends like way better than people.

It's also important to remember like, you may have an agent that you tasked to do a particular thing, but ultimately, what it has access to outside of that thing that you wanted to do. I'll give you a really good example of some... I'm looking at a screenshot right now because I literally hit this like 30 minutes ago. I was doing something for 1 of our products agent registry and connecting to some MCP servers and Claude came out and said, I need to flag a mistake I just made. My last command printed your GitHub PAT in full in this conversation.

I meant to print only the 1st word of authorization header and the token's length, but the header contains just the token with no bear. So I was asking you to do a thing. I needed to be able to have an agent connect to this MCP server. And because I have the GitHub Kly local, it was able to see this personal access token. So like even though I literally said nothing about like go read this token, right, it still has the capability to do that.

And which is why isolation is so important and like what your agents actually have access to. Because if I'm using an agent on my terminal, I I'm logged into AWS, I'm logged into Azure, I'm logged into GitHub, logged into GCP, and guess what? So is my agent now.

I wanna drill down on the use cases a little bit more that that Kyler brought up because both of you are working with tons of clients. You see agentic orchestrations and workflows being built. If you could like list off, like what do you think are the top 3 most common use cases for deploying agents today?

I I think the 1st is like the standard stuff. Hey, I need this thing to to look at my Kubernetes environment, make sure that it's working as expected. Maybe some type of like SOC style agent to give me alerts if something comes in through the firewall that

Yeah.

It shouldn't necessarily be there. So stuff like that for sure. But we see a lot of different things, like we work with a lot of financial institutions, we work with a lot of governments. So, you know, when you're thinking about for example, hey, I have a trading platform or I have something along the lines of like, I need a quote to be sent to me via my bank, right, for a new mortgage or a loan or my credit score or whatever it is, like, all of that is being done via somebody with AI underneath the hood. So I would say the use cases are very very broad.

But 1 of the other huge use cases aside from those 2 categories is also like, I'm giving agents to my HR team, my finance team, my marketing team. And guess what? They can now do everything they want to do and they may not be as technical as the engineer or the architect or whoever. Right. And I say may because I know a ton of great marketing people that are super technical.

Right? So like, they may not be catching the things that the agents are doing, but guess what? If you put an AI gateway in the middle, if you put policy enforcement and security and governance, it's gonna be able to capture all that traffic. It's gonna be able to lock it down, your rate limiting, your prompt guarding, etcetera. So that's the other like 3rd major use case, which is like, I have this entire organization now that's using AI.

Okay. Secure it.

Okay. I I think I've got my head around kind of how it's being used, the 3 3 use cases or buckets that you put things into. So to to successfully do that, you're not just running 1 agent anymore. Not just like 1 instance or process or container or VM running running an agent. Now we're running multiple agents.

So, Nathan, how do you go about scheduling and orchestrating those agents being spun up? Like, is this something happens on demand? I I send a Slack message to an agent and it spins up a pod for me or something like that?

Well, so this this is something wonderfully unique to to the, AgenTic workloads is we we did all this great work breaking up the monolith into microservices, Kubernetes and DevOps and everything became agile. And then we brought state back, which was the thing we got rid of that made all of that so possible. And that's that's the huge differentiator and that's why a lot of tooling and a lot of ways we approach thing just isn't quite working for agents because you have to preserve that because you don't wanna be in the middle of a chat and then come back and the context is all gone and you're restarting the chat and having to remind, I just told you this, why are we doing this again? So that is... It's it's almost an anti pattern for all of the micro service tools that were out where it was like, have to get state out.

This thing needs to be able to nuke and recreate imperceptibly. Like that was the whole pattern, the design pattern. And now we're bringing agents in that are like, no, we can't do that. And that is the biggest kind of, oh, everything we've been doing is not gonna work and the biggest challenge that that people are really facing. You have to preserve this state.

But then also because these platforms were designed for ephemeral stateless workloads, like it might be 1 1 pod might be just designed for a certain endpoint in a much broader API and it doesn't matter if that nukes and comes back its data sources elsewhere. That's all gone. And now now the concept of how our a pod was designed, it it now has to be long lived and it has to be stateful and everything's broken. That's basically what happens. I

I'm sort of envisioning in my head the way that you might build something else that just needs to answer a quick request might be a Lambda type function where, you know, I send off a request, it spins up this temporary instance, answers my request. Maybe it has a time out of a few seconds to stick around in case I have a follow-up request and then it shuts it down and if I send a new request, new instance pops up. That does not work if you're trying to have a conversation with an agent.

It really doesn't. And great example, like using the the lander there, that's... That that was great for those kind of things but we can't do that. So when you have an entire architecture and environments built around that thinking that it doesn't matter, There's some gaps and we're focused very heavily on solving those gaps. So

I'm assuming that most of the folks you're talking to are all running agents within Kubernetes clusters. That'd a pretty safe assumption, Michael?

Yeah. The majority for sure and the reason why is because like what Kubernetes is really good at is orchestrating and scheduling workloads. Like, if you just ripped out everything around Kubernetes and you just kept it around the idea of like, I can orchestrate workloads and I can automatically schedule them to where they need to run, that'll be most efficient and effective. Like Kubernetes is the the still the best place for that.

Right?

Like, nobody's gonna put VMs or whatever in front of that in terms of like ease of use when we're talking about scaling and when we're talking about orchestration. And the good thing is is that even with Kubernetes, like, you can run your Kubernetes based agent runtime. You can bring in any type of BYO style agents. So like, if you're using CrewAI or using LangGraph or like whatever other framework, if you want to programmatically build your agents, you can run all those in Kubernetes as well. So any way, any method that you want to run agents today can a 100% be done in Kubernetes.

So, let's build out the architecture a little bit. Sounds like we've a Kubernetes cluster and we've got pods that are running the agents. Are there other components that you typically see deployed in that environment to allow for the proper orchestration and what sort of life cycle of those agents?

Yeah. So this is definitely where your sandboxing will ultimately end up coming into play. So you have your scheduling, you have your orchestration, you have the you have the run time to run the agents, again, in whatever framework or declarative or however you want to run them. And then the next question is, okay, I have this run time, but like where exactly are the agents going to run? How are they going to come up?

How are they going to come down? How are they going to be suspended? How is state gonna be stored? Are there snapshots involved? All these different things, right?

Like the lower level primitives. So like what we were talking about before, like, is the agent gonna run? How it's gonna run, etcetera. Right? Like that's almost like the user interface, the user experience.

And then when we get down to the lower level, these are these lower level components and primitives that we have to think about in terms of again, like how are these agents gonna scale out? Where are they gonna run? How are they gonna run? And that's where something like agent substrate comes into play when we're talking about Cage and being the user interface and the user experience and agent substrate being the sandbox and isolation lower level primitive.

Okay. And so to put some examples here for like the the orchestration, the the scheduling, is that like I've got a monitor agent every 5 minutes. It wakes up and it looks around and it sees if there's something wrong. And then we save the state and it just hibernates for 5 minutes. Is that kinda what you're talking about versus sort of event driven architectures like like Ned's example, like I messaged it on Slack and it woke up?

Yeah. So it absolutely could be. And the reason why is because think about how long an agent is running. So like, I open up a terminal, right, and I open Codex or OpenCode or Claude code or whatever, and I have my agent now running and it's now consuming resources. And because it's consuming these resources for virtually no reason because I may be not be...

I may not be interacting with it. Mhmm. Whereas with Kyler, to your point, I want an agent that I interact with it, it spins up very quickly, it does the thing, and then it goes down. Not not down in terms of like, it goes away or it's exited or whatever the case may be. Right?

It's killed. It's more of like, hey, I want this agent to be sitting in this warm state Mhmm. Not consuming resources, but still fully connected to its session. That way, when I interact with it, it comes up quickly just like any other agent. Just like I'm interacting with it in a terminal.

And that's 1 of the low level primitives of agent substrate that gives you that. And because of that, you don't you don't have to think about it. Right? Like it's not like you're just sitting there twiddling your thumbs waiting for a new agent to come up. It's almost like the agent's been running this entire time.

You just Mhmm. Don't see the difference. But underneath the hood, right, the plumbing is the agent coming up, the agent going into this warm state, and then coming back up when you go and interact with it via a new instance. Totally.

And you said the sandboxing. In in my head, I'm thinking like, why do I need a sandbox? I've got the tools that I give the agent, I can limit, you know, like you can't use a hammer because I haven't given you a hammer. Or I have a firewall or so or I have an I'm permissions role or something like that. What does sandboxing do beyond those tools?

How does that help me audit and and like lock these guys in so they don't, you know, get on the news like we've been seeing for the past few months from all the big AI companies?

So Michael's our... He's our sandboxing ninja. But just just before that, it means so many different things to so many people. And I think everyone's using it thinking we're all on the same page. That's not quite the case.

It is not the everything's safe, I don't have to think about it. That some people might believe that it is.

Yeah. So when when I think about a sandbox and like the industry in general, conceptually, what is a sandbox? If I take a workload and I put it into a docker container on my laptop or on a Linux box or wherever, you are sandboxed. Right? Right.

It's it's an isolated function of a workload that is running. However, this isolated function still has the ability to do full ingress and egress. Right? Full east west depending on how your Docker network or your Kubernetes network is set up. So like, yeah, it's sandboxed, but is it sandboxed?

And when we go back to like the hugging face and the open AI attacks, this is what we saw. Like, the whole reason conceptually that this sandbox agent was able to go out and pull credentials, trick the agent, etcetera, is cause it had egress access to this repo. Right? Mhmm. And in my head, like that's not a sandbox.

A sandbox is I think very much about defense in-depth. Right? Like block all and open as necessary. Right? So you're starting off as you have access to nothing.

Oh, you need access to this LLM. Okay. Let me put in agent gateway proxy up that way you can route your traffic through. Oh, you need access to this MCP server. Same thing, I'll set up an agent gateway back end, I'll route traffic to my MCP server.

So as I'm figuring out what this agent needs, I'm putting building blocks on top. Right? I'm starting out with nothing, you can't access anything. But as you need access, I'll give it. And that's what a sandbox really should be.

It's a matter of like what it has access to. And then it's also a matter of where it's running. Right? So we have things like software level isolation, which you can think about gVisor. It's effectively just isolation running in the user space.

So if you're thinking about Linux, right, it's running in the user space, it's running in that name space. It's isolated, but it's isolated at the software level. Whereas with something like hardware level isolation, utilizing maybe micro v m, you then have chip level isolation. So you have this little guest v m that's running at the chip level, right, on the CPU. And so it's not in user space.

It's not a makeshift isolation, right? Like you are fully isolated in a watered down kernel sitting at the chip level. So thinking about these different levels of isolation and sandboxing, you you really need to dive deep to understand like what exactly you're going to get out of it. Because again, you can look at a lot of sandboxing style solutions and they're like, yeah, this is a sandbox that's running in this isolated environment and there's your workload. But it's like, what does that mean?

You know, like I I go to the gym, it doesn't mean I have a 6 pack, right? Just because I walk into the gym. You know, it's it's diving deep into these different levels of what actually works underneath the hood. And that's my rant because my my engineering brain goes crazy sometimes, so...

It seems like like Nathan, what you said that everyone defines it differently, like that has been a hilarious discussion as of DevOps for the entirety of DevOps. Everyone is confident they know exactly what it is and no 1 agrees. So and it's interesting. This is such a frontier kind of place to be with running local models and using different... It's like a layer 7 gateway for MCPs and and Linux jails, but applied to Kubernetes.

And that is... I imagine this is changing really rapidly. How do you...

I love the way you frame that.

Yeah. Go ahead.

Love the way you just frame that. Like, yes, the the terminology. I mean, DevOps is a brilliant example. Some people would deem, oh, I I sent something to a REST API. We did DevOps.

I said, no, you just did the same command via a different port. Like, really? You can't make that claim. Like, is it item potent? Can I nuke and repeat?

Like, you haven't adopted the methodologies and practices. You just used an API and like software defined suddenly meant anything that was mildly automated by a playbook. And that's why I... At at the beginning, I focus on those kind of 3 areas. Like like, forget the terminology, just keep going deeper and deeper across the surface area looking at that scope and attribution.

I mean, optimization, yes, that kind of follows, but you've gotta get that scope and attribution down 1st and not get too hung up on terms like sandboxing. Like, okay, that that ticked 1 part of scope that as Michael so eloquently put, I did nothing for egress and ingress. You can't just take it off. So forget the terms and the buzzwords going around. Like, look at that blast radius.

Like, what could go on here? How many new services could it sign me up for because it's got access to the AWS CLI while I'm asleep at night. Like, that happens to people. And that would have been in the sandbox and still have signed you up for those services. So you just have to step back and keep thinking, like, what's the scope of it, what it could do, and what's the attribution and where I can be terrified in the morning and see my credit cards taking a beating because of the token ops to use overnight.

Yeah. We we've seen how creative agents can get when they have a task to do. Mhmm. And they will leverage every permission and tool that they have access to to accomplish that goal. And I think in a way, you really have to think more broadly about what scope you're giving them.

Because if you're just running a regular piece of software, it's pretty deterministic. You know what it's gonna try to do and so you can kind of tweak the permissions and scope. And if you screw it up and give it like too much permissions, you might never find out because that's not in the the tree of what the program will do. But an agent, man, is gonna find out what it can do and it's going to do that thing. So you really have to more narrowly scope.

I like the idea, Michael, of like the denial rule. Start with denial and then add permissions as you find out what it actually needs to do.

Yeah. And you also have to think about the the primitives around what this agent can interact with. Like, I'll give you an example. Over the weekend, I was working on some side project and I was using Claude. I...

Yeah. Maybe I should not do that anymore because I keep hitting... Anyways. So I was using Claude and I'm like, alright, we're gonna do this. Claude created a branch I was gonna make sure that the code was in this branch.

I was gonna do the PR and everything. And then Claude was like, oops. I accidentally pushed to main.

Yeah. Mhmm. So...

And and you know, there there's some conversation around that where people can say things like, well, you should have had branch protection on. Sure. Yes. Like there are these primitives around it. But I semi disagree on that notion because if I'm interacting with an agent and I'm telling it to do a thing, it should only have access to a thing that it needs to do.

It should not go above and beyond regardless of what protection I have around it. And again, which is why sandboxing is so important, right? Like in a sandbox environment, I may say you have access to this branch and that's it. Like this is the only path in which you have access to. Whereas on my local terminal, if I pull down a bunch of branches or I create a bunch of branches, main's gonna automatically be pulled down and therefore it's going to have set access to that as well.

So, yeah. There there's a couple of different ways to slice the cake.

I wanna shift it a little... Things a little bit to this idea we were talking earlier about the idleness of agents and how that doesn't necessarily follow the pattern of a lot of other applications. Like, if you think about a microservices application, it might have like a minimum number of required pods that you'd have and then you have some sort of horizontal auto scaling if demand spikes. Right? But when that demand goes away, you scale back in.

Agents don't seem to fit that workflow. Is that... Would that be correct, Nathan?

Scale to 0 is is really important because of just how much time an agent actually sits there doing nothing. Mhmm. It's it's kinda like... I don't know if this is a good metaphor. It's it's like a a Slack DM.

I mean, I'll Slack Michael and we'll have a regular argument over which frontier model is better for code review. It's our favorite chat to have on a daily basis. And okay. I... That's an interaction back and forth all the time.

But then there's like 1 of the channels that the team's in, they'll drop a note and be like, hey, can someone... Does someone know where I can get some of this? I'm not watching that. And I go off and I do work and I come back like a few hours later. And that's expected kind of communication.

That's That's the kind of communication that's going on with agents. Like you say, hey, you know, tell me x about y and you respond to some messages, there's some slack, take a call and you're like, oh, it's probably done now and I come back. That whole time that that's sitting there doing nothing is holding on to resources. And if I've got that running in in a provider, okay, great. That's costing me money in the provider.

But there's another angle I think a lot of people don't talk about that's growing a lot which is the sovereignty side. Many organizations, not just those under the EU act but any regulated industries trying to work out how to run these things locally. Our our products all run on GDC have been certified for, you know, Google Distributed Cloud, which can run Gemini locally now. So people can have, like, total air gap solution now and do all of their own cloud stuff. And so in those environments, what happens to all that compute?

You cannot buy RAM anymore. It's so hard to even get GPU. Do I want agents idling, holding onto pods and resources that are not doing anything? So this even becomes a problem of like doing the best with what I have because I cannot buy RAM anymore. I cannot buy the resources to grow my agents into more and more tasks that I really want to because I'm stuck with this finite amount.

So there's both the kind of the saving and there's the not wasting for what I already have.

And these agents that, you know, just like you said Nathan, right, like with Slack or whatever, it's like you do a thing, you walk away, doesn't matter what's happening, hopefully you get a response back. We're... I think Nathan and I are gonna get together to create our own protocol around this. May maybe we'll call that UDP if they... If that hasn't been taken already.

So

I know that we wanted to talk about solo.io's solution to this problem, which is Agent Substrate. So can you tell me a little bit about how Agent Substrate solves the problem of all these idle pods sitting around not doing anything most of the time?

So the way I think about it is, like we kind of already talked about, agent substrate is this like lower level primitive. Right? Like, it is the underlying system what's going on. If you think about like how you interact with Kubernetes, for example, you're interacting with it in a declare... In a declarative fashion.

Right? Like, it's... You're not, you know, you're you're creating a Kubernetes manifest. You are saying what to do, you're not saying how to do it. Right?

This is in an imperative fashion. Whereas, with a bash script or whatever, right, you're saying do this, do this, do this. So this underlying platform, you don't really have to like worry about almost. Right? It's doing everything underneath the hood and there's some magic that's happening because you set it in a Kubernetes manifest.

Right? I want 2 replicas. Go make me 2 replicas. I don't care how you do it. Go and figure it out.

That's like underlying primitives of platforms. Right? Substrate is very much the same way. So it has its own networking via the 8 net router. It has its own ingress and egress via agent gateway underneath the hood.

It has workers which are where... They're sent their pods. Right? And that's where your actors run. Your actors are agents.

And some of the... Is done by the agent substrate API and some of it is done by the Kubernetes API. So things like scheduling and orchestration is done with the Kubernetes API. Things like very quickly and efficiently spinning up actors and spinning them down into that warm state, that's via the agent substrate API. So that's all the underlying primitives.

But here's the thing, how do you interact with it? Well, maybe you got the cube CTL 8 commands, maybe you got curl, right? That's effectively it because it's just these endpoints. So you're essentially just curling these endpoints to interact with these agents. And I don't know about you, but like I don't want to sit there and do that all the time on the terminal nor does anybody from HR and marketing and finance, right?

Like they want an easy and developer experience focused way to interact with these agents. Right? To interact with these lower level primitives. And that's where Key Agent Enterprise comes into play. So when you're utilizing Key Agent Enterprise, you have the ability to have a full blown API.

There are CRs, so you have everything in in custom resource definitions via Kubernetes. And then you also have this UI. And within the UI, you can click the agents button and interact with your agent just like you would any other agent. You can do it via the terminal if you want to interact with your agents that way. You have the ability to see tokens used, full traces from where the agent came from to where it's going.

So you have this user experience which is drastically important in the AI world, right? Just like in the Kubernetes world, a bunch of different vendors and a bunch of different tools popped up to make interacting with Kubernetes much easier, right? Because people didn't want to sit in the terminal all the time. So very similarly, what we're doing is we're putting Cajun enterprise almost on top of substrate to ensure that you can interact with it in... Or with rather the experience that you want while getting all of the efficiency, all of the performance, all of the isolation, all of the sandboxing, all the security, etcetera underneath the hood.

Okay. That was a lot of things. So agent substrate is not just 1 thing, is what it sounds like. It's actually multiple things strung together or multiple components that are running together to provide this sort of unified agentic experience.

Yeah. So you know how like within within a Kubernetes cluster, right, you you look at it as 1 thing. But underneath the hood, you have core DNS, you have your queue proxy, you have your API server, you have your scheduler. Right? You have all these different components in Kubernetes to make it this 1 thing and to make it work well together.

Same concept with agent substrate. You have your 8 net router, you have your workers, you have your worker pools, you have the agent substrate API server, etcetera. So same same concept, right? But you're looking at it as 1 thing.

Same concept but oriented around the mission of we must preserve that state. Yes. And that's...

Minus the whole stateless thing.

That does bring up an interesting question. You mentioned earlier in the conversation that state is super important to agents. How is the agent substrate maintaining and feeding that state to the agents when they need it?

That's that's snapshotting. So remember when when like Lambda was 1st coming out and there were... It was all about how how do we spin up fast enough. Like, they got it down to like a 100 but because it was API endpoints, that was not enough. And this was a big thing that kind of delayed it and that that kind of related...

Resulted in the micro VM. They needed boundaries, containers weren't quite good enough and how... And then and then to solve the problem was with these micro VMs having warm start pulls. Right. And that was easier to solve because there was no state there.

You just had to have some ready to go and bang. Now my response time just got fixed. Applying that same concept, this is why we can't bring some of those kind of Lambda style, serverless style things to solve this is because they just don't care about that state. So the thing that had to be solved was how quick can I snapshot and restore while also knowing what session it was that I'm restoring to? So there's those 2 parts, the orchestration part that's going on and then there's the I know how to network you back to this 1 because that's your session then that's your state running in that particular agent of the person that I'm talking to.

And that last part was the key like the really gnarly bit to solve was like how do we do that safely in a nice little sandbox, but also do it in a way that it's imperceptible to the person interacting or other agent that's interacting. Luckily, it's not a high frequency trading kind of response time we need here when we're doing natural language communication with an agent. So we bought a little bit of time back with that, which is great. But it is still it is still super fast. I I do wanna put a small caveat around that though.

I remember when when everyone started going to containerization from from the monolith and there was some, I won't name them, some really lazy vendors that just took their heavy VM and put it inside Docker. And went, look, us DevOps were amazing, but it was a 4 gig image in a container. It's like, I don't know that you get what we're trying to solve here. You also don't wanna try and do that with substrate. If you wanna preserve that snapshot restore, don't just NPM install everything because we know how big they can get.

You wanna you wanna help it out a little here. I'm I'm not saying that it's not fast, but we've already seen some data points. In fact, 1 of our colleagues out in out in Europe, Michael, he actually did some tests and he was like, I can make it really slow. And he showed some of his image sizes and it's like, I know this is dumb, but I just want you to see, It can be bad. So you gotta you gotta think about how you're kind of packaging it up.

But do that right. It's amazing.

Yeah. And snapshots overall are very important because I'll I'll constantly hear this from customers, which is like, I'm inter... Let's say you're interacting with an agent, right? You have this context, right? You have your session there and it has a whole bunch of information.

You troubleshooted this and that, you did this and that, etcetera. And then you wanna come back to it tomorrow to do more. Or another colleague of yours is going to pick up where you left off and they still need that context. Right? They still need to see what was going on.

That session, right? We call it context, but that session still needs to be preserved somewhere, right? Like this isn't a femoral data here because that would be stateless. So because of that, we need the ability to put these snapshots in a location that's safe, right? So that's s 3, that's Google Cloud Storage, like wherever it is.

So you have the ability to put your snapshots wherever you want from a storage standpoint. That way when you go and re interact with that agent, it can go and it can pull that entire context.

Is this a tool that I can connect to like a cloud code or I, you know, Googled my my Visual Studio Code or... Yeah. I I see Nodding Michael, that that's awesome. So you'd be able to sort of bring in house the... A lot of what you're...

I mean, you might save a lot of money. Right? Is that 1 of the pitches here? Is it just security and stability and observability? Or is it like, you can also like save some serious money?

Yeah. Good question. And then thank you for bringing that up because I probably would have forgot. So we have a concept, this object called harness in the Kagen API. And with that, you can specify whatever harness you want to use, Codex, Claude, BYO, like if you want to bring your own framework.

And then we have the cage and harness. So there's a bunch of different ways to interact with the harnesses that you're already interacting with and the providers that you're already interacting with or or rather buying, right? So like if you have like an enterprise subscription to Claude or to whatever, you can go and you can interact with that via the harness object. And then you could spin up your agent with that harness in the back end.

Yeah. Cage is a critical layer on top of all of this. You don't have to have all your your developers experienced with Kubernetes and manifests and all those deployment cycles. They know only that layer above and all the magic's taken care of underneath. And so you only have to show them how to do a deploy, which is very, very simple with Cajun because Cajun's oriented around agentic life cycling.

Like, you know, how do I deploy, how do I evaluate, those kind of things. So that's that wonderful abstraction layer that anybody who now can... Has experience with agents can be deploying in these environments, which is is pretty important part of it. Otherwise, it's like going back to you have to teach them all the tooling and everything underneath and how it works and how to use it and that that that's just inhibiting learning curves and slowing things down.

And and it's also whoever else wants to interact with the agent as well, right? Like your... Whoever's managing your agentic environments, right? Like I've spoken to the ML Ops teams and the data science teams that are managing agents, sometimes it's the platform and DevOps engineers, whoever it is, they're going be the ones that are deploying and managing it, of course. But like with Cage and for example, there's no reason why you don't set up some access policies to say, hey, this person can't see traces and etcetera, right?

They can only see their agent. And then you can give this UI to HR and marketing and finance and whoever else to interact with an agent in a safe and secure way versus like going into the chat GPT UI or the Anthropic UI or whatever, right? So you have the ability to give them that developer or that that agentic experience that they can get anywhere except now you have the ability to govern it and secure it and observe it.

I have I have so many technical questions about how the agent substrate does all of this, but I feel like that would add another hour to the conversation.

Mhmm. Maybe more.

Yeah. We we could sit here for a while. It's only 01:00. I have until about 8 tonight, so no problem.

If people are interested in digging more into just the idea of Agent Substrate and how it functions from a technical perspective, I know you already have, I think, a u... A series of webinars that deal with that. Is that right, Nathan?

Yeah. We've been running a agent substrate 1 0 1 series and actually are kicking off a 2 0 1 for those who wanna see under the hood and what's running, that's coming really soon. But yeah, it's it's it's such a big topic. And and and I I wanna be careful not to say that that means it's so complex, you should be afraid. That's not what it's about.

What we're just trying to do is just help people understand. Actually, it's really well architected and it scales brilliantly. And so we just thought, you know what, we're so excited about this. We just... We wanna share what we've learned and we wanna get people over that 1st like, should I be scared here or are we good to move forward?

And so we we just love talking about it. You can't stop us. So every week, we're like out there chatting. And when we have a weekly podcast, we do it as well, which I think probably 50% of the time we're talking about substrate on that thing as well. So yeah.

We're we're we're getting out there as as best we can.

Yeah. I I watched the 1st couple episodes of of the 1 0 1 series and I'm starting to I'm starting to get the full picture now. But we'll we'll include a link to the playlist for anybody who's interested.

Awesome. Awesome.

How would you know if you are someone who's responsible for, you know, agentic stuff within your organization, you're managing some of the Kubernetes clusters today, how would you know that you're a good candidate for looking deeper into Agent Substrate? Like what what would the warning signs be that you're like, oh, I need something.

Honestly, anyone anyone trying to go to production and Yeah. They want to have like some control over what's going on. And and also just a a security con... Conscious person. Things that are running are vulnerable.

If you think about it that way, what if everything when it wasn't running disappeared? That actually... It was actually Michael... I'm stealing Michael's idea. He was the 1 who pointed this out to me and I was like, oh, wow.

I hadn't considered that. Yes. Everything being shut down except for the actual moment it's doing something for me is a really good security pattern to have as well. So honestly, I think literally anyone going to production should be thinking about this.

This is the joke. Right? The the the most secure computer is 1 that's just turned off except you actually did it.

Lowest barrier to entry. Right? Like there's... If it's off, you can't attack it in any way shape or form. And that's super crucial.

So like that's 1 piece of it. And then the other piece of it as well is like... And this is something that I struggled with in the beginning. Like originally, 2 years ago or so when the whole idea of like let me run an agent on Kubernetes, like I was like, I don't know. I think there might be a different system for this.

And then like as I went, I was like, well, what's the other system? Right? So like even today, I mean, in the majority of enterprises, who's not running something in Kubernetes in some way shape or form? Know, a lot of people are. In in whatever way, if you're using a managed Kubernetes service or if you're doing it on prem, like however you're doing it, there's always something running there.

So instead of going and learning something brand new, use what you have already. You might as well. And funny enough, it's also still the best place to run workloads. So there's no reason to not do it that way. Because what's the alternative, right?

The alternative is if you want to interact with an agent, you need a standard client server architecture. So the server is running the agent and it's exposing the endpoint and your client is whatever it is that you're using to interact with the agent. So like, do you want to do that for like all of your agents? You want like a bunch of these servers set up so to expose agents and then how are you gonna observe them? How are you gonna secure them?

Who's managing the code base? So in my opinion, there's virtually no reason to not go the substrate route.

Mhmm.

Yeah, that makes sense. Yeah. As soon as you realize, well, I'm probably gonna package these agents as containers. It's like, well, what would you use to orchestrate containers?

Right. Yeah. Yeah. Exactly. 100%.

And and 1 thing if it's okay, I I wanted to circle back to what Nathan said earlier where like, it's it's a lot but don't think about it as like, oh, wow, this is super complex. I need to learn this thing. Like this brings us back to I I feel like Ned, I feel like we've... We had this conversation like many many years ago where people were saying like how complex Kubernetes is. And the thing is is like, yeah, like distributed systems are complex.

But at the same time, then we got things like AKS and GKey and EKS and these managed services almost that sit on top and give you an easier way to manage that underlying platform. And that's what Cage and Enterprise is doing. Right? Like it's sitting on top and it's giving you a much easier way to go and interact with this platform. So you don't have to go and think about the lower level primitives.

Now, can if you want to, right? If you want to go and you want

to tinker, go ahead. Right? But you don't necessarily have to. Okay. So in terms of like going to try this out yourself, all of these different solutions, KA agents, agent substrate, agent gateway, these are all open source and free to use.

Right? Yep. But if

you need support, if you need, you know, someone to help you implement it, that's when, you know, someone might reach out to Solo and be like, hey, I need a little help here.

Yeah. So we're an open core company, right? So like everything that we're doing, we're we're doing it in the open source way. And like if you go and you look at the agent substrate maintainers, right, like the majority of folks are people that are working at Solo, right? So like they're contributing to open source, they're contributing upstream and downstream.

But where you want to think about enterprise is like, of course, we're gonna put goodies on top of it. Right? So like, you want the ability to have that more like managed style service or that like more of a user experience. You want the ability to see full observability data. You want OBO'd slash token exchange to be able to authenticate properly.

You wanna pass all that traffic through agent gateways. So, you know, your point a is sub strand cage and your point b is an LLM or an MCP server and you want rate limiting and policy enforcement and governance and all of that stuff around data sovereignty and just everything that you can think of from a security and observability standpoint, that's what you're getting from the enterprise solutions.

And and the market feedback is hard to ignore. It's it's like the whole kids in the car, are we there yet? We we have customers that are like, can we have it now? Can we have it now? When can we have it?

It's actually

pretty incredible. The the market like evidence of demand is... Yeah. It's just blowing us away. The the queue of people.

Like, when x? We want it.

Yeah. It's like every day I'm probably on a conversation in some way shape or form with like new and existing customers that are like, we need this substrate thing

Mhmm.

Which is awesome. You know, it's it's it's definitely exciting to be a part of something that like, you know is going to be the next next big thing. And it might already be the next big thing, like it's already here. And you can see it just from like, not just you, right? Like you're looking on social media, you're like, oh, wow, this is cool.

It's like the feedback that you're getting from the largest organizations in the world. I mean, it's it's it's it's fun stuff. Def definitely get involved with the project too. Like it's open source, right? Like there's weekly substrate community calls from Google, you know, there there's a bunch of different ways to get involved.

It's a lot of fun.

Awesome. Well, we'll include links to all of that good stuff in the show notes if people are interested. Nathan, Michael, thank you so much for being guests today on Day 2 DevOps. Thanks to Solo. For sponsoring this episode of Day 2 DevOps and virtual high fives to you dear listener for tuning in.

If you have suggestions for future shows, we'd love to hear about them. Hit either of us up on LinkedIn and you can send some feedback via packet pushers dot net slash follow-up. Either 1 is fine. You can find me, Ned Belovance at nedinthecloud dot com and my amazing cohost, Kyler Middleton, blogging over at let'sdodevops.com. And we're both entirely too active on LinkedIn.

Stop by and say hello.

Do you know that we also publish day 2 dev ops as a video podcast on YouTube? Well, you probably do if you're watching me say this right now. But for everyone else listening at home, you can see the faces that go with these disembodied voices over at Packet Pushers YouTube channel. Swing by and hit that subscribe button.

Bing. Until next time, just remember that doing DevOps is awesome and so are you.