Search
Follow me:
Listen on:

Day Two Cloud 146: Deploying And Managing Cloud Infrastructure With Pulumi

Episode 146

Play episode

Today’s Day Two Cloud digs into an Infrastructure as Code (IaC) platform called Pulumi. Pulumi can be used to build, deploy, and manage cloud applications using familiar languages such as .Net and Java, as well as well-known tools and practices.

Our guest is Matty Stratton, Staff Developer Advocate at Pulumi.

We discuss:

  • Pulumi’s view of Infrastructure as Software vs. Infrastructure as Code
  • Pulumi as an SDK
  • Languages Pulumi supports
  • How Pulumi tracks state
  • Paid vs. open-source versions of Pulumi
  • Use cases and examples
  • More

Sponsor: StrongDM

StrongDM is secure infrastructure access for the modern stack. StrongDM proxies connections between your infrastructure and Sysadmins, giving your IT team auditable, policy-driven, IaC-configurable access to whatever they need, wherever they are. Find out more at StrongDM.com/packetpushers.

Show Links:

Get Started With Pulumi

Pulumi Blog

Learn Pulumi

Pulumi Resources

Pulumi on YouTube

Pulumi on Twitch

@mattstratton – Matty Stratton on Twitter

Matty’s blog

Matty Stratton on LinkedIn

Transcript:

Please note this transcription is generated by an automated system and may contain errors.

[00:00:01.150] – Ethan
Sponsor StrongDM is secure infrastructure access for the modern stack. Strong DM proxies connections between your infrastructure and Sysadmins, giving your It team auditable, policy driven IEC configurable access to whatever they need, wherever they are. Find out more@strongdm.com PacketPushers

[00:00:27.870] – Ethan
Welcome to Day Two Cloud. We got an action packed show today, I think, don’t we, Ned? Here, we’re talking about Pulumi. So if you like Terraform or maybe Ansible or Chef and you’ve spent time in those worlds, Pulumi is also in that world. And we dive into what Pulumi is all about, don’t we, Ned?

[00:00:46.990] – Ned
We sure do.

[00:00:47.710] – Ned
And if you’re thinking, oh, this is another infrastructure as code conversation, well, it’s a little bit more than that because now we’re talking about infrastructure as software. So if that’s of interest to you, tune into this conversation with Mattie Stratton from Pulumi.

[00:01:00.250] – Ethan
Mattie Stratton, welcome to Day Two Cloud. And for the audience that doesn’t know you yet, who are you and what do you do?

[00:01:07.110] – Matt
Hi. That’s a great question. We’ll keep it tight. My name is Mattie Stratton. Currently I am a staff developer advocate at Pulumi. I’ve been in the DevOps space for a long time, and I’ve worked in Ops for an even longer time. Okay.

[00:01:20.520] – Ethan
So working at Pulumi, and that is the focus of the show. So in a sentence or two, give us the what is Pulumi answer.

[00:01:28.050] – Matt
So I always like, you’ve probably heard of infrastructure as code. So Pulumi can be considered an infrastructure as code tool. We call it a universal infrastructure as code. I like to think about it as infrastructure AKS software and really use Pulumi to build and deploy and manage your modern cloud applications. But the big kicker is you’re using familiar languages, tools and practices, right. That you already know. So in a very short sentence, it’s like it’s Infrastructure’s code using something like. Net or Java or Go instead of a domain specific language. There’s more to it than that. But that’s sort of the overview, right.

[00:02:04.220] – Ned
So you brought up a new term, which is a software, and we like terms. We like defining them. And since you came up with It, or at least for the purposes of this podcast, you did, would you like to define infrastructure as software for us?

[00:02:17.470] – Matt
I would love to. I always like to say words are hard. Don’t worry, I’ve got new words for you so we can argue about them. Or as Andrew Clay Schaefer likes to say, who wants to fight about made up words? The definition of made up words with me. So when I think about infrastructure stuff, I’ve been kicking around this space for a while, and I was a user of Puppet. I was in the Chef community for a long time. Then I went and I actually worked at Chef. I’ve tried pretty much almost every different way you can do infrastructure as code. And the thing that’s interesting to me is quite a bit of what we call infrastructure as code is number one. It’s not treating our infrastructure as code at all. It might be creating our infrastructure using some code, and that might be a pedantic difference. But I’ve always sort of thought about versioning our infrastructure, like all those things, like applying it to our infrastructure. So when we talk about infrastructure as software, because code is only part of software. Right. So if we’re creating software, yes, we write some code, but we test it, we put it through life cycles.

[00:03:15.630] – Matt
We have APIs and integrations and all these other things and libraries and components that all come together. So when I think about infrastructure and software, it’s like, how do we actually lay this down and use software engineering practices towards the deployment, the managing the building of our infrastructure resources?

[00:03:36.510] – Ethan
Okay, I think I understand the pedantic difference that you’re making. Infrastructure as software in that context makes a lot of sense because infrastructure AKS code fails as a definition and that infrastructure isn’t code. We’re using code to manage it. So I think I get where you’re going. When you were defining what Pulumi is. Matt, you made the point. That the big kicker. The big thing here is you’re using the language that you already know, some kind of a traditional programming language.

[00:04:03.830] – Matt
Okay.

[00:04:04.300] – Ethan
So how is Pulumi different from an SDK, then?

[00:04:07.110] – Matt
Well, it is fundamentally, at the end of the day, that is exactly what it is. It’s an SDK, right? It is some type of a connection. So when we think about my colleague Lee Briggs talks about this a lot about that. When you think about an infrared tool and you will hear me say, I will continually refer to this as an infrared tool, because we do know that term. And I’m definitely not here to be like, let’s change the words we use. I do think words influence how we think about things. And there was a reason that when I worked at Chef, I didn’t like calling them Chef scripts. I called them Chef recipes because a script is an ordered there’s a thing to it anyway. But I don’t care what you call a damn thing. Just do it right now. That said, when we think about an infra code tool, there’s the authoring experience and the execution experience. Right. So how do we create our definition of the infrastructure that we want? And then how is that thing that I authored, executed and made to be true? Because in a lot of ways, if we think about it in the code is very simple.

[00:05:04.630] – Matt
At the end of the day, that’s how I always talk about it. I’ve been talking about this for forever, and people Azure like, it’s complicated. It’s hard. I’m like, you know what it is? It is a test and repair loop. Is the thing look like this? Yes. Cool. Do nothing. Does it not look like this? Fix it and move on with your life. Right. That’s all in for code is. Okay, now we got 40 more minutes to talk about, I don’t know, the Cubs or something, which, by the way, terrible, actually. Designated hitter in the National League, the worst. So when we think about Pulumi as an SDK, fundamentally what we have here is we have SDKs for all of those languages that translate into the declarative state. So there’s a lot of things that people in this space like to argue about. Right? Again, argue about the definition of made up words or real words that we apply a made up definition to get a lot of declarative versus imperative. We used to like to talk about the word item potent a lot. Like we are mathematicians. Any mathematician listens to config management nerds to talk about item potency.

[00:06:04.410] – Matt
And they’re like, but that’s not actually what it means. Good job for messing that up. But the thing is, a lot of times people say, oh, well, Pulumi is not declarative because you’re writing it using an imperative programming language. But what happens is and the reason I bring this up is not to get into this debate, but to think about that authoring versus execution. So what we’re doing is using various SDKs in the different programming languages. You’re writing code, and that’s generating basically a graph, it’s generating a plan, it’s generating a desired state in a declarative way. Okay. So when we think about when you say how is it different than SDK? I guess to a certain point, my question is, what do you mean by an SDK? Do you mean like the cloud providers APIs that you might consume directly? Is it because there is, for example, there’s the AWS CDK, the Cloud Developer Kit, which is very similar. So fundamentally, they’re all thinking around similar ways. And the thing is underneath the hood, that is 100% how this works. So again, it’s an SDK to talk to APIs, right? Right.

[00:07:14.840] – Ned
And you’ve placed some levels of abstraction between directly talking to that API with the rest calls versus the declarative state. And there needs to be something imperative in there that’s running that loop to talk to the APIs, get information, check that information. If it’s wrong, set it to something else. And now we’re taking it one level deeper. You have an imperative thing that you’re writing that needs to get translated into a declarative thing. Some of the other solutions out there kind of skip that portion and they just go straight for declarative. I’m going to declare whether it’s in YAML or some other DSL, but there’s definitely some deficiencies with that approach as well. Was Pulumi designed as reaction to those existing DSL solutions?

[00:07:57.330] – Matt
I think there’s a couple of things. So when I think about it, I want to give a little bit of history and I’m going to get some of it wrong. But hopefully, maybe Joe Duffy, the creator of Pulumi, is listening and will correct me later so I was around from the very beginning of Pulumi. But I’m going to answer your question when we give a little bit of a background. So it took me about a year to come to Pulumi. So when I first was interested in coming to Pulumi, other opportunities, things happen. Then I came around a year later. But one of the things that was really important, when I was first starting to talk to Plum, I talked to a lot of people in the space that I know and asked their opinion. And what did they think? One person who’s very well known in the config management space, I won’t identify who it was said, look who’s very familiar with what Pulumi was trying to build is they said to me, look, Mattie, you need to find out what the big bet is, because right now it looks like all this is trying to be a better Terraform.

[00:08:50.380] – Matt
Is there room for a better Terraform? Absolutely. But is that what you want to be involved with? And he said, look, I know Joe. I know all the people involved. I know there’s more to this. I just don’t know what it is. So when I came around the second time is when I was talking to Joe Duffy, who’s Pulumi’s founder, and this was right before what we call our Automation API was launched. And he was telling me about that and I said, AHA, this is what makes sense. I think the idea has always been not just about writing for code using a general purpose programming language, just because that’s just like just layering programming language over Terraform or over whatever kind of thing. But how do we actually and Automation API was the beginnings of me saying, this is treating our infrastructure as code and what that is without going too deep into it, because we could talk for 45 minutes about Automation API is imagine you’re writing an application in like a Flask application in Python, or you’re writing a web portal or a CLI in Go, and you embed your infra code ability directly in the application.

[00:09:59.850] – Matt
Right. So it’s not like it goes out and calls does a Pulumi up or does whatever. You actually are able to interface with Pulumi itself as an API inside your software, which that’s the real interesting stuff to me. And we’re seeing people are building their infrastructure, automation platforms inside their organizations using polymer as the core engine of that. But the people who are leveraging it don’t actually have to know Pulumi. I will put it to you this way. A lot of the stuff to me when I look at Pulumi is I always say, this is what I always wanted Chef to be. So if you don’t know Chef, Chef’s big thing was again, we can fight all day about what’s a DSL? Yes, it was a DSL. But you Azure. Basically, when you’re writing Chef code, you’re writing Ruby. So you have all the everything I’m telling you that Pulumi does, that was Chef, except only Ruby. So it was great as long as Ruby was what you wanted. But you could do things like write conditionals and loops and leverage Ruby’s ecosystem and testing tools and Ides and all that great stuff. But what people always wanted, all my customers were always looking for some way to bring this into a bigger story.

[00:11:10.920] – Matt
Because when you’re deploying, when you’re building, when you’re creating your infrastructure, it’s not just one little thing, it’s all part of an ecosystem. Right. I tell you how many customers at Chef they were like, okay, cool. So I wanted to do this, but it also needs to register the server and service now and update it over here and add it to the page of duty rotation and do this thing. And so we ended up writing a lot of Janky bash to make that work. But if you can have that all happen inside an API to me. So I think, again, a lot of it is saying, okay, how do we apply software engineering tools and practices to infrastructure, which sounds an awful lot like DevOps. That’s like what we’ve been talking about for twelve years. This is why it makes a lot of sense to me.

[00:11:54.630] – Ethan
One of the points you made when you were critiquing Chef from back in the day was, well, it was fine as long as you wanted to use Ruby.

[00:12:01.720] – Matt
Well, right.

[00:12:02.350] – Ethan
Pulumi doesn’t have that restriction. So what languages does Pulumi support?

[00:12:05.810] – Matt
And I’m curious to know everything but Ruby, to be honest, which is kind of funny to me because I’m like Ruby is the thing I do. I don’t even know the number now because one of the big things is last week at our latest update, we announced support for Java. And the reason why I say I don’t know the number is Java unlocks every language that uses the JVM, Kotlin, all this stuff. So I should know the number. Off the top of my head, I’ll give you the broad strokes. So you got JavaScript and TypeScript Python, goallof. Net. So you want to use C Sharp, you want to use F Sharp, you want to use BB. Net, Java. And then also and this was a big thing is also YAML, which is a little potentially controversial because I’ve been saying a lot of stuff about this is better than YAML, but there’s reasons it’s really important to have. And I’ll talk about that in a minute. When you talk about the building blocks of how you would use Pulumi, why YAML matters.

[00:13:06.570] – Ned
Yeah, it’s one of those contentious ones where it was because we Azure not going to lie.

[00:13:11.340] – Matt
We probably maybe not officially, but they were still finding a lot of stuff in the marketing that’s like, oh, where we say this is better than using YAML and now we’re using it, but we’re not using YAML by itself. So I guess the little piece I’ll give you into this little core idea. One of the powerful things about using a programming language is you can create what we call components, but basically they’re libraries. And think about this. Let’s say you are in your part of your organization, of your company. You’re the subject matter expert on Kate’s, on Kubernetes. When we do Kubernetes, we do all this. And then you’ve got other people in the organization that are like, All I need is a Kubernetes deployment. And frankly, I only care about two things about that deployment. And there’s really 15 different things that point inside of Kubernetes would have. So I can sit in whatever language I write in Go or Python or C Sharp or Java. I write a component which can be consumed by any of the languages Pulumi supports, including YAML. So you can use YAML to sort of construct your building blocks where you’re like, okay, look, I just have a couple of things.

[00:14:22.090] – Matt
Now, you’re going to lose a lot of the power of the programming language, like conditionals loops, abstractions, all this. But you’re like, I don’t actually care about that. I just want to take the thing that Ned wrote and it gives me two parameters, and I just want to put it in here so you can do that. You don’t have to use YAML for that. You can use any of the languages. But YAML makes it even easier, right?

[00:14:44.410] – Ethan
Thank you for explaining that. Because when you said YAML, I’m like, wait, one of these is not like the others. Yaml is configuration information, key value pairs, if you will. It’s not a programming language. What are you talking about? But I think I get you now. So, Manny, another question, though. All those languages that you mentioned, if I’m not proficient in one of those is Pulumi, is that a tool I should be looking into?

[00:15:09.750] – Matt
I think so, for a couple of reasons, because there’s different ways you would approach Pulumi. One is that, hey, I’m a software engineer that’s always already proficient in Go or TypeScript, and I’m just learning Cloud. Well, okay, now at least I only have to learn one thing, which is the cloud part of this. I don’t have to go learn HCL or some other technology or tool. The other thing. And this is something I’ve been telling folks. And again, I’ve been doing Chef for a long time, and I had a lot of folks who like, I don’t know, Ruby, I don’t know how to program. I’m an Ops person. So two things that I would always say is, number one, one of my little tropes when I worked at Chef was I’m the world’s worst Rubius. But I can write some Dang good Chef code. But we used to talk about a Chef we had actually part of our workshop was called Just Enough Ruby for Chef. And I’m actually working on some similar things with Pulumi. It’s going to take a lot longer because it’s just enough X times, however many programming languages. But what I’ll tell our audience who don’t consider themselves programmers, and I’m one of those people, I alluded to it earlier.

[00:16:12.730] – Matt
I spent two decades working in Ops. I still don’t consider myself a programmer. I can write some code, but so many things that we do in Ops are just like coding. We just have different words for them. Right? We actually get this stuff and the level of coding, like the expertise you would have to have in Python, for example, and actually quite a few obstacles already know Python decently, and even if you don’t, you Colo probably figure it out. I don’t really like Python a lot. Luckily, everybody else on my team, my other Das, like, they pick it up. And I’m like, this makes no sense to me, but it’s okay, I’ll write it and go. But usually the level of what you’re trying to do when you’re getting started is just again, simple variables conditionals, passing objects around. This is all stuff we actually know from other tools we use and other ways to approach it. So it seems like a lot, but it’s really not. So the reason to your point about it is that if I don’t know that, if any of the language is it worth it, I would look at it.

[00:17:17.990] – Matt
It also depends upon what you’re already doing, because there’s two different approaches. There’s one that I’m doing nothing. Right. I always say our biggest competitor is Do Nothing Incorporated, right? We don’t have any info code. We just sort of do. That’s cool. But if you’re already like, hey, we have all this Terraform, we have all this stuff. You probably already know where your pain is around that, and it might not. Here’s the thing too. Again, people like, should I use Pulumi or Terraform? My answer is yes, do something. And I think that’s where that matters. But I would definitely say, don’t feel that the language barrier, so to speak, is insurmountable, because also, I’ll tell you something, you’re an Ops person that hasn’t done any coding. Adding a little Python to put a Python arrow in your quiver not a bad idea. Yeah.

[00:18:12.560] – Ned
And I would say most of the apps people I’ve ever met have either done a decent amount of batch scripting or PowerShell scripting or even batch scripting if we’re going back far enough. They’re totally unfamiliar with the concept of writing a script. They just haven’t gone into that more formal environment of a general purpose programming language. And I think they’ll discover it’s not that big of a lift once they actually get there. That’s certainly been my experience.

[00:18:40.390] – Matt
And the type of coding we’re writing is not that far off from a PowerShell script or a Bash script or whatnot. Again, yes, you can go to create these components and all these other abstractions and things like that, but it’s part of the glide path. Okay. All right.

[00:18:59.330] – Ned
Now, my understanding is that Pulumi itself has an open source version that’s free to use. Folks can jump in and start getting to learn it. But I assume there’s also a paid version because your company wants to make money.

[00:19:12.940] – Matt
They got to pay me somehow. Yeah.

[00:19:15.790] – Ned
Can you just point out quickly what’s the differentiation there? What’s included in one versus the other?

[00:19:21.350] – Matt
Absolutely. So fundamentally, it comes down to state. Okay. Which I know we were going to talk about a little bit later, but we’ll jump into this. We talked about this somewhat. We talked about infrared being a test repair loop. Okay. Well, for that to work, your tool, whatever it is, has to know the state of your infrastructure. That state has to be stored somewhere. That is the one true problem of infra code. Right. Everything else is just stressing on that. So with Pulumi, and like you said, from a product, from the Pulumi client, if you will, all that stuff is all open source, free use as much as you want. It’s what we call a back end. So the back end is where your state is stored. So we support a whole bunch of different back ends, only one of which, I guess two costs you money. So you could store it in an S three bucket. You could store it in Blob storage, you could store it in local storage. That stuff is all free and is missing some features as that’s how that goes, because it’s just a big state file. What we have is we have what we call the polymer service.

[00:20:23.280] – Matt
And so that is our SaaS basically polling a back end as a service that gets you a lot more features. And even that is free. It’s free for individuals. So if it’s just you, you’re just starting to do some stuff, it’s totally cool for you. Then you kind of go into team and enterprise and business critical and just a couple of things. Again, we’ll drop a link in the show notes to the pricing page. You’ll see that you don’t need to have me read it all out to you. But it’s a couple of things. Like how many people have access to it. Right. Our back is always the thing that you charge for. Right. Which is role based aspect, which is sort of say like, okay, so and so needs to be able to do this thing. And then you start to add up. A lot of it has to do with how many concurrent updates can happen at any particular time. And when you look at our business critical, so again, you’re paying for just a lot of it is I don’t say paying for support, but paying for dedicated support. But for when you’re getting started out, go ahead and it’s not impossible or even, I think, terribly hard to switch back ends.

[00:21:35.360] – Matt
So we always tell people when they’re getting started to just start our workshops always use our service because it’s the easiest. Some cynics would say that’s me trying to trap you into the features and then you don’t want to leave. But frankly, it’s me being lazy because it’s just easier and it exists. But even a lot of the stuff that even for people who are listening that might have been looking at Pulumi even a year ago would be like, oh yeah, I looked at that. But the S three back end didn’t support locking. And so a lot of that has been updated and we’ve also made it. We’re trying to make it as easy as possible for people to see there are more back ends because we’ve sort of gotten that before where they’re like that’s. One of the biggest questions I get to be honest is, well, I don’t want to use your staff, which could either be because you don’t want to pay us or you have a regulatory reason not to use it. And we’re like, we have all these different things that people are like, I have no idea. And I just happened to notice it’s actually on our pricing page now.

[00:22:32.080] – Matt
So with that, if you want to use it now, you can use the Pulumi service inside your file. Like you can self host it, but that is paid. So the Pulumi service itself is not free in any way except for individuals. But if you’re like, hey, I love the Pulumi service, but I have a regulatory reason to have to put it inside my network.

[00:22:52.590] – Ethan
Matty, I want to start getting more specific. So we’ve been talking conceptually high level what is Pulumi? Okay, so let’s start this way. Walk us through a simple scenario of using polymer to deploy Infrastructure’s code. And this is day two cloud. So maybe pick AWS or Azure or something. And how I’d use Pulumi to stand something up there.

[00:23:10.450] – Matt
Absolutely. So I will even skip over. One of my biggest annoyances is what I call the S three bucket example, which is how do I do a thing to configure an S three bucket and raise your hand if your entire job you’ve ever had to just configure an S three bucket. And that is all the reason it’s a good example is it’s something everybody understands. But if I want to think a little bit more about what is applicable. So let’s say I’m going to go ahead and maybe what I need to do is deploy a VM. Let’s say I want to spin up a machine in EC two, but it’s going to be on its own VPC. Okay, let’s do it that way. Because I think that’s a good idea. Because only having one thing makes us less interesting and also lets us lose a little bit of the power of this being a programming language. Okay, so what I’m going to do there and I will skim a little fast on certain things. You’re not watching me type. And we’re going to pretend that I’m going to use TypeScript just as an example. And it only matters because I’m going to use the word TypeScript once.

[00:24:10.430] – Matt
So what I do is I sit down and I say like, okay, I’m going to write new TypeScript AWS. And what that does is because a Pulumi program is just a whole bunch of files in a directory. And if you want and this is what you’re into, you could go create all those files by yourself. But that sure seems boring. So we have a whole bunch of templates off the pole me thing that will go and scaffold that. So the reason I bring this up is so the Pulumi TypeScript. And then specifically AWS, is it’s going to go ahead and say like, okay, this is a TypeScript program. So that means I need a package. Json, I use an index TS. Aks my entry point and it kind of builds out all the initial files. If I did go, it would give me a main go. If I did a Python, it would give me main Pi and the requirements. Txt. So it’s going to do that for me. So I get started. Cool. And now I’m going to go in there and I’m going to say, okay, I’m going to create a new resource and I’m sort of pseudo coding and audio.

[00:25:07.930] – Matt
But I’m basically going to say first I need to create a VPC. Let’s say this is dedicated. I know in real life you probably already have one you’re attaching it to, but we’ll just do this. Right? So I’m going to basically say it’s a new VPC and I’m going to give it the various different arguments or properties that it might have. And again, there’s a lot of stuff that when you’re creating a VPC and need a bunch of other stuff, we’ll skip that for a minute. Now I’m going to go ahead and say I’m creating in the next resource that I create, I’m going to define it as an EC. Two, I’m going to say I’m creating a new machine. Now one of the things is I tell it what VPC to attach to. Well, guess what? That’s a property off of the object I created, which was that first VPC. So I can reference it by, let’s say I called it VPC, VPC name. Boom. I’m passing those things across from each other. So this is the programming language Power, right? Is that I’m like, okay, I created a thing over here. I can reference it somewhere else.

[00:26:04.080] – Matt
It’s a really simple idea. And then let’s say I just do that piece. Then what I would do is I would run a command called Pulumi. Up. And first it’s going to basically just go in. It does a very quick check of my code. Like if there’s any compilation error or basically syntax errors, it will flag and say, wait, this is Mattie, you wrote some credit JavaScript fix this lie and I can’t figure it out, assuming that it doesn’t mean it’s going to do what you need it to do, but at least it will not trigger it’s going to sit and say like, okay, cool, if you run this, it’s running a preview. It’s going to say this is what’s up. I’m going to create a stack, which in Pulumi version, a stack is an instance of your infrastructure. A lot of times they might correspond to Dev or QA or Prod, but they’re really just an implementation. So it’s the same Pulumi program, but maybe with different configurations. Okay. It’s going to say we have to create a stack and we have to create it says I got to create a VPC because I don’t have a VPC like this and I got to create the VM.

[00:27:01.350] – Matt
And then it says, do you want to want to do that? And then I would say yes, and then it’ll go ahead and spin up and do all of that stuff. Very similar to a lot of other infrared tools where you run them at that point. I can do this inside of a CI CD pipeline if I want, et cetera. Now, what will happen is assuming that that runs and everything’s fine. If I run Pulumi up again, it will basically say nothing has to change. Yeah, that’s the fundamental, simple scenario of that. And again, now we talked about, let’s say you may already have VPCs, so you can do input and output from various stacks. So let’s say I had a different Pulumi program that configured all my networking in AWS. I can actually slurp that value in from that other stack in my Pulumi program that does my EC two just like that one. Again, that’s a lot to explain in mouthwords without seeing, but it’s really important to have those input outputs that go between stacks and not only in your program, because that’s a lot of times how things work. Right. Or I’m going to spin up a Kubernetes cluster in my core infrastac.

[00:28:05.630] – Matt
But my deployment just needs to know the info about that Kubernetes cluster. Right.

[00:28:11.590] – Ned
And it’s slurping that from the state that’s being stored.

[00:28:15.360] – Matt
Absolutely. Okay.

[00:28:17.170] – Ned
That gets us to state, which you already talked about a little bit. But what’s actually stored in that state data that Pulumi is using?

[00:28:25.610] – Matt
Well, at the end of the day, it’s all just JSON. Right. So it’s a big JSON blob that’s like here’s all of the resources and all of the parameters around them. Now, I know we wanted to talk about secrets later, so we’ll just put a pause in it that there may be information in there that is secret and that has been encrypted in various ways we’ll talk about when we get to secrets, but examples. And one of the things I will tell you and this is sort of the cool thing about doing cloud related things. Again, to go back to Chef. I used to get people would always say how come I can’t point Chef at my Apache server and have it spit out a cookbook to build that server? And I’m like, because Chef doesn’t know one of the 1015 thousand files on that serverless you changed or matter to you or have anything to do with anything. Well, you can do that with Pulumi and your AWS infrastructure because it’s all APIs that have defined endpoints that have defined values. So you can actually import that and it just is translating that state.

[00:29:25.740] – Matt
So fundamentally state is a bunch of key value pair that says like here’s the properties of all the resources and then that’s maintained historically as well.

[00:29:35.260] – Ethan
Got you. We pause the podcast for a couple of minutes to introduce sponsors strongDM’s secure infrastructure access platform. And if those words are meaningless, Strong DM goes like this. You know how managing servers, network gear, cloud VPC databases and so on. It’s this horrifying mix of credentials that you saved in putty and in super secure spreadsheets and SSH keys on thumb drives. And that one Doc in SharePoint. You can never remember where it is. It sucks, right? Strong DM makes all that nasty mess go away. Install the client on your workstation and authenticate policy syncs and you get a list of infrastructure that you can hit when you fire up a session. The client tunnels to the Strong DM gateway and the gateway is the middleman. It’s a proxy architecture. So the client hits the gateway and the gateway hits the stuff you’re trying to manage. But it’s not just a simple proxy, it is a secure gateway. The StrongDM admin configures the gateway to control what resources users can access. The gateway also observes the connections and logs who is doing what, database queries and cue cuddle commands, et cetera. And that should make all the security folks happy.

[00:30:41.290] – Ethan
Life with StrongDM means you can reduce the volume of credentials you are tracking. If you’re the human managing everyone’s infrastructure access, you get better control over the infrastructure management plane. You can simplify firewall policy. You can centrally revoke someone’s access to everything they had access to with just a click. Strongdm invites you to 100% doubt this ad and go sign up for a no BS demo. Do that@strongdm.com packet pushers they suggested we say no BS and if you review their website, that is kind of their whole attitude. They solve a problem you have and they want you to demo their solution and prove to yourself it will work. Strongdm.com PacketPushers and join other companies like Peloton, Sulfi, Yext and Chime. Strongdm.com packet pushers. And now back to the podcast. You just mentioned the AWS APIs that you’ve got endpoints. You can find information. That’s exactly what you need to know. I think part of that translation magic is happening because Pulumi leverages Terraform providers. If I did my homework right.

[00:31:52.190] – Matt
That’S another one where people get tripped up. So we don’t leverage Terraform providers. What we do leverage is so to create a provider, which is something like maybe it’s an AWS provider, maybe it’s pager duty, maybe it’s whatever you need to know. The schema, right. The schema of what are the crud operations that can be done against that provider and what are their definitions? As it happens, the wonderful community around Terraform has defined a lot of this already, and they are open specs. So what you can do is you being us, a Pulumi or even you as an individual. If we don’t have a provider for a thing you want, where there is a Terraform provider, you can generate one, but it’s just generated off the spec. No Terraform code is actually run, right. So when you go and convert that. So if you look at our pager duty provider, for example, that’s just taking the API spec, which has been defined in the Terraform provider, but it’s all Pulumi code. There’s no actual execution of Terraform. Now, that said, you’re alluding to something, which is really cool. We have a couple of different kinds of providers.

[00:33:00.750] – Matt
Those are, we call those bridge providers where we take the spec that comes from a third party like Terraform and do that. We also have what we call native providers, and those are built from the open IPI spec of the cloud provider itself. And the most mature AKS, and it’s been around the longest is Azure. So, for example, when Azure ads, when they add a new feature to Azure that is in the Pulumi provider the next day, wow. Because we build that provider nightly based on the open API spec. Now, that requires now, of course, Azure doesn’t necessarily publish everything through that spec. So things we know about that aren’t there. We can put in manually. We have just added the GCP one is in preview the native provider and the AWS one. The thing with the AWS one is that exists, and it’s using the cloud control API, which Amazon themselves have not yet fully fleshed out. So the AWS native provider doesn’t have everything. The fun thing is you can mix and match providers. So if you’re building on AWS, you absolutely can use what we call AWS classic use AWS native for defining the things that are in there because you know, that’s the future.

[00:34:15.910] – Matt
But if you’re like, okay, you know what, this particular service is not in the native one. I can pull it up in that one versus like if it’s a bridge provider, it takes a little slower than that. But that gives the ability to say, hey, we don’t have to go recreate the wheel on all this stuff. This already exists, especially if you’re someone who has written a Terraform provider for your service, that you work at page duty and you own the Terraform provider, you can very easily just make the pole provider for that. You don’t have to because it’s already there. But as an example.

[00:34:50.330] – Ethan
So Mattie, on the polymer website, there is some phrasing there. It’s a cloud engineering platform, and I think you’ve just highlighted why you could go that way. The APIs are there, they’re open. They’re very well documented, you know, everything that you can do with them. So what happens if I’m managing that icky metal in my data center? Can I still use Pulumi? Assuming, let’s say the device like a network switch. Let’s say, let’s say it’s got a decent API. Could I use Pulumi?

[00:35:18.610] – Matt
Yeah. I mean, that’s really what it comes down to is having an API that you can build the SDK around. So that’s why Pulumi doesn’t work as inside the operating system config management like you would use for Chef or something. So it’s always like Ansible is a good answer for it. Oh my God, I can’t believe I just said use Ansible. But for example, inside that or Chef or Puppet or something that’s really good at managing inside the OS can be a better together story. Right. So when you’re bringing up your AMI or whatever, or I like to call it Amy, don’t tell Cork. You can then have part of that, what we have what we call as a command provider. So you can actually run commands in your Blue Me program as well against things. So that could work. That could either be maybe you’re running Chef client or so in your example of like, hey, I’ve got maybe a network switch or something like that. There Azure ways around it. Either it’s maybe the command will get you as far as you need to go because maybe you’re like, okay, I can get almost all of it with an API, but then there’s just one thing that I have to run as a command.

[00:36:25.310] – Matt
Right.

[00:36:25.710] – Ned
So even if there is no direct provider for that device, there’s still a way to continue using Pulumi to manage infrastructure as long as it’s not too complicated.

[00:36:34.760] – Matt
Right. And then you might do that with again, the same thing. Maybe you’ve got a really good Ansible Surface for that particular device, but everything else works well in Pulumi. You can connect those things to get that, whether it’s that last mile or whatever.

[00:36:48.720] – Ethan
So I don’t have to have a provider as such to interact with that the oddball device.

[00:36:56.110] – Matt
Well, you could use the command provider. Well, you’re probably going to end up wrapping it into some kind of a component so that it makes sense to you. Because again, most of the people who are going to consume this on your end are going to want to be able to pass in just the one parameter to that. But yes, you can kind of bring that command provider as part of this would be fine. If you want to mess with this sometime, we can do terrible things. My friend Lee wrote a blog. I’ll give you the link and I’m going to misremember the name exactly. But it’s like deploying Kubernetes and ridiculous programming languages, but actually under the hood because of the YAML provider, because any programming language that can spit out JSON, we can turn into Puluming at this point. And so he wrote a bunch of stuff about spinning up like Kubernetes clusters in Fortran and stuff like that. There’s a lot of inadvisable things you can do, and those are always the most fun.

[00:37:57.430] – Ethan
But how hard is it to actually write a provider? Is that something that really pure developers need to do?

[00:38:03.700] – Matt
It depends. As always. One of the things that’s really great is that our providers, again, if you’re talking about writing like a native provider, that’s going to go and be slurping in from the API specs and keep rebuilding it, that’s different. That’s harder. So the bridge providers are very easy. But a component provider, we call it a Pulumi package. These are actually all packages. But the example I was giving, where you’re going to write a thing that’s exposed to other folks is actually not that terribly hard. And one of the cool things is you can write it in any of the Premier supported languages, but it will generate the SDKs for every language. Which was a new thing in Polum three auto. That was last year because it used to be that if you want it, you had to write each of those SDKs by themselves, which meant you’d be like, all right, well, we’re a TypeScript shop. Like, that’s what I know. But then someone else in your company wants to use your package and they want to do it and go, they can’t. Now it’s just part of the authoring experience to just generate the SDKs and all the languages.

[00:39:07.380] – Matt
And along those lines, there’s stuff where you could even again, that YAML provider actually really helps you with some of this. You can really get there. So it’s gotten a lot easier. I’ll be honest with you, I need to spend some time with that because I work here and I’m like, there was so much stuff that just happened and I’m trying to keep up. So I know that more in theory, but yeah, I think it’s gotten a lot easier. And we’re continuing to make that experience easier and better because that’s the key. Right.

[00:39:36.450] – Ethan
Talk to us about secrets management, Mattie, that we alluded to earlier. But it’s kind of an important piece here.

[00:39:42.090] – Matt
They’re very important because one of the things secrets happen, right. You might need to pass in an API key. You might need to PaaS in a database password or something. So the first kind of just overall important thing is, yes, Pulumi supports secrets, as in you can store secrets in your Pulumi code by default. It wants to use the Pulumi service as a secret provider. However, you can use I don’t want to say any secret provider, but pretty much anyone that you would likely. Well, I shouldn’t say this, but yes, you could use balls, you could use various different ones. You just configure and say that. But the reason why this comes up of how it works is when you Mark a value, whether it’s an input or an output or a config value that you’re using, you and your code say this is a secret, and so that value will never show up in a log, it will never show up in the console, it will always be redacted and it will be encrypted when it’s stored in their service back end, but it will also be encrypted if it’s in your configuration. So I didn’t really go into config too much when we talked about stacks.

[00:40:54.170] – Matt
Let’s say I’ve got a Dev in a QA and a prime. Okay. And let’s say just for example, one of the things that would might be different between all of them is the AWS region. So I would set a config value that says region equals whatever, blah, blah, blah. And then I would set that different for each one. So my Pulumi code is basically saying config require it’s, saying require a config called region as it happens to AWS. Provider does require this, but you have to set it. But I could also say require DB password, because again, probably the database password or the API key might be different in different stacks. But I’m going to say require secrets. So someone has to set that in the config file, but it will be encrypted in the config file. So when you look at the config file, you’re just going to see the encrypted value of that and it will be able to be decrypted by the Pulumi service, but you will never be able to see it and it will never show it. You know what I mean? Like when you do your point here being.

[00:41:47.360] – Ethan
Yeah, if I need to sync all of my Pulumi files to version control, GitHub or whatever, I didn’t give anything away.

[00:41:55.370] – Matt
I mean, as long as you remember to say they were secrets, we can only do so much. Right. Okay.

[00:42:04.510] – Ned
Now I want to bring things way back to almost the beginning of our conversation where you said infrastructure software.

[00:42:10.320] – Matt
Right.

[00:42:10.570] – Ned
Yeah, we’re talking time.

[00:42:12.040] – Matt
All right.

[00:42:13.070] – Ned
And we talked about how there’s a lot more to software than just writing the code. And one of those big things, one of the things that you absolutely should be doing is testing your code, whatever that means, whether it’s unit tests, whether it’s regression tests, whatever it’s going to be. How does Pulumi test code?

[00:42:32.910] – Matt
So this is one of those beautiful things where it’s like the hey, it’s software, man. How do you test TypeScript? How do you test Python? Just test it. That is the short answer is yes. Hey, use the test, especially when you’re talking about things like unit tests. Right. Which are arguably this is my old school infra coder hat on the least important tests because you only want to test them if you’re testing something you did. So I’ll give you an example. A lot of times when we write unit tests and our infra code, we find ourselves testing that the infrared tool itself actually worked, which you don’t need to do because we do that for you. So again, I’m going to give it a Chef example because it’s the simplest is there is a resource and Chef called Package. So let’s say I wanted to install NGINX. So I have one line of Chef code that was packaged and then quote Engine X. That’s how you install NGINX in Chef. And then someone would write a unit test that would be like did the package NGINX get installed? Why did I have to test that right now?

[00:43:44.080] – Matt
But what if we’re talking about Apache where the package name is different depending upon the version or the distribution you’re using, then I have introduced some logic. So I need to test that if distro as Ubuntu with the package name this. So when you think about your unit test, you need to be making sure you are testing your logic. I was like you don’t go and test to make sure that networks, right? I don’t know, maybe you do. People have their own things. I’m not here to make judgments. Now that said, what you basically are testing in a unit test with Pulumi Azure, your resource inputs. You’re saying like the input that I was giving, which might be so a lot of times you’re just sort of saying like okay, if whatever I’m passing into it. Like again the equivalent of did I pass the right package name, which you would not do in Pulumi? But did I pass like a subnet the way that I would expect you into a network or something like that, especially if I’m doing some kind of logic versus just reading it from a config value. Why did you need to test that then?

[00:44:48.860] – Matt
We talk about what we call property test. So these are resource level assertions like against while the infrastructure is being deployed. So those you actually sort of have to deploy. But again the difference is unit tests are super fast, they’re in memory, everything’s. So they take milliseconds. A property test still takes seconds because it’s all mocked. And if you mock it, right, you don’t actually deploy your infrastructure. It’s mocked out to say like it mocks out the AWS API and says okay, this is what would happen, right? Because we understand that. Then you talk about integration tests, which are you actually deploy some Ephemeral infrastructure and then run your external tests against that, right? So those take minutes, they take a little or maybe longer depending on what you’re spinning up. But you actually are testing against external endpoints and you really need to kind of do all of them, but you put them at different phases through your testing. But yeah. So if you like a lot of our examples, you can write unit tests in Python and TypeScript. Maybe you’re using Mocha as your framework. And we have a lot of really great examples around that because we do it, we have to test all of it.

[00:46:01.200] – Matt
Right, right. But the beautiful thing is, with the exception of the integration testing, where we actually have good examples around it, there’s no special Pulumi thing you’re testing, especially for unit test. It’s just your programming language, man.

[00:46:15.650] – Ned
Right now, I’m assuming I’m not doing everything from the ground up here. I don’t need to write everything. I can leverage the work of Ups.

[00:46:26.140] – Matt
Let’S say I hope so.

[00:46:27.870] – Ned
Is there a special site or a place you can go to find a common implementation of like a VPC or a Kubernetes cluster or something along those lines?

[00:46:37.350] – Matt
So we’ve got a couple of things to look at. So when you’re talking about the reference, we have something called the Pulumiverse, which is relatively new, but I’m really excited about it. So this is 100% community driven, except that Pulumi. We help fund it, we support it, but we don’t run it. And this is the place for the pole community to interact. They collaborate, whether they’re on libraries, projects, learning resources, all that stuff. So one of the big things at the Pulumi versus the awesome list, right? Like we have there’s a lot of resources have these. We have the awesome pole. So this is just a curated list of tons of tutorials, books, tools, all that kind of good stuff. But this is also where people can go in and creating providers. So for example, the provider for Concourse that got created by that, that’s not an official Pulumi one, but it had somewhere to live. Because what we encourage people to like for these community created providers and resources, we encourage people to put them under Pulumiverse, which is just a GitHub organization, because first of all, then you don’t have to worry about being the only maintainer and such.

[00:47:55.600] – Matt
And they’re also in addition, we have the Pulumi registry, which is available and where you can find kind of all the packages that we are Azure of that people have published them to either there and you’ll be able to see if they’re Pulumi official or community, which doesn’t mean that the community ones are not good. It’s just the kind of thing where you know if you can come and yell at us versus yell at someone in the community. But I’m really excited about the Pulumi verse. We have a program that we officially announced last week called the Pollinaries, which is kind of our Champions program. And the preliminary verse and the preliminaries are loosely coupled in that the governing board of the Pulumiiverse, some of those folks came from Pollinators, but this is very similar. If you’re familiar with like box populai and the public community, this is what we’re trying to encourage here. And yeah, I’m really excited about it. I’m excited to see how this continues to grow. And it’s just been very hard for me to keep my hands off because I want to be like, I can contribute to it all day long.

[00:48:57.830] – Matt
But I keep saying, like, hey, I work at Pulumi. This is that. I mean, I’m part of the community, but we explicitly have a rule that you can’t have more than one member of actually, there is one member of the governing board of the Pulumi versus the Pulumi employee, which we then have to keep rotating as we keep hiring people.

[00:49:18.550] – Ethan
I would have gone for Puluminati. That would have been my choice right there.

[00:49:21.830] – Matt
That might have gone through some of the, you know, we have Pulumi, which are people who used to work at Puma.

[00:49:30.850] – Ethan
So Mattie, people have been listening to this, and now everyone that’s listening obviously wants to try Pollmer. So where would you guide them so they can get started, documentation, whatever.

[00:49:41.290] – Matt
So the first thing I would suggest is if you go to Pollie.com and it’s pretty much all over, we have a link that says getting started. And I think I dropped it in the notes here, but it’s Pulumi.com getstarted. And what you’ll do there is just sort of sit down and say, like, okay, which cloud do I want to do? I want to try. Now, if you want to even do it the even simpler way, I would go to Pulumi.com Learn, and these are self guided educational modules that my team has worked on, specifically Laura Santa Maria. And my team has written these up, and they’re amazing and they aren’t super duper complete. Like, again, you’re going to look at this, you’ll be like, oh, why am I doing this in Docker? But it’s to get the idea of the concept right. So you’re like, this is how this works, and we’re continually adding more learn modules. Yeah, if you go to pulumi dot Com resources, this also will show when we have upcoming workshops. Our workshops are all based on that learn material as well. So if you like to do a workshop, a virtual workshop where you’re like, cool, Mattie or Kat or Laura or Rocko are going to be like doing this along.

[00:50:48.500] – Matt
And I can follow along. Those are great, too. We are starting to experiment with trying to get out for in person workshops. Again, look at that resources place. And I also really like, Gloomy TV will take you to our YouTube channel that’s very active, in addition to, you’ll see the usual kind of, like, announcement stuff. But there’s two series on actually technically three series I would pay attention to on Pulumi TV. One is called Modern Infrastructure. We used to call it Modern Infrastructure Wednesdays, but it was easier to not have to have it always be a Wednesday. So that’s David Flanagan. We’ll do a lot of deep dives into doing some cool stuff. Laura Santa Maria does a series called quick bites of cloud engineering that are just kind of short videos around ideas like there’s one about what is state, what do we mean by state? What do we mean by cloud engineering? And I run our weekly livestream called Pulumi which you can if you like Twitch, you go to Twitch. Tv Pulumi. It’s also live streamed on our YouTube channel and so again, there’s lots and lots of different fun ways to lose lack of information.

[00:51:56.990] – Matt
No. And if anybody ever has as you’re trying to get started, our community slack is awesome for that but if you ever have any questions, you’re trying to figure stuff out. My job is to help people get going so you can always find me on Twitter, at Matt Stratton on Twitter and find me in our community slack which is slack gloomy.com and yeah, this is what I love to do.

[00:52:22.970] – Ethan
Great stuff, Manny, thank you for taking time with us here at day two, cloud man, we tried to record this a while ago and you had a power outage of all things.

[00:52:31.930] – Matt
Yeah. Boy, did that turn out to be expensive we’ll just put it that way I’ll put it this way while we were sitting and recording my doorbell camera went off and it was the electrician coming to pick up the check that I left for him and I could just feel it coming out of my bank account as he walked away but everything’s safe now. That was the important thing. It was a safety thing. My house will hopefully not burn down now.

[00:52:59.710] – Ethan
Well, thanks again, Mattie for appearing on day two cloud today and if you’re still listening virtual high fives to you out there for tuning in, you’re awesome. And if you have suggestions for future shows, hit us up on Twitter at day two cloud show or go up to Ned’s fancy website Ned in thecloud.com if you’re not a Twitter kind of person now, did you know that you don’t have to scream into the technology void alone? The Packet Pushers podcast network has a free slack group open to everyone. Visit PacketPushers. Net slackandjoin. It is a marketing free zone for engineers to chat, compare notes, tell war stories and solve problems together again. Packet pushers. Net slash slack and until then, just remember cloud is what happens while it is making other plans.

More from this show

Day Two Cloud 147: Google Cloud Is Not Just For Devs

Today on Day Two Cloud we peel back the curtains on Google Cloud with a GCP insider to find out how Google Cloud differentiates itself, its embrace of a multi-cloud approach, and more. Our guest is Richard Seroter, Director of Outbound Product Management...

Episode 146