Follow me:
Listen on:

Day Two Cloud 090: Hashicorp Vault For Beginners

Episode 90

Play episode

Today’s Day Two Cloud is a deep, if impromptu, episode on Hashicorp Vault. Due to an unexpected schedule change, hosts Ethan Banks and Ned Bellavance had to come up with a topic on the fly. It turns out that Ned knows a thing or two about HashiCorp Vault, so hey presto! Please note this is an unsponsored show.

What is HashiCorp Vault and why is it worth talking about? Well, if you’ve got secrets you need to keep (passwords, certificates, API keys, and so on), Vault is a management tool that stores and controls access to sensitive data. It can also be used to manage the lifecycle of credentials, and can encrypt and decrypt data as a service.

Ned and Ethan discuss:

  • Major Vault use cases
  • How to create a production Vault environment
  • Simpler Vault builds for lab work
  • Vault storage options
  • Master keys and encryption keys
  • Managing the secrets that manage the secrets
  • Vault secrets engines
  • Authentication methods supported by Vault
  • Using Vault to generate temporary secrets
  • Secrets lifecycle management
  • Controlling access to secrets
  • Securely accessing Vault secrets from Python
  • Preventing Vault system failures
  • Recovering Vault if the worst happens

For more in-depth training, see the links below to get Ned Bellavance’s PluralSight course Hashicorp Vault.

Sponsor: CBT Nuggets

CBT Nuggets is IT training for IT professionals and anyone looking to build IT skills. If you want to make fully operational your networking, security, cloud, automation, or DevOps battle station visit

Show Links:

Getting Started with HashiCorp Vault – PluralSight

Managing HashiCorp Vault – PluralSight

Hashicorp Certified Vault Associate: Getting Started – PluralSight



[00:00:26.070] Welcome to Day Two Cloud. This is a weird show because what just happened in the background that you never see is we had someone cancel due to a family emergency. And that happens, you know, that happens from time to time. But Ned and I were like, we have the slot saved on our calendars. Why don’t we just record something?

[00:00:44.040] Well, OK, listeners, there’s something that I’ve been looking into is how to better manage secrets. So let me give you a scenario here. If I do a little bit of Python programing, because I’m doing, in my case, network automation or I’m hitting the Twitter API or something like that, I’ve got secrets, usernames and passwords and API keys and things that I don’t want anybody else to know, but my script needs to know. I had to pass that information into my script.

[00:01:09.570] And so what I do these days is environment variables. I don’t want to code them in the script. I’m not that lame, but I will do environment variables where I’ll pass them through from the environment to the script. That’s OK. It’s still kind of sucks, though.

[00:01:24.000] It doesn’t really manage them very well. I don’t like that. And so as I’ve dug around, I’ve come to learn there are various secrets, management tools, one of which HashiCorp Vault. That seems to be a pretty popular solution.

[00:01:37.590] I’ve been meaning to dig into it. And what do you know, Ned? On a day when our guest had an emergency and was unable to record with us, it turns out you our very own Ned Bellavance have been working on a HashiCorp Vault course. Now, is this course actually available right now on Pluralsight or is still in the works?

[00:01:57.300] – Ned
The first of the courses is available on Pluralsight right now. This is a brand new HashiCorp Vault course that is centered around the vault certification at the associate level. That’s the only certification that exists right now. But if you’re listening to this in the far future and there’s a Pro Cert or whatever, this is for the associate cert and it’s going to be one of two courses. The first one is published, and that’s the bulk of what you need to know.

[00:02:22.200] But then there was some additional stuff that the course is already three and a half hours. If I added anything else, it was just going to be overwhelming. So you got to break it up a little bit. Right.

[00:02:30.780] – Ethan
OK, this ends up being a perfect topic of discussion that because your head is full of things related to vault and you know, you’re working on this additional material and stuff. So listeners to Day Two Cloud usually Ned and I have a script. There is like a lot of questions. We’ve been planning this out. There’s been research and homework done. Instead, I’m just leaning hard into Ned and I’m going to start picking his brain about vault, reacting to the things he said. Day Two Cloud unscripted, live, raw. That’s what you get today.

[00:02:58.140] – Ned
I feel like I should be in a cage match or something.

[00:03:01.500] – Ethan
OK, so so let’s let’s start at the beginning Ned, the start of the beginning. HashiCorp vault is for secrets management. Tell me if I’m right or wrong here. My understanding of what this does is it allows me to securely store those very special little bits of data that authenticate me to some resource. If it was networking, it could be things like a username and password or an enabled password. Again, going back to my Twitter example, I have a series of Twitter keys that I got from when I registered as a dev to get all those keys.

[00:03:33.630] I can store that stuff in vault, use a library, use some methodology to go and retrieve those secrets securely, pull them into my scripter process, whatever it is that needs those secrets and not worry about having to in plain text, like as an environment variable or embedded in the script itself, the horror store, those secrets there

[00:03:56.340] – Ned
That is essentially correct. There’s a lot more to the product than that. But if that is your use case, if you need to store sensitive data somewhere and then retrieve that data in a way that it will be encrypted in transit and hook into whatever software you’re using to retrieve that secret vault is something that will do all of that. It can do more. But it it’s perfect for your use case if that’s the thing you want to do.

[00:04:21.570] – Ethan
So what is the more then?

[00:04:23.400] – Ned
So we can think about secrets or sensitive data data that you want protected through encryption or securely stored somewhere. You can store it, you can generate it, or you can just encrypt it as a service. So those are the three kind of buckets that you can put things into what vault does. So if I have a static secret, something I’ve generated outside of vault, and I want it to just securely store that thing and it doesn’t have to be a username and password, it can be any data, anything that you can base64 encode and send to vault.

[00:04:56.760] It can store it. So if you have certificates, API keys, username and password, just a bunch of key value pairs that you want stored in vault, it can do any of those things. So that’s that’s the store portion. There’s also the generate portion. So what if I want to dynamically create credentials, let’s say for AWS, I’m running some sort of pipeline or script and I don’t want statically defined access keys in that script or even like another place where I’m storing them.

[00:05:25.220] But they’re still static, so they’re just there forever. I want to dynamically generate credentials for my script, have those credentials have a lifetime associated with them. And when that script is done or after a certain amount of time has elapsed, those credentials are revoked and vault takes care of that lifecycle. So it’s not just storing a secret, it’s actually generating those credentials or those secrets for you and managing the life cycle of it. So you know that those credentials are going to be somewhat short lived.

[00:05:53.690] And then lastly, it can encrypt stuff for you. So if you just need encryption as a service, you want to send some data to vault it will encrypt it and send that encrypted data back to you. It manages the keys it’s using for encryption, but the actual data doesn’t live in vault. It’s just providing that as a service.

[00:06:11.570] – Ethan
I was with you up till the last point because I had a very straightforward metaphor, a product that I use all the time. I use a password manager that can do all kinds of things. Now, it doesn’t just do passwords, but it’ll also store your credit card numbers and a lot of other things like that. If you want to sort of store it, it’ll generate passwords for you. I use that feature all the time. It stores them encrypted, of course.

[00:06:33.500] And then there’s a ciphered exchange if you’re doing that remotely because it’s a cloud based service, doesn’t do lifecycle management. You know, that’s that’s me. I’m the lifecycle manager as I dig through it. So it’s a little awkward there. But that encrypt thing at the bottom, if I want something encrypted, hey, vault, encrypt this and then give it back to me. Yeah. You lost me on why I would do that. Gimme a use case.

[00:06:54.230] – Ned
It sounds a little weird and there’s a few different use cases for it. Let’s say you have some sort of application that writes data to a NoSQL database like document DB or something, and you want values in particular columns to be encrypted rather than relying on the application to do that encryption or the database engine to do that encryption, you could have vault send the data to it, it’ll encrypt it, send it back to you, and then you write it out to the database.

[00:07:20.870] You can do the same thing with object storage, a file share, just anywhere that doesn’t natively provide the encryption you want or you want additional encryption on top of that. Here’s the way that you could do it without having your application be responsible for managing the keys.

[00:07:34.550] – Ethan
Potentially adds a bit of complexity to your application. But there’s any number of things that you might be storing in the database that you’d want encrypted. OK, I see where we’re going with that, I mean passwords pop to mind. You don’t want to be storing those in plain text. And who does that? No one does that. But I don’t know if passwords would even be a good use case for vault here, but at least it pops to mind is something that might be stored in a database that needs to be encrypted.

[00:07:56.030] – Ned
Right, right. Personally identifiable information that you want to write to a database you don’t necessarily trust where that database is being hosted or you don’t trust it enough. Maybe, you know, it’s in the cloud and you’re like, I’d feel better if I held the keys for the encryption on this and it just stored the data for me and I can retrieve and decrypt it on demand. Here’s a good option for doing that.

[00:08:17.450] – Ethan
Vault is what is it, open source with a commercial variant from HashiCorp something like that?

[00:08:22.460] – Ned
Yeah, it’s the standard open core model. If you want these additional features, those are enterprise and you have to buy enterprise licensing from HashiCorp. So I will restrict this to only things that are available in the open source version. And then if we wanted to talk about the enterprise stuff, we can. But I think for your use case, especially everything that you talked about you want to do is supported by the open source.

[00:08:46.400] – Ethan
That’s not a big demanding sort of a process because I imagine it’s going to be very use oriented. If it’s going to be just going to be sitting there doing nothing. So it’s not a lot of CPU, not a lot of RAM, not a lot of server I’m imagining.

[00:08:57.830] – Ned
No, and the deployment options, you can deploy it on Windows or Linux or a bunch of other operating systems because they wrote it in go. So you can use the go compiler to compile it for whatever operating system makes sense for you. And there’s a container available as well. So if you want to run vault in a container it already and just grab the latest version and apply a config, you can do that too. So there’s a lot of options there in how you deploy it.

[00:09:21.530] – Ethan
So that part even is boring. All right.

[00:09:24.740] – Ned
I won’t say it’s boring. There’s a lot of minutia on how you actually install and configure it. Well, it’s not like you’re managing Microsoft Exchange. All right.

[00:09:32.390] – Ethan

[00:09:32.960] – Ned
You don’t want to go home and drink a tall glass of whiskey at the end of the day as a former Exchange administrator.

[00:09:43.460] – Ethan
A tall glass of whiskey!

[00:09:44.630] – Ned
Yeah, yeah. Skip the beer. Just give me the whiskey for real. I’ll drink it straight from the jar. I don’t care.

[00:09:51.860] – Ethan
So with my vault architecture, what is in there is very important.

[00:09:56.330] – Ned

[00:09:56.750] – Ethan
So are there best practices for things like should I run two instances of vault that back each other up or manage the database in a particular way so that I know my repository is going to be robust and backed up and so on?

[00:10:08.660] – Ned
OK, let’s let’s break that down. How does Vault go about ingesting your information and storing it securely? So if we just start with a single vault server, just one. Right. And then we’ll worry about high availability later. So it has a front end. That’s the API. Any way that you want to interact with Vault is going to happen through that API and obviously you’ll add an SSL cert, so all communication is encrypted. There’s no other way to interact with Vault other than the API.

[00:10:35.290] If you’re using the CLI, it’s actually using the API is just an implementation. Same thing with there’s a UI Web interface. It’s all going through that same API.

[00:10:43.780] – Ethan
So that API, is it just a rest API.

[00:10:46.210] – Ned
Yeah, rest API that sits in front of everything and then they have what they call the barrier. So the front end of the API hits the barrier. That’s where everything stops. And then in the barrier, all that’s running in memory and anything that leaves the barrier leaves encrypted. All of your secrets and other information are written to a storage back end, but they’re written out encrypted and those keys are managed by vault.

[00:11:11.560] So even if someone steals your storage back end, they can’t do anything with it because all the data is encrypted. It’s only unencrypted when it’s sitting in memory, which means that you should disable any kind write memory to hard disk. Most people don’t need virtual memory anymore, but you should disable that if you have it enabled because you don’t want it getting written to disk unencrypted. So that breaks down where the data is and when it’s unencrypted and when it’s encrypted. So the next question is, how does Vault manage the keys?

[00:11:43.300] – Ethan
Well, that’s that’s where I was going because you were talking about the bits that are in memory vaults managing the keys. And I’m like, wait a minute, that feels like my next vulnerable point and I’m scared.

[00:11:50.760] – Ned
OK, so there’s going to be two important keys here. There’s the encryption key and the master key, the encryption keys like a key ring of keys. But the master key is broken into pieces using a special algorithm. And those pieces don’t live in vaults. They live with you and your other shareholders. So you break it into three, five shares. And in order to start up a vault and read in the information from that storage back end, you need a certain level, a threshold of shareholders to put in their fragments of the key and then the vault can assemble that back into the master key in memory, decrypt the encryption key that’s sitting on the storage, load that into memory, and then use that encryption key ring to decrypt anything that’s been written out to the storage back end.

[00:12:41.200] – Ethan
So you’ve got this fragmented key to get to the decryption key and then when you’ve got the decryption key, you can interact with the backend database.

[00:12:49.930] – Ned

[00:12:50.470] – Ethan
But OK, that fragmented bit there of the that was the master key, right?

[00:12:54.850] – Ned
That that was the master key. Yes.

[00:12:56.570] – Ethan
OK, you said I’ve got to split that up among shareholders. What’s a shareholder?

[00:13:02.230] – Ned
That would be a trusted person that will have one fragment of that key. And you want to have enough people there that when you’re doing something with vault, like starting it up or if you have to recover something from vault, from a backup, you want all those people there sitting in front of the keyboard, the screen, making sure no one else is doing anything weird with the data. And even if someone loses their particular key shard, it’s not enough to unlock vault. You need enough, the threshold number of keys.

[00:13:33.010] – Ethan
So shareholders are meat bots. I thought you were too. I thought it was like a metaphor for, you know, a concept we see here. We call this server over in the corner of a shareholder. No, you’re talking about an actual human that in some secure manner is they’ve got it on a USB drive or something like that has a fragment of your key.

[00:13:50.590] Yes. And most commonly, when you initialize vault, it creates that master key during the initialization process and the shareholders will supply a PGP key to encrypt that share. And that’s what actually the output is from the initialization process is the encrypted key using their PGP key. And then only they can decrypt that.

[00:14:11.530] – Ethan
Their key fragment.

[00:14:12.670] – Ned
Their key fragment.

[00:14:13.720] – Ethan
Really. OK, OK, so I get this. I get this. But this implies something really important about vault. You’re not cranking up and shutting this thing down all the time.

[00:14:24.400] – Ned

[00:14:25.000] – Ethan
There’s a very intentional process that is involved in starting a vault server. So if the vault server were to crash or fall over or be disrupted because of whatever, you have to go through that startup process all over again with those meat bots.

[00:14:40.180] – Ned
Man, that seems like a pain, doesn’t it? Like.

[00:14:42.040] – Ethan
That’s it does. It sounds awful, but…

[00:14:44.200] – Ned
It’s the price you pay for security Ethan.

[00:14:45.940] – Ethan
I know, right?

[00:14:47.080] – Ned
Just to be clear, there is another option. Rather than storing the master key in fragments, you could also store the master key and some sort of trusted external entity. And that could be a key vault. Azure key vault service would be an example. The AWS KMS’s service could be an example, or you could do it with an on premises HSM, which is a hardened security module, which is those are expensive and they live outside of the server and there’s all these tamper proof protections on them.

[00:15:19.840] The master key could live there as well. So you actually have a few options. For storing that master key without breaking it into these fragments, and so then the vault server could start up automatically as long as it has access to wherever that master key resides.

[00:15:37.210] – Ethan
Well, you’re reminding me of a data center I worked in that had a rack with a Thales box in it. And that Thales box was was all about that. And that rack was was very special. Access to the Thales track was via particular pass code that only the data center manager and like two other people had in the whole organization because of the nature of what the box was.

[00:15:57.490] – Ned

[00:15:57.910] – Ethan
Not only was the Thales box doing super important security stuff on the software side, physical access to it was therefore restricted into a data center that was very hard to get into to begin with, with man traps and bio scanners and all the rest of it. So kind of a big deal. So you’re talking about a solution along those lines.

[00:16:14.500] – Ned
Along those lines. So if you already have that in your data center, you can use that to auto unseal the vault when it starts up. If you don’t have that, you could take advantage of one of these cloud HSM basically, or key management services. And if you don’t have that or don’t want to use that, then you’re back to key shares. And then we’re now talking about high availability because. Yes, what if one of those servers falls over? I need my vault service still available to me.

[00:16:40.630] – Ethan
Well, if I’m doing this in a lab, I mean, I can be multiple personalities and just hold all the key fragments myself, I suppose.

[00:16:46.870] – Ned
Or you can set the shares to one and just get one key back. You can do that. It obviously is not the most secure way. But if you’re running this in a lab environment where you’re not that concerned, you can for the vault configuration, you can set the share and threshold to one and it’ll just give you basically the master key.

[00:17:04.390] – Ethan
But again, for a grown up production environment, you’re taking the security very seriously. You’re encrypting and storing secrets that are very important and say you use case is automation of some sort.

[00:17:16.150] Well, if someone were to have access to your infrastructure, be through your automation, they could wreak all kinds of havoc, whether very subtle, just watching what’s going on and dropping little man in the middle and processes that can watch things go by or they can do horrifying things that they wouldn’t want.

[00:17:36.580] My point being, you don’t want to sacrifice security in your vault environment on the altar of convenience. You do want to take it very seriously and treat the security requirements as they deserve to be treated, which is robustly.

[00:17:49.220] – Ned
Exactly, yeah. And if you do want to set it up in a highly available setup, you can have multiple vault servers that are running in an unsealed state, but only one of them can be active at a time, at least for the open source version. You can only have one active vault server. You could have two passive servers and set them all up behind a load-balancer or use DNS Round-Robin and the passive ones will actually proxy requests to the active one. So it doesn’t matter which server the request hits, they’ll get proxied to the active.

[00:18:22.180] – Ethan
Because the active ones, the one with the current database.

[00:18:24.970] – Ned
The active one, is the one that’s allowed read write access to your storage back end. In order to do high availability, you need a shared storage back end. But only one of them can have active write access to that storage back end. If you get the enterprise version, they actually have performance nodes, which are passive nodes that have read access to the storage back end so they can respond to read requests but not write requests.

[00:18:47.800] – Ethan
I try to imagine the scenario where you’d need that for load, but I guess there must be some really large deployments where you’re checking in and out secrets all the time.

[00:18:56.350] – Ned
Well, I guess that gets into a good segue of how you might interact with that and some of the different ways that you authenticate with Vault. So maybe that’s where we can move. Next is OK, I’ve got my vault stood up and I want to do stuff with it. How do I authenticate to gain access to do stuff on vault? And they have a thing called authentication methods. Isn’t it nice that they named it something that is the thing it does?

[00:19:20.800] – Ethan
Well, it’s funny actually. If we just take one step back, all the stuff we talked about was just getting the vault infrastructure ready for us to use and being very careful that we are securely standing it up and and all of that. Right. We haven’t even gotten to the point of, yeah. Now I want to actually pull some data out of there, which is where we’re at now. OK.

[00:19:39.850] – Ned
Right. When you initialize vault, it gives you a root token. Tokens are the way you interact with vault. So if you think of any other rest API, you almost always need an API key or a token to interact with that rest API. Right. That’s just kind of how APIs work. Yeah, well, Vault gives you a root token which has full privileges to do anything with Vault. So obviously that’s a dangerous token. You don’t want to keep that around for a long time. So the first thing you do is set up an alternate authentication method to get tokens another way.

[00:20:12.190] – Ethan
You’re going to say Oauth2, aren’t you? And I’m going to come through the camera and strangle you. I hate Oauth2 so much.

[00:20:17.800] – Ned
That is an option. It’s not the only option. If you’re in a traditional enterprise that has active directory, you can set that up as an authentication method, people can just login or OpenLDAP or GitHub is another authentication method, you can use usernames and passwords from GitHub and map them to vault. You can use AWS’s IAM for authentication. You can use Azure Active Directory. So there’s a lot of options for configuring those methods.

[00:20:47.440] – Ethan
[AD] We paused the episode for a bit of training, talk training with CBT nuggets, if you’re a Day Two Cloud listener, you are you’re listening to the podcast right now, then you’re probably the sort of person who likes to keep up your skills, as am I.

[00:21:00.850] Now, here’s the thing about Cloud. As I’ve dug into it over the last few years, it is the same as on Prem, but it’s different. The networking is the same, but different due to all these operational constraints you don’t expect. And just when you have your favorite way to set up your cloud environment, the cloud provider changes things or offers a new service that makes you rethink what you’ve already built.

[00:21:18.080] So how do you keep up? Training, now for training companies? What did you think I was going to say? Obviously, training and not just because sponsor CBT nuggets want your business, but also because training is how I’ve kept up with emerging technology over the decades. I believe in the power of smart instructors telling me all about the new tech that I can walk into a conference room as a consultant or project lead and confidently position a technology to business stakeholders and financial decision makers.

[00:21:45.820] You want to be smarter about cloud CBT Nuggets has a lot of offerings for you, from absolute beginner material to courses covering AWS, Azure and Google cloud skills. Let’s say you want to go narrow on a specific topic. OK, for example, there is a two hour course on Azure security. Maybe you want to go big. All righty then. There is a forty two hour AWS certified SysOps administrator course and there’s a lot more cloud training offerings in the CBT Nuggets catalog.

[00:22:13.390] I just gave you a couple of examples to whet your appetite. In fact, CBT nuggets is adding forty hours of new content every week and they help you master your studies with available virtual labs and accountability coaching. And I’m going to I’m going to shut up now and get to the part that you actually care about, which is the special offer of free stuff that you get from CBT nuggets because you listen to this entire spot, you awesome human first visit, CBT nuggets, dotcom slash cloud.

[00:22:39.580] There you will find that CBT Nuggets is running a free learner offer. They’ve made portions of their most popular courses free. Just sign up with your Google account and start training. This free learner program is a great way to give CBT nuggets a try. Now, as a bonus, everyone who signs up as a free learner will be automatically entered into a drawing to win a six month premium subscription to CBT nuggets. So this is a no brainer to me. Just go do it. CBT nuggets, dotcom slash cloud that’s CBT nuggets, dotcom cloud. And now back to the podcast that I so rudely interrupted. [/AD] [00:23:16.450] One thing I noticed in all those methods is that there’s granularity available, it’s not, you know, simple authentication you can assign like if it’s a radius of, I don’t know, the actually said radius, but…

[00:23:27.580] – Ned
Radius is in there. Yes. It’s also…

[00:23:29.710] – Ethan
Well, in theory, if there’s unique attributes for vault, you could assign attributes that use it that, yeah, they can get into a vault or no, no, they can’t get into vault and then have all that be audited and so forth

[00:23:38.230] – Ned
And it gets much more granular than that, because even if you authenticate successfully the way that they control access to various actions within vault is through policies. So that’s the other construct. Once I successfully log in and get myself a token from my successful login, that token has policies associated with it. And those policies say here’s what you can do on vault and anything that’s not explicitly stated in the policy, you can’t do.

[00:24:05.140] – Ethan
Those would be actions. Does that also limit the resources that I can get access to, like, say I am OK with someone having access to a particular, I don’t know, set of network devices, but not access to my social media API keys?

[00:24:22.000] – Ned
Yes. So everything in vault is a path. If you think about it, it’s a rest API. So everything in an API is referenced by a path of some kind. So really what you’re doing when you create policies, you’re saying here are the capabilities or actions that a person or a client is allowed to do along this path. And you can add a star to the end to extend that path out. Or you can put a plus sign in the middle of the path to say, I don’t know what that value is going to be right there. But just, you know, it’s going to be something and you can grant permissions on that path.

[00:24:54.850] – Ethan
A path is a text string.

[00:24:56.230] – Ned

[00:24:56.500] – Ethan
It sounds like that vault knows how to parse.

[00:24:58.780] – Ned

[00:24:59.080] – Ethan
Does that mean you need to know how to compose that string or is there a tool that, like, helps me build it?

[00:25:04.540] – Ned
You do need to know how to compose that string, but it’s not especially difficult because there are standardized paths for everything. So when you add a new authentication method, the standard path for that is going to be auth. And then the name of the method that you enabled it with, slash whatever the path is to users or roles or something along those lines. And you can always refer back to the official API docs to figure out what that extended path is.

[00:25:30.910] – Ethan
So I’ve got my authentication method. I am now allowed to do things within vault because I have a token. The path would be associated with the token.

[00:25:40.150] – Ned
The policies are associated with the token and in those policies is the path and each path has capabilities granted to the holder of that policy.

[00:25:50.650] – Ethan
So I’m going to make a guess here that one of the things I can do is check a secret into the vault.

[00:25:56.110] – Ned
Yes. So now we need to enable a secrets engine.

[00:26:01.160] – Ethan

[00:26:02.770] – Ned
So out of the box vault has all these different secrets engines. They’re also called plug ins sometimes, but they’re mostly called secrets engines. And there’s just different implementations. And you can mount a secret engine as an instance and you can mount that same engine multiple times and on different paths and have different permissions for different instances. So one of those engines and the one that you’re probably thinking about when you’re thinking about checking in secrets is the key value secrets engine.

[00:26:31.780] – Ethan

[00:26:32.140] – Ned
What does it do? It stores, keys and values at a path. That’s it. It’s just that easy.

[00:26:39.130] – Ethan

[00:26:40.720] – Ned
Yeah, they keep naming things so well. I’m almost angry about it.

[00:26:45.310] – Ethan
I gotta pause here for a second though Ned, as we’ve been talking through this, as we’ve been going through the the details of this, I feel like there’s a lot of footgun’s involved where you can really shoot yourself in the foot if you get it wrong. On the one hand, you could be like, this is convoluted. I’m just going to use like the root token for now. A foot gun. Shouldn’t do that.

[00:27:02.390] – Ned
Don’t do that.

[00:27:02.390] – Ethan
So bad idea, etc. There’s a lot of things you could do that are bad like that. On the one hand. On the other hand, you could be like, oh, baby, I’m a control freak. I love this. I am going to just bury this thing in policy and granularity and no one will ever be able to use it because it’s just been so overengineered. You just can’t get stuff in and out of Vault. Is that seem plausible to you?

[00:27:26.680] – Ned
If you have that kind of security team, then yes, they could render it totally useless by locking every little bit down so no one can do anything. So that’s bad. But that’s more on your security team. And I’m sure they’ve done that before in terms of not using the root token. Yeah, don’t do that. All you need to do is once you fire it up for the first time, create an administrator policy and assign that policy to you can enable username password temporarily as an authentication method, throw username and password in there, assign the administration policy and then login with that and you’re good. Get rid of the root token. You can do all your administrative tasks, but you don’t have the the God like permissions of the root token anymore.

[00:28:05.740] – Ethan
It feels one of those products that’s so capable and it’s security oriented that you could just be like, screw it, it’s too hard. I’m going to, you know, be too generous. And then on the other hand, you could be too too granular. But let’s go back to checking a secret in. And so now I’ve got the permission to check in a secret I assigned what sort of a secret it is by firing up a secrets engine, you gave an example of a key value pair secrets engine. Give me some more secrets engine examples.

[00:28:33.330] – Ned
Well, the one I talked about earlier, dynamically generating credentials for the cloud. There’s a secret engine for AWS. And what that Secrets Engine does is generates temporary credentials for AWS. When you mount and configure that secret engine, you have to give it credentials to go out to AWS and create those credentials dynamically. Oh, so there’s some configuration that needs to happen there. But once it has those permissions now, it’s able to dynamically generate credentials and hand them back to you when you ask. And also associate a lease with it saying this is good for 30 minutes. And after that, I’m going to go back out to AWS and revoke your access keys.

[00:29:13.190] – Ethan
You’re reminding me of something really important here. I keep thinking in my head, even though you’ve already explained that it isn’t limited to this, I keep thinking of it as a store, a place that I keep credentials, which it is. But that isn’t all that can do. And so the secrets engine here, you’re talking about credentials that don’t ultimately. Well, I don’t know. Do they do they even live in vault? Does it matter? I don’t know.

[00:29:32.330] It’s actually generating them going out to AWS, which is where the authentication is actually going to happen. We’re just using vault as kind of a proxy. And and that lifecycle management you were talking about for these credentials, because I want them to be temporary. I don’t want to be alive for 30 minutes.

[00:29:47.420] – Ned

[00:29:47.870] – Ethan
Boy, that’s just a mind expander, dude. It is. It’s a big deal.

[00:29:51.260] – Ned
It shift things. And it’s not just the cloud providers. It also has one for almost all the major database engines out there. So if you have a database engine, you want to be able to dynamically generate credentials for your developers or whomever or CI/CD process or whatever it is, it can go out, add some credentials to the database engine and then revoke them later and you’ll have different roles configured for the different types of credentials you want returned.

[00:30:17.390] – Ethan
I love this and it also scares the crap out of me because it puts a vault in the middle of so many transactions potentially.

[00:30:23.450] – Ned
Yes. And that’s why you want redundancy. And that’s also why there are so many controls that can be put in place because it could be ripe for abuse if you don’t configure this thing properly.

[00:30:36.080] – Ethan
So let’s simplify the example then and get specific with with my particular very boring, I suppose, use case. But I’ve got these I’ve got API keys that I want to store. They’ve been generated and handed to me by the API provider. I want them to now be checked in and living in a vault as a repository.

[00:30:54.230] – Ned
So you would have your authentication method set up on vault for however you want to authenticate. GitHub is one of the easiest and I think most obvious ones because you already have GitHub credentials, most likely. So use those to authenticate to your vault server and then you’ll set up a key value engine where you want to write those API keys and you will write them there and that’s where they can live. I assume you want to get them out at some point, though, right?

[00:31:19.290] – Ethan
Very much so, yes. And securely.

[00:31:21.170] – Ned
All right. So you as the human are the one who checked that secret in. But you’re going to have some sort of software or something that’s pulling that secret out. What does that software look like? So you’re going to use Python. Python now needs a way to authenticate to vault. There’s a number of different ways it could do it. The most common one for just an application that doesn’t have any other alternative is authentication method called AppRole.

[00:31:43.550] And AppRole is basically like username and password, but a little more advanced than that. And your Python scripts will have a username and password it can use to authenticate, to vault, to get secrets. So it still needs some way to authenticate.

[00:31:57.050] – Ethan
Of course.

[00:31:57.650] – Ned
Now, if you were running that Python script from, say, an Azure virtual machine, it could get a token from Azure active directory and authenticate that way based off of its own machine identity.

[00:32:10.160] Or you could have certificates loaded on the machine it’s running from and it could use those certificates. So there’s a lot of different ways to authenticate. But ultimately, Vault needs some way of trusting where that request is coming from. And that’s going to be an authentication method of some kind.

[00:32:27.450] – Ethan
It feels like a chicken and the egg problem, it is.

[00:32:29.810] – Ned
It’s a little bit, you’re not the first person to say exactly those words to me. So some of the things you can do to make it more secure from the AppRole perspective is you can lock it down by an IP address range and say we’ll only accept this username and password or role ID and secret ID from this specific network range. If it’s coming from anywhere else, just deny it. So at least then you’ve got some protection. And like I said, you could do it through certificates instead.

[00:32:57.260] Just have a certificate that’s only available on that box. So that’s a way of that box saying, yes, I’m the box you expect. Or like I said, if you’re running in the cloud, then you’ve got some options there as well. So there’s definitely a bunch of different ways to do it.

[00:33:10.010] – Ethan
Before we get to that next point, it just hit me that with the controls, the policies. You can build within vault, you could assign to that authentication method that the script is going to ultimately use what credentials it has access to. So it’s not like, oh, if you can figure out what the script, how it’s authenticated to vault, you got the keys to the kingdom. Nah, you don’t not if you’ve written your policies right. So you could lock down one script to have access to one particular set of credentials. And that’s it if you want it to be that tight with it.

[00:33:39.460] – Ned
Yeah, absolutely. You can just say you have read only access to this path. You can retrieve the secret and that’s all you can do at this path with this policy. Everything else to vault is a mystery to you. You have no access to anything else. So you’re locking things down in that regard. And then there are libraries for Python that you can use to call out to vault.

[00:34:00.760] So rather than using environment variables now, your actual scripts can just use the libraries for vault to grab those credentials and use them.

[00:34:10.120] – Ethan
Which ultimately is way more secure than, you know, environment variables which inevitably are sitting in plaintext somewhere.

[00:34:17.110] – Ned
Somewhere yeah.

[00:34:18.190] – Ethan
Somewhere might be might be obscure where they are, but they’re there. If you know where you’re looking, they’re there.

[00:34:23.170] – Ned
Yeah. Another common practice is to inject those values into environment variables but have them reside in memory. But then if someone has access to the system, they could just list out the environment variables. So that’s generally for short lived things. If you’re spinning up a container or a virtual machine to run a batch process or something, those environment variables are short lived and maybe no one has external access to that system anyway.

[00:34:47.830] – Ethan
OK, so in Python, I’ve got library support. I’m going to assume there’s good community and documentation out there for interacting, you know, Python and Vault. What other API consumers are going to need authentication methods that you see out there?

[00:35:03.910] – Ned
API consumers? In what regard?

[00:35:06.820] – Ethan
In vault like if I’m consuming the vault API to get secrets in and out of there, you know who else? Python’s one, because that just happens to be an environment code in. Give me some other examples.

[00:35:16.810] – Ned
I mean, I haven’t gone too far down the developer road, so I got to imagine that all the major programing languages have a library that supports vault because really it’s just a rest API. It’s not that hard to take an existing rest API library and bolt vault onto it. But ultimately, if your language or your script doesn’t have that, you have some other options. One of those is the vault agent, which is a process that runs on whatever machine that script is running from.

[00:35:44.290] And it handles the. Potentially the authentication as well as the storage of secrets for you. And it can act as a translation layer between your script and vault. It can place the secret in a nice place in memory or an environment variable in your local system. So you don’t even really have to change your script or application that much.

[00:36:04.300] – Ethan
Another moving piece, Ned. Now, there’s a vault agent that I can have. I got to chase down all my vault agents and or is that another thing where vaults really fussy about where the agents live and you’ve got to, you know, pair an agent with a server or something like that?

[00:36:18.700] – Ned
Well, the agents are going to use the same authentication methods that anything else would. It’s just they already know how to authenticate and retrieve secrets from vault, whereas your application might not. And, you know, if this is a home grown application and the developer left five years ago and you don’t have the source code, you don’t have the option of updating your application. So here’s a shim that basically between the vault server and your application, that will be that translation layer.

[00:36:44.110] – Ethan
Yikes. They thought of everything.

[00:36:45.970] – Ned
Not immediately, but over time. Yes. It certainly has grown right.

[00:36:53.320] – Ethan
There’s a lot of pieces here. I’m, you know, thinking this all through. There’s the setting up a vault and doing it in a secure and a careful way so that it is truly a vault. It’s not a steel cage with a big open window on the side.

[00:37:08.590] You can just jump in and out of. It’s truly more like a bank vault, all that challenge with the master keys and so on in the beginning to get it all built is that you truly have a vault. Now that we’ve got the vault, who can get in and out of the vault and under what rules and so on. Got that. I can store stuff in there. I can get stuff back out of there. What happens when I have API key rotation? Or got to generate a new set of API keys, then I got to therefore update something that is involved.

[00:37:40.440] – Ned
You have a few options. Version two of the key value engine has versioning built in so you can have multiple versions of a key. So if you need to keep the current API key and the previous one, you can do that or you can just write the new value to that same path. It will create a new version that will be considered the latest. So if you don’t specify a version, when you request the key, you’ll get the latest and it can keep up to ten previous versions of the key.

[00:38:08.140] You can actually configure that. But by default, it keeps ten previous versions of whatever data is at that path. Yeah, it’s got versioning, you can soft delete things, you can permanently delete things, you have a lot of options. Now, if you wanted to rotate keys on a regular basis, that’s more getting to the generation side of things where Vault is actually generating the API keys based off of some algorithm and then handing back that current key and handling the whole lifecycle.

[00:38:36.710] So that would be a more dynamic engine, whereas the key value is very static in that regard. You’re managing the lifecycle, the data, and you’re just sending it to vault as a keepsake.

[00:38:45.710] – Ethan
We were talking about the back end database and you mentioned storage. The way you phrased it, it’s like it’s not something special or unique inside of vault. It’s just, vault’s got to store stuff somewhere. It’s going to store it encrypted. So, you know, give it somewhere to store things. Is that kind of reality?

[00:39:03.110] – Ned
There’s so many different storage back ends. You can use something that’s like a key value storage back end, something like console or etcd are both supported. You can use a traditional database like MySQL that’s supported as well. You can just use a file system that’s also fine.

[00:39:18.980] – Ethan
I was going to ask you that. Yeah.

[00:39:20.330] – Ned
And starting with version one dot four or five, they added their own internal storage engine that uses the raft consensus protocol. So it still uses the local file system on each vault server. But you can have multiple vault servers and it uses a replication and consensus protocol to do distributed storage. So now you have redundancy of your data still sitting encrypted, obviously on the file system, and you don’t need to run totally separate application to provide that redundancy. And you don’t need like a storage array either.

[00:39:54.620] – Ethan
Oh, OK. So here’s a here’s a thought. If I can store the data in whatever construct MySQL you just listed off, all those different, you know, etcd, et cetera, options I’ve got, backing them up is a thing that I can do. That’s we know how to do that. But let’s say my vault infrastructure blows up on me. I’ve got a backup of that database somewhere and I need to just rebuild vault from scratch. There’s an issue here with the encryption key, though, right?

[00:40:23.450] Because Vault has to know how to decrypt the data sitting in that database. If I’ve blown everything up, what should I be thinking about here to make sure I have the ability to recover?

[00:40:31.850] – Ned
I would say you need three things. You need a backup of that storage back end, wherever it was. You need the vault server configuration file, which hopefully you’ve back that up or you have that stored in source control somewhere because that doesn’t have sensitive data. And it’s just, you know, it’s a configuration file. And you need the master key or the key shards. Those are the three things you need. The actual encryption key that’s used to encrypt data on the back end is stored on the storage back end. It’s just encrypted by the master key. Oh, so hopefully you have all the key shards for that master key or you’re out of luck.

[00:41:06.230] – Ethan
Wait a minute, though, OK, I’m just thinking about the master key. So the master key is authenticating me effectively then to. To what the data repository.

[00:41:14.420] – Ned
The master key is what decrypts the encryption key. So the encryption key lives in the storage back end and it’s pulled into memory and decrypted in memory by the master key, which is either broken up among multiple people or it’s stored in some sort of key management service.

[00:41:31.250] – Ethan
Yes, got it. OK, but again, another footgun. You really have to have a process for this? You need a playbook that, you know, the worst has happened. Your vault infrastructure is blown up and you’ve got to rebuild it from scratch. Here’s the things you need. Like you said, you need three things. You need the back end database. You need the master key and you need whatever you need.

[00:41:49.160] – Ned
You don’t need the vault configuration, but it’s going to make it a lot easier if you had that configuration.

[00:41:54.620] – Ethan
Last of all, configuration file, right?

[00:41:56.210] – Ned
Yeah, hopefully you have that somewhere. So, yeah. I mean, standard backup process applies, backup the server, back up the storage and have that master key somewhere and you’re good.

[00:42:07.610] – Ethan
Standard backup process. Yes, but what’s different here is the criticality of this and how often backup just is taken for granted. The vault service been running for two and a half years. No one’s really thought about it much. And then something goes sideways and just, you know, you’re in a bad state of having to do something that is more involved than simply instantiating a new process and reattaching the database, having control of those keys, knowing what they do and where they are and how you’re, as you were just describing, securely storing them has got to be known to the people that are in the position of needing to recover that vault repository or you are screwed. You literally just lost access to all of the secrets that might have been in there.

[00:42:48.980] – Ned
That could be pretty bad. Like that’s a resume generating event for sure.

[00:42:52.820] – Ethan

[00:42:53.360] – Ned
You can start out really easy with vault and just get used to it. There’s a dev version of the server that you can run locally on your system. You just download the binary and do vault server dash dev and boom, you’ve got a working instance of the vault server running in memory. So if you want to just play around with it, it’s super easy to get started now once you want to move this to a production scenario. Obviously, it gets more complicated, but that’s true of basically any software that’s worth a darn. Yeah. Its production. So it’s going to be complicated.

[00:43:21.190] – Ethan
But worth it. Yeah.

[00:43:22.450] – Ned

[00:43:23.110] – Ethan
Is there community for vault? Is there like a, I don’t know, Slack group or something where folks are active to help you with with Vault if you get in over your head.

[00:43:32.380] – Ned
I’m not a member of any Slack groups that are specific to that. There’s certainly a discussion group that is fairly active on HashiCorp. So if you go to the Vault website vault project dot I O you’ll find links to their discussion site and they have dedicated vault forums basically. So that’s a good place. If you’re stuck on something, people will be very helpful there. And that’s definitely where I’ve gone in the past. I get stuck on something.

[00:43:58.510] – Ethan
Well, I know at the top of the show we mentioned that you have educational material on this, so all your stuff’s on Pluralsight, right?

[00:44:04.900] – Ned
Correct. So I have three courses on Vault. One is a getting started and the second one is a sort of managing vault. So that gets really into like the storage back end, setting up the high availability, all that kind of stuff. And then the new course I just published is really centered around taking the associate level exam. So it’s very focused on covering all the objectives in that exam and it’s using the latest version of Vault. So I would probably start there if you’re brand new because it couldn’t hurt to study for it and it gives you a solid grounding.

[00:44:34.600] And then if you want to know more about managing a vault server, you have the other course. Beyond that, I also have a vault study guide on Leanpub. So if you’ve gotten through the courses and you think you need a little more prep, you can do the study guide and study that and then you should be ready to take that exam.

[00:44:49.560] – Ethan
Have you gotten through that exam yourself?

[00:44:51.370] – Ned
Yes. Yes, I took it.

[00:44:53.110] – Ethan
How bad was it?

[00:44:54.100] – Ned
I didn’t think it was that bad. But again, I did all this other stuff, so I went in cold and I got through it. I should also say that I helped review some of the questions after I took the exam. So I had some back end knowledge of what was going into that exam after the fact anyway. But yes, it’s definitely passable. It’s not one of those tricky exams where they try to really trip you up and use weird wording and stuff. It’s like either you know it or you don’t. And if you don’t, you can just take it again, yeah, it’s fine.

[00:45:24.630] – Ethan
Yeah, I can no shame in failing a tech exam that’s happens to everybody. All of us did. Oh well Ned. Thank you man. This and again, if you’re listening, we truly just winged that.

[00:45:36.100] I’m guessing you maybe Ned you were looking at some slides or something, help you with some of the details and if not know how you dude you’re a stone cold killer man. That was awesome. You remembered all that stuff just off the top of your head. That was great.

[00:45:48.970] – Ned
Well, I mean, I did just finish the course.

[00:45:52.090] – Ethan
Fair enough. Fair enough. Well, Ned thanks, man. Thanks for sharing your knowledge on Vault. And of course, Ned’s got the course on Pluralsight. If you want to dig in and learn more. Virtual high fives to you if you made it to this far in the show, if you have suggestions for future shows, more stuff you want to hear andNed I chatting about, we would love to hear it. Hit either of us up on Twitter at Day Two Cloud show.

[00:46:13.780] Or you can fill out the form on Ned’s fancy website, Ned in the cloud dot com. If you like Engineering oriented shows like this one, go to Packet Pusher’s dot net slash subscribe. Day Two Cloud is just one of the shows in the packet pusher’s podcasting network. All of our podcast newsletters, our websites, everything’s on that subscribe page is all nerdy content designed for your professional career development. And until then, just remember, cloud is what happens while IT is making other plans.

More from this show

D2C218: What’s Inside The AI Magic Box?

AI and machine learning are being more widely used in IT and elsewhere. Today's episode opens the AI magic box to better understand what's inside, including software and hardware. We discuss essentials such as training models and parameters, software...

Episode 90