Follow me:
Listen on:

Day Two Cloud 142: OpenZiti Serves Up Zero Trust For Applications (Sponsored)

Episode 142

Play episode

Today’s Day Two Cloud episode jumps into the deep end of the networking pool to talk about OpenZiti, an open-source project that brings zero trust principles to networks and applications. OpenZiti builds an overlay or mesh network to enforce zero trust. It has several moving parts including edge routers to enforce zero trust policies when you enter the network, a controller, SDKs to integrate OpenZiti with your applications, and tunnelers and proxies that work with your existing applications.

Our guest is Clint Dovholuk, an OpenZiti developer and zero trust advocate. The term “zero trust” is currently being abused by vendor marketing departments, so Clint makes a case for why OpenZiti delivers zero trust. He also gets into technical detail about how Ziti works.

The OpenZiti project was created by NetFoundry, a for-profit company that sells a commercial SaaS offering based on OpenZiti. NetFoundry is the sponsor of today’s episode.

We discuss:

  • What zero trust means in OpenZiti
  • How to bootstrap trust
  • Dealing with identity in a zero trust environment
  • OpenZiti components
  • OpenZiti SDKs
  • More


  1. Application embedded Zero Trust really is the future
  2. OpenZiti is free and open source – you can install it and run it right now today!
  3. Adopting an OpenZiti SDK really is easy
  4. If you don’t want to host OpenZiti, NetFoundry provides free forever tiers

Show Links:

OpenZiti – GitHub

OpenZiti Repo – GitHub

Ziti Dev Blog

OpenZiti Discourse Group

OpenZiti on YouTube

@openziti – OpenZiti on Twitter

@OpenZiggy – OpenZiggy mascot on Twitter

Clint Dovholuk on LinkedIn


[00:00:04.330] – Ethan
Welcome to day two, Cloud.

[00:00:05.860] – Ethan
We are jumping into the deep end of the networking pool today, talking about the Open ZD project, which is zero trust. It is applicationbased, and it is a lot more. There’s a lot going on here, isn’t there?

[00:00:20.290] – Ned
There is. And I think when people hear the words zero trust, they get a little concerned that it’s going to be a bunch of marketing hype and fluff. This is not that we dismiss that notion right away and then get down into the guts of what openZ does and how it can apply to either your applications or just your general systems.

[00:00:38.810] – Ethan
Our guest today is Clint Dovaloch. Clint is a developer and zero trust advocate. He knows where the bodies are buried in this project. I think he’s probably forgotten more than we even get to cover in this episode. He goes deep and it goes fast. So strap in and enjoy this conversation about Openzd. Clint, nice to have you today, man. I’ve done a lot of homework for this show. Getting set to try to understand this product, Open ZD, what it’s all about. And to set the stage, Clint, you’ve got to tell us what zero trust means to you, because it’s one of those things that’s been marketing wash everything.

[00:01:18.070] – Ned
Zero trust.

[00:01:18.780] – Ethan
So what does zero trust mean from your perspective?

[00:01:21.490] – Clint
I got to tell you, Ethan, that it absolutely is. It has become almost like a poison pill. We don’t want to talk about zero trust anymore. We want to talk about not trusting our network instead. So zero trust is a term that you’ll hear throwing about everybody zero trust. You can’t throw your rock into the Internet ocean without hitting some sort of zero trust product, right? So Open ZD is a free and open source project. It’s available on GitHub, and it’s all about bringing no trust or not trusting your network into your applications, hopefully all the way into your application space. But we have lots of pieces of that overlay network that we call zero trust itself. Not trusting the actual network. How do you not trust a network? You need things like strong identities. You need things like policies. You need things like X 509 certificates, Azure, those strong identities. So for me, zero trust really means trust. No one don’t trust the network.

[00:02:21.890] – Ethan
Okay, trust. No one don’t trust the network. Okay, now, you said free and open source, but here you are, Net Foundry sponsoring the show. So does that mean it’s free and open source? And it’s the version you don’t want that only supports five minutes. If you want something that actually does something, you got to upgrade.

[00:02:40.430] – Clint
Well, you absolutely can go get that version if you’d like. There is up to ten nodes, free version for sure. But no, openZ is totally open source. I was telling somebody recently, I wrote all the quick starts. So if you go out to openZ GitHub. Io, there are four quick starts. One I don’t want to install Docker. One, I love Docker. One, I love Docker compose. And one I’ll run Docker. I’ll run it anywhere and just let me figure it out. Right. So those quick starts, you can absolutely go run and take right now, if you are relatively competent with a Bash shell and you have some server in the cloud somewhere, you can go host your own right now, make your own virtual private server in no time and no limits on this thing.

[00:03:23.490] – Ethan
It’s just truly free, open source, as you’re saying.

[00:03:26.120] – Clint
Okay. Yeah. Totally free and open source. So it’s the open core model. So Net Foundry has the offering, but you can absolutely run it yourself if you want to run it yourself. There’s a little bit of work involved. Obviously, you get to set up servers, you maintain it and that sort of thing. There is a UI that comes along with it. We call it the ZD Administration console, the Zack. So you don’t have to be a CLI Ninja, but we do have a CLI as well. And so you can go out and install all this stuff. I am the open source shill for Netflix, which means I go out and I sell the open source stuff. I don’t sell the pay for product. You can go get it. Go get it if you want to pay for it for sure. Because not everybody wants to run their own network. Right. And maintain it and have staff that sit around the clock monitoring your network. But you can absolutely run the whole thing yourself.

[00:04:13.880] – Ethan
Clint, you’re not an open source show. You’re an open source evangelist. Come on, man.

[00:04:18.000] – Clint
You got. Oh, I’m sorry. You’re right. Speaking of the vROps, right. So there is also our mascot, Ziggy. If you go out to Twitter and you want to follow Ziggy, you can one of the things one of the hashtags that you’ll see Ziggy use a lot is closed. Kimono. His Kimono never opens. No ports are ever open on an open ZD network. Oh, boy.

[00:04:44.690] – Ned
I have to say that I really appreciate that you’ve included Aui in the open source, the core version of it. That is something that is often overlooked or only available in the paid for version. That’s considered an add on an enterprise feature. And as much as I like using the CLI, I don’t always like not having UI to fall back on or just get a visual representation of what’s going on 100%.

[00:05:10.380] – Clint
Sometimes being able to see how all those little pieces of your policy puzzle go together really comes. It steps in place when you can see it in the UI, when you’re looking at it in a CLI, you have to have your screen big enough. You have to know the right incantations. You can’t really explore. You can explore by just typing the same thing over and over again. Zd space, Edge space list, enter, and then you can do that over and over again. But if you have a UI, you can just go there and play all these are all the things that are available to me. Oh, what’s an edge router. What’s a policy. You can explore that way.

[00:05:47.090] – Ned
One of the things you mentioned in your definition of zero trust is trust. No one like I have to trust something, right?

[00:05:56.830] – Clint
Sure do.

[00:05:57.860] – Ned
That’s an opponent that I’m trusting. In the world of ZD.

[00:06:01.650] – Clint
That’S a great question. In fact, it’s such a great question that one of our engineers wrote a five part blog post about how to bootstrap trust because it is a really complicated procedure. At the end of the day, what it comes down to is there’s always somebody. There’s always ideally, I think a human that sets up some sort of trust chain in an open city network. It’s going to be a PKI public key infrastructure. When you run those Quick starts I talked about, without a doubt, the single hardest part of setting up an open CD network is setting up that whole trust chain. When you do that, you’ll get this thing called the controller, and that controller will be seated with some trust. It will be a self signed certificate out of the gate. You can absolutely use a legitimate certificate if you are well versed enough with how to actually create a PKI. But you don’t have to. The Quickstart will do it all for you. Then you’ll have a certificate that you know that you made. And if you can’t trust yourself, well, then that I don’t know who you can’t trust.

[00:07:07.070] – Ned
That’s a valid point. As someone who’s had to stand up a few certificate authorities in the past and a whole PKI infrastructure, that’s sort of redundant. But yeah, set up PKI. It is a lot of work, especially when you’re building that root CA because it has to be done offline. It has to be stored in a special location. Some people would lock away a hard drive that has the key in a physical safe. But we don’t need to go that far.

[00:07:35.750] – Clint
Also, don’t forget the hardware root of trust, right. If you wanted to bring that into the equation, you can certainly do that as well. So that whole PKI can absolutely get very complicated.

[00:07:48.170] – Ned
I know, I just laser etch the entire string onto a piece of titanium and various backyard because I’ve heard that’s very secure and mail it to somebody.

[00:08:01.470] – Ethan
But Glenn, what are we assigning this identity to in an open ZD network? Is it like a firewall or router or is it an application? At what level are we assigning identity?

[00:08:13.540] – Clint
Yeah, that’s a good question, too. Identities are literally at every level. So when you create a zero trust overlay network, first of all, you’re going to make that controller. First thing you’re going to have to do after that is you’re going to need a piece of the puzzle called an edge router. The edge router is what establishes the very first part of your mesh network. So openZ is all about mesh networking as well. You can create multiple edge routers. To create the fastest path amongst the Internet, we used to have an engineer who would refer to that as the Internet weather. What will happen is, if the weather is bad on one link, it will route your traffic to another path. So you’ll need those edge routers. Those edge routers themselves all have identities of their own. And to create that identity, you need to bootstrap that trust. You’re going to basically create a certificate signing request, and that CSR will be sent to the controller and get signed. So the router itself, when you turn it on, it will create a private key. That private key will then generate a CSR. Csr will go to the controller that comes your certificate.

[00:09:17.890] – Clint
Now, the controller has certified you. It knows what fingerprint you’re going to be presenting and knows your certificate. It can perform mutual TLS between the edge router, the controller. The same can happen when you add that second edge router. Now, those two edge routers form a mutual TLS link as well. So that’s your initial basis. That’s your first, say, two identities, those two edge routers, then every single device you add has a requirement of having that same strong identity. We have things that we call tunneling applications. They Azure, effectively VPC clients, but better, obviously, because there’s zero trust. And those tunneling applications in themselves are capable of handling multiple identities. So you might be participating like I am. I think I have four networks that I’m participating in, one of which is our bastion network here at Netflix. You cannot access our production environment without going through a bastion. Well, we just wrote a blog post about taking our bastion dark. What does it mean to have a dark bastion? Doesn’t the Bastion have to be listening on the open Internet? Well, you don’t need to with the zero trust overlay network, so that client will have an identity or multiple identities in it.

[00:10:26.110] – Clint
And then the next step is actually, if we can get it, is to install that identity all the way into your application itself. That’s really the end game of the Openv project is to get that zero trust goodness. All the way into your app. But that’s a journey. So getting there, we have lots of steps along the way. Every single piece has that strong identity.

[00:10:47.490] – Ned
Okay, you mentioned a lot of different components there, and I want to dig into some of them. So I’m assuming the controller, that’s sort of the control plane. That’s what establishes connections with the different edge routers or other components and pushes, like, a policy to them or something along those lines.

[00:11:03.700] – Clint
Yes, absolutely.

[00:11:05.010] – Ned
Okay, so the first thing I got to do is stand up that controller, right?

[00:11:08.270] – Clint
Yeah, exactly.

[00:11:10.410] – Ned
Okay. And then we got the edge routers. What are those typically running on? Are they running on a virtual machine, a physical server? All of the above?

[00:11:19.210] – Clint
Yeah. They can run on really garbage equipment. So when I run these myself in a Dev environment, when I’m developing in the cloud, I will use the free tier. And the free tier is almost always sufficient for all of my development needs. If you are pushing lots of bites across, you’re going to end up hitting limits. So you need to get something that doesn’t have a limit on how many vROps you can do. But generally speaking, these run on the minimal machines. So two CPU, four gig of Ram is more than adequate for, I would say most tasks until you start pushing lots of data. And then you’re going to be CPU bound and you’re going to be want to watch in the CPU, the network. And that’s really where it’s not memory constrained. It’s going to be CPU bound.

[00:12:06.690] – Ethan
Now, the edge router you described edge routers as forming a mesh. They end up with mutual TLS connections to each other, which isn’t immediately meaningful to me. Mtls because TLS can be applied to a lot of different protocols. So what is the transport protocol between these edge routers?

[00:12:25.500] – Clint
It’s TCP. So it’s going to be making a TCP connection that establishes a secure connection, both from the client’s perspective and the service perspective. The M stands for mutual TLS, so both sides will have to negotiate an actual secure socket connection. So it’s going to be TCP.

[00:12:46.170] – Ethan
So it’s a TCP wrapper around the original packet that’s being routed through the edge router mesh.

[00:12:51.420] – Clint
Yes. What will happen? We haven’t talked about onboarding package yet. So let’s talk about that client that we call the tunneler app. Right. The thing that runs on my local machine, because what will happen is you’re going to want to go to one of the cool things about these tunneling apps is you can just have private DNS, too. Really cool aspect of ZD. For example, I have a client that runs Matter most. Matter Most is a chat app, very much like Slack, except our chat app is dark. You can’t attack our chat app unless you’re on the open CD network. You can’t even see it unless you’re on our actual network. That is for matter most. So in order for me to send bites across the network, I have that little agent that runs locally, and it will intercept those IP packets. The job of our tunneler is basically to wrap that packet in a tiny bit of metadata, because the client will also connect to that edge router. That’s important. So the client has formed a mutual TLS connection to the edge router as well. And now there is a path for the data bits that my client locally has absorbed to send them along that mesh.

[00:14:00.780] – Clint
So we’ll wrap it in a tiny bit of metadata that basically says I need you to send these packets to this place. And then once those packets get to whatever that place is, there’s a process that says these packets are destined to go to mattermos. Netfoundry IO. I need to now create a legitimate socket and send traffic to Mattermost whatever. Io. So that’s really useful in a lot of situations. I talked about Matter Most, but something like Jenkins. For example, if you wanted to have a private Jenkins server, you would be able to have a private Jenkins server. So the usual TLS is the first step of that path. The second step of that path is the controller will decide where to send those bytes. So that’s where the mesh comes in. You say send my bites to Matter most, and the controller says the fastest path right now. And it does that by latency. It checks all of your links, adds them all up, add some dynamic stuff, sprinkle some magic fairy dust on there, and say this is the fastest path. Right? And it sends your bites over that fastest path. And if that fastest path changes, it reroutes you.

[00:15:04.780] – Clint
And as long as you don’t lose your edge router on either side, all those bites are just magically sent across the fabric to the other side as fast as they can get there. I think I answered the question. And then there’s also end to end encryption along this line, too. So the very first thing that will happen is we use Lib Sodium in our SDK. Lib Sodium is well known for being able to run on small devices. So we have a Csdk that you could use and build your own apps. This Tunneling app is written around that Csdk, and that Csdk uses lipstodium. Lip Sodium uses Chacha 20 Poly 13 five, I think is the actual algorithm, which is really good algorithm for low power devices. So you don’t need a tremendous it does not consume a tremendous amount of CPU. You can run those little agents on, say a Raspberry Pi or whatever you wanted to put it on as far as the source and the destination. Also, doesn’t matter what you’re running on. And encryption is there. And then on top of that, if you happen to ride an actual secure protocol, like if you Azure Tunneling TLS, you basically would be encrypting this traffic three times.

[00:16:16.540] – Clint
One’s for Https, one is for end to end encryption.

[00:16:18.930] – Ethan
And one is for that per link encryption usually not the most efficient thing to do to encrypted data, to encrypt it. Again, usually that Colo be a negative.

[00:16:26.730] – Clint
But it definitely is. But we take that secure layer really seriously, right? So there’s no chance you can compromise one of our routers and read those packets, because encrypted, that’s really important. There’s no way that you can sniff the link traffic because it’s mutual TLS. That’s really important. So if you want to run FTP, then you can run FTP, but you can do it securely over and oversee the network.

[00:16:49.730] – Ethan
Now, you said the controller programs fastest path. I assume it basically tells all the edge routers. This is fastest path right now. And so as a packet is hopping through the mesh, it doesn’t have to get intercepted by the controller or anything.

[00:17:03.340] – Clint
No, there are routing tables at every single router. Yeah.

[00:17:07.390] – Ethan
Okay. Now I’m a packet nerd, Clint, so I couldn’t help myself but ask you those questions. But practically speaking, if I’m running an Open Zeding network, how much of everything you just described about how packets get through the mesh do we actually have to care about?

[00:17:23.950] – Clint
I would say this is the best part of openZ. So if you have one of these tunneling apps, and if that app is legitimately absorbing some of your traffic, and if you have that mesh network, all you really need to care about is where you want to send it, and then the packets will get themselves there, however, is the fastest way at that moment. And we’ve done plenty of tests where we’ll beat OpenVPN by a bunch I don’t want to dump on anybody. And then lots of other tunneling type technologies, it’s at least as good as that. And then because of that, smart routing, where it comes really interesting is the Internet. Weather is a real thing. It’s shocking how that’s a real thing. And so you might think that going from New York to Chicago is always fastest through Ohio, but sometimes it’s faster to go through Tennessee. It just doesn’t make any sense. But that’s the case. So you don’t have to think about too much of those packets when they traverse. You do need to know how you’re going to onboard them and where you want to offboard them. Then where that becomes magical is when you’re in a multi cloud environment.

[00:18:28.770] – Clint
And so if you have packets that you want to tunnel from AWS into Azure or Oracle or even your own private data center, because everything in the open Sea network is all based around the fact that you are outbound only to form the first connection. So that tunneler will create a connection out to those edge routers, which they don’t have to be in the public. If you were like a tier one telecom, I used to work for a global crossing. If you remember that company, tier one telecom, if you wanted to run your own internal private, then it’s not cloud. But if you were part of that’s as cloud as you can get, right? Like that’s, the Internet. So if you were part of the actual Internet, you could keep all those things on Net and have that ability. But because it’s all entirely transparent to you, you just get it on the open city network and it just offloads them wherever, regardless of your cloud, regardless of its Kubernetes, regardless if it’s private data center, regardless if it’s my laptop right here. Like we can tunnel to my laptop right here. You can SSH onto my Windows subsystem for Linux.

[00:19:28.600] – Clint
It’s so easy.

[00:19:29.750] – Ned
I’m still trying to connect the dots a little bit about how the traffic actually flows from one of these applications and finds its connection on the other end because you’re saying it’s all outbound traffic, but something somewhere has to be listening, or at least be able to be contacted by that app to form that link. So can you dig a little deeper? Because I’m missing something, I think.

[00:19:53.260] – Clint
Yeah, those edge routers. So generally speaking, every single CD SDK client of which the tunnels are an SDK client, every single one will have to rendezvous somewhere. Since they’re all going outbound, there has to be somewhere that they’re going. That somewhere is called an edge router. And so there are edge routers that are out there in the cloud, almost always listening on Port 443 that require you to present a certificate before they allow you to connect. So that’s a core tenant of that zero trust idea is authorized before connecting. So if you try to connect the Port 443, you don’t present a certificate. You do not create a connection. It’s that simple. Also, since it’s Port 443, because of the way Openzd is basically creating that synthetic connection, everything looks like Port 443. So Port 443 and 80 are almost always open outbound everywhere. Some places are a little bit more strict about it, but all you need is 80 or 443 outbound and you can create an open CD connection. You do need that one machine that’s out there listening or in case of DDoS attack, you can have ten of them if you want to have ten of them.

[00:21:04.220] – Clint

[00:21:04.610] – Ned
And that’s forming that mesh that then the packet will ride to whatever the destination is.

[00:21:09.240] – Clint
Yes. Mesh also high availability. Right. So you can’t have one of them go down. You need to have a few of them. Okay.

[00:21:17.920] – Ned
And the list of edge routers that’s provided by the controller to the client when it checks in.

[00:21:24.020] – Clint
Yeah. This is also a really neat thing. So openZ allows you to decide what routers you can connect to. So I could say, hey, Clint is allowed to connect to the New York City router. Ethan is allowed to connect to the Los Angeles router. Ned can connect to the Florida router. And that’s the only one that you’ll end up connecting to. You can have your policies also define what services. We call them services that you can then access once you’ve connected to that overlay network. We haven’t even gotten into services yet. We were talking about pushing the package, which is cool. But behind the scenes, those packages are going somewhere. They’re going for that application. And openZ is really awesome about limiting what you can connect to. So it’s not an IP address. It’s not purely layer three. It’s really more like three because the Port is important too. But it’s not exclusive to the IP address. I guess it’s more like layer four, I guess, because TCP, whether it’s three or four, you need to know what Port your also destination is going to, because if you’re trying to send, say let’s do SSH traffic, which usually is Port 22 and you don’t have a service listening on Port 22.

[00:22:37.700] – Clint
It goes nowhere. It doesn’t get absorbed, it just gets dropped. You don’t even know it’s there.

[00:22:42.210] – Ethan
But we’re functioning up in the network stack. We’re not down in the kernel like we’re say EBF might be. So I can’t see what’s going on within an application. Is that right, Clint?

[00:22:52.160] – Clint
You cannot. Well, it depends on if you wrote the app. Right. So EBP is a little more of an interesting scenario. It would act sort of like what our tunnelers do, where basically it’s job would be to intercept packets and then somehow transport them to their destination. That’s basically what our tunneling applications do, just not specifically using EVF. I don’t remember exactly the details in the Linux world because usually you think eppflinux, but I believe we have a couple of different modes in our tunnelers which are supported, and I can’t remember if it’s a ton or not ton. We’ve gone through a couple of iterations on that. But effectively we need to be able to absorb those packets. You ask if you can see those packets. If you wrote the application, then sure, you could see those packets because you’re receiving them. The Open ZD tunnellers, they absolutely can see those packets, but it’s all running locally. It’s also all open source. I hate saying this because there’s a lot of code to go look through, but you could always go look through the code, right? Like that’s the classic vROps. We’re in the open source for a reason.

[00:23:59.470] – Clint
I like to say how can you trust a security product that’s not open source? Because you want to be able to go and inspect it and run your own code. Due diligence, I guess.

[00:24:08.520] – Ethan
I’m thinking of what different kinds of inspections can we do with openZ. So we’ve got this filtering mechanism, this very rich filtering mechanism that you’re describing. But then if it’s a proxy, let’s say if it were a proxy, you could do something like bringing an encrypted stream decrypted because you have the private key, do some inspection, reencrypted send it out the other side, as opposed to watching the packet fly by and running a filter against it.

[00:24:39.740] – Clint
All you know is Port 443. You don’t even know if it’s is it really SSH in there. Is it really FTP in there? You don’t know. All you see is 443. Right now with an open CD network, you would have no visibility. And that’s very much by design. Openz is an SDK. It is at its heart a bunch of SDKs of which is a Csdk. You could absolutely go. And if you wanted to implement, let’s say you’re out there listening and you have an appliance that you sell to people in the cloud and you want to be able to give them zero trust appliance. You can absolutely take one of our SDKs, Bake it into your software. Like maybe it’s a web application firewall. Right. Common thing looking for SQL injections or whatever you’re doing with it. If you wanted to have a truly zero trust dark connection and you were the provider of such a product, you can take the Open Cdsdk, Bake it in, and then tell your customers that you can have a zero trust connection now. So out of the gate, ZD provides you none of that ability. But because it is free open source software, because it is based around SDKs, you absolutely could take it and incorporate it into your product.

[00:25:51.910] – Clint
So, for example, a zero trust load balancer. Right. I have a ton of these things that call Zidifications. We got lots of ideas, one of which is NGINX. Right. We started Zidifying. Nginx. What does it mean to have you talked about a proxy? Right. What does it mean to have a zero trust engine? X? What does it mean to have a zero trust load balancer like Hoxy? What’s really interesting is if you have an Openzd overly network and openZ is also built around high availability, it’s also built around high scalability. So if you want to, you could absolutely create a couple of redundant paths to machines on the other side. So if you wanted to have that load balancer capability, ZD basically provides that for you. Already we have round Robin kind of load balancing. We have smart load balancing. You can cost the links if you want to send them to the primary node and not to the secondary node, because you need to send them all to the primary node. You can have that kind of control over your network with the Open Seating network.

[00:26:53.870] – Ned
I want to compare this to an API gateway because some of the functionality you’re just mentioning. And if we’re going to take the idea of a load balancer and maybe expand on that a bit, that’s kind of what an API gateway functions as, or an ingress controller, if we’re talking about Kubernetes, because those two concepts also seem to be merging a bit. And the idea is I can do stuff at layer seven, I can inspect the Http request, I can make modifications, I can add headers, I can log things if I want to because of the way that openZ sounds like it’s working. I can’t use my existing API gateway with this because I don’t have access to that to get into the packets anymore and the requests. So is Openc going to provide that functionality or can I decapsulate or whatever before it hits my API gateway?

[00:27:44.030] – Clint
Yeah. Generally speaking, the first question I would ask when I talk about API gateways is can anybody on the open Internet hit your API gateway? And do they know where it is? Will they be able to discover it? Generally speaking, I think the answer is yes. Right. Usually there is a secure gateway that listens on the Internet on a well known address. So it is quote, unquote available. Right. So it’s available for attack with an open CD network would not be available for attack if you would have to know where that edge router is of which you can have numerous of them. Right. So it’s not a single solitary endpoint. They’re all discreetly hidden from probing eyes, if you will. You could find them if you were really diligent. But if you kept around Roberting your own edge routers, then I suppose you could defeat that as well. But it is similar. So you mentioned a couple of things like Kubernetes and providing ingress. That is another area where I think ZD really shines, because you can take an Open ZD Helm chart and install a pod, basically, which will provide you that zero trust ingress into your Kubernetes space.

[00:28:56.500] – Clint
So if you wanted to be able to have zero trust access to Kubernetes without having the classic Ingress controller that actually is out there and available in listening, but you can do that with Open Z. That’s a little bit different than the classic API gateway. As far as the metrics and whatnot that are involved, the controller of Open Seating network will emit a rich set of metrics. So if you wanted to be able to see who was sending data to what service, at what time, how much data you can go and get that sort of information as well. In fact, this is to go back to the Net Foundry sponsored podcast. This is where the Nas stuff comes in, because you’ll get those nice charts and graphs that you don’t get that in the open sea stuff because you’re going to have a data Lake somewhere to stream all this data to. You have to be able to run a query using Elastic or whatever you want to use.

[00:29:53.270] – Ethan
You’ve broken my brain a little bit here in this part of the conversation, and it started back in your comment about NGINX. I admit some NGINX servers myself, but they’re all public facing. I got a firewall that sits in front of filters on Port 80 and Port four, four, three. If the inbound traffic is on those ports, yeah. You can go to the Engine X server and make your requests. If I were to re architect this and involve openZ, what does that look like?

[00:30:16.670] – Clint
Well, it depends on if you are trying to provide public access to those things or not. So that’s like number one, right?

[00:30:23.640] – Ethan
Let’s say yes, public access.

[00:30:25.530] – Clint
If you want public access, then I don’t think you trust is for you. Okay.

[00:30:30.410] – Ethan
That’s where my brain broke.

[00:30:31.870] – Clint
Like, wait a minute, I don’t think this is if you need a random person to be able to go there, then you probably are not looking for what zero trust provides. However, if you are, say, a company who wants their employees to get to that thing from wherever they are in the world, you are absolutely in Open CD zero trust territory. Right?

[00:30:54.020] – Ethan
Because now I can hang it off in public space, not worry about general Internet getting to it by internal customers that can connect. Now they can get to it.

[00:31:03.750] – Clint
Yeah. And that’s really where it shines when you want to do that sort of stuff. I use the example. I don’t know if it was before, but I love using openZ because without changing a firewall in Amazon, I can pick my laptop up and I can go to Starbucks, or I can go to my mother’s house or whatever, and I can SSH to all my resources because I have a zero trust connection that is not reliant on my source IP address being the current one I’m sitting in right now. And then if you think about Whitelisting, whenever people actually go back to this thing that I think they used to call the office. Right? So if you have people who are SSHing from this thing called an office, then you have to whitelist that IP address or you have to have private connectivity into that office. It gets to be a mess with openZ. I like to say the Internet is your land, really? So you treat like everything like it’s on the Open Internet, have a totally blocked firewall. Don’t let anything in. Establish all outbound connections, and it doesn’t matter where you go.

[00:32:07.910] – Ethan
Okay, let’s actually flesh that out a bit. Do I have an Open ZD client on my Mac or my Windows machine or something to gain access?

[00:32:15.770] – Clint
You absolutely. Could I have one on my Windows machine? We have them for all of the major operating systems, including mobile devices. So we call them the ZD Desktop Edge or the ZD Mobile Edge for Android, for iOS, for Mac, for Linux, for Linux is a thing called the ZV Edge Tunnel, but you can get them for all the major operating systems, and one is available in the store. The Windows one you can get from GitHub. We have a place where you can install it from. There are some challenges when it comes to being in the store, particularly for Mac and Apple products. You really need to be in the store, and those environments all are somewhat more conducive than others to zero trust. Some want to maintain that control over your network more than others. So Apple’s ecosystem is a little bit more locked down, so you need to get things from the store. Linux is much more of a land grab. You can get it from wherever you want to get it from. But we’re working on a package manager. We don’t have them available in package managers just yet, but we’re actually working on deploying them to Ubuntu and Fedora and whatever your favorite distro is.

[00:33:23.290] – Ethan
So this is starting to feel more like a tail scale or zero tier. Is that a fair comparison to make?

[00:33:29.120] – Clint
Yeah. In fact, lots of people ask the exact question, how are you different than how are you different than? So zero tier, from my understanding, it works a little bit differently. I do believe they’re doing UDP hole punching. If I checked the last time around, I think, and I believe Wiredguard works the same way. So it’s a little different in that regard because the ZD connections are all entirely outbound. They’re also TCP, not UDP. We have looked at using UDP in the past. Obviously, there are some benefits of having a transport that is based around UDP over TCP. Generally speaking, from the testing that we’ve done, it’s not been anything that’s been particularly noticeable. But also zero trust is not about low latency. A lot of times it absolutely can be low latency. But if you’re thinking Fintech type stuff where microseconds matter, those three layers of encryption might impact that microsecond that you care about in the human scale, I never notice it. Right. I don’t even know when I’m using ZD, but the computers that probably could notice in that level. Okay.

[00:34:42.000] – Ned
So I could install a client on my laptop and use it, and I would be able to connect to one or many ZD networks. That’s something you indicated earlier. How do my applications know which ZD network to use?

[00:34:58.240] – Clint
Yeah, it’s all up to you. You get to control that. So those identities all come with a bunch of meta information that we call services. And so those services basically declare what so let’s say we’re going to use a tunneler because that’s where people will always start. It’s easier to start with a tunnel, and it’s approachable. Right now, you can go and you can take whatever application that you have that you want to trust and turn it zero trust by bringing one of those tunnelers and installing it. So let’s say you did that. And let’s say it’s SSH just to make it easy. Right? I want to SSH from my local laptop to my virtual private server in the cloud, and I want to be able to maintain that thing. So basically what you would do is you would go into that controller and you would define a service that says, hey, Ned is able to SSH to virtual private server, and that would be what you would do. Now, the definition of the service is not the authorization of the service. So you then have to create a thing called the policy that says Ned is allowed to dial SSH and then VPs is allowed to bind SSH.

[00:36:06.810] – Clint
And then basically Ned would be able to create a service that gets Ned into the SSH server. That would be what you would have to do. There’s a lot of different camps on this one because we also support things like Cider blocks. But Cider blocks are not particularly zero trust. Right.

[00:36:23.480] – Clint
We talked about not trusting the network at the same time. Reality is you might need to have access to a certain address space. So we understand that. So if you wanted to intercept the Cider block, you can intercept the Cider block, set it to the other side and offload the Cider block if that’s what you want to do. So we support those sort of things too, because like I said before, it’s a journey and you might start off like that and then go, you know what? That’s kind of a little too much access. It feels too much like a VPN. Let me actually turn it into not a VPN. Let me lock this down to one machine with ports one through 65, 535 open. And then you go, maybe I shouldn’t have all those ports open. Maybe it’s just really this four.

[00:37:03.570] – Ethan
It still feels all five tupoli, though. I mean, is there some kind of tagging metadata that I can group things on?

[00:37:09.960] – Clint
Sure. So if you wanted to say anybody with an attribute of SSH able can perform an SSH action and can dial that SSH service, then you can assign an attribute to these identities and that attribute is attribute based access control as opposed to role based access control. So when you define these services, you can add these attributes that basically say, hey, if an endpoint shows up and that EndToEnd is declared with SSH clients on it, then all of a sudden it’s capable of performing SSH. Also, we haven’t even scratched the surface of what you can do with Openzd. I didn’t get into third party certificates and that sort of trust. You don’t even have to trust ZD certificates. You can bring your own third party certificate and slap it into the controller and say, listen, I don’t even trust your certs. I’m going to use this Cert that I’m providing to you, and it’s only the certificate that’s not the private key. So you can say anybody who shows up with a certificate with a Cert that’s valid for the CA, you can trust them. So there are loads and loads of features on the backside of Opens that we haven’t gotten to.

[00:38:17.860] – Clint
Attribute based access control is absolutely one of them. All the policies are a back. So if you wanted to control United States people can use the United States routers. You can make that policy, or HR people can use the HR services, you can make that policy. Right. So the policies are very flexible, attribute based control as opposed to role based control. That way you don’t need to know what your roles are ahead of time. You can just add a tag to it and then when those end points show up, they’ll just be able to have those services and it all comes down to your client immediately, within 15 seconds, there’s a short delay. But you’ll see those service mesh show up and all of a sudden instead of 52, now you have 53 services you’re like. Oh, that’s neat. What was the new service I now have access to? Oh, it was Jenkins. We’ve hidden Jenkins.

[00:39:08.610] – Ethan
Okay. So as my head is exploding here, as I think of different use cases, one thing that is fairly obvious, I suppose that this crowd would be multi cloud network routing overlay. So I’ve got a bunch of stuff and a bunch of different clouds and I can stitch them all together using openZ and apply zero trust to all those resources. Is that a plausible scenario, Clint?

[00:39:30.910] – Clint
Totally is possible. And really one of those situations that the openZ really shines. So we have a situation where at least one Open CD customer, I don’t know if I call them customer, but their customer, they will have their SaaS platform running in Amazon, but they need to be able to monitor things that are running in Azure or Oracle or wherever they might be because of the overlay network, because of that zero trust, because of that punching out capability, because of the mesh, they don’t really even need to worry about wherever that their client software is deployed into because it only needs outbound internet. So as long as they have outbound internet and as long as they’ve set up the policies right when their agent comes online, they have secure connectivity right into their monitoring system. So yeah, multi cloud is I think is a real strong suit for Open City. Yes.

[00:40:26.040] – Ned
Someone who has had to stitch together multi cloud in the past. It’s a headache. It’s a real headache. There’s a first mover issue with a lot of their built in VPN solutions and plus it’s a VPN solution. So it’s very broad and it’s Cider based, whereas what you’re talking about is much more just an overlay and you have significantly more control. So I do like that. And I like the fact that I’m not adding a whole bunch of public IP addresses to my infrastructure that have to be static. To what degree is the whole system automatable? That’s the word I just made up. Can I Gray word.

[00:41:05.090] – Clint
It’s a great word. So the entire thing is driven by APIs. So if you are capable of issuing a Rest request, and if you are capable of formulating JSON of the correct flavor, you can absolutely poke and prod the controller and make it do what you want it to do. In fact, the CLI the Zdcli that you would use to create edge routers. In fact, if you look at the quickstarts, you’ll see I use a ton of Zdcli commands inside the shell scripts that you can download. All those Zdcli commands all interact with that Rest EndToEnd and that Rest endpoint. Here’s another great question. Can you take that management API off the public internet? And the answer of course is yes, you can. So if you wanted to be able to protect that actually, because that’s like the Kubernetes API, right? That’s the keys to the Kingdom. If you can get access to that and you have the right username and password, there’s a lot of Hoops. But if you jump through all those Hoops and you have that access, then you can define a new identity and you can say, hey, this Ned guy can now go and get on Ethan’s network, and he wasn’t supposed to be able to do that, but now he can.

[00:42:10.200] – Clint
You can do all that through an API. It’s a rest based API that is exposed on the controller. Currently we’re working on multi controller support, so in the future it’ll still be API based. It will just depend on where you actually send your quest to.

[00:42:24.490] – Ethan
Well, is this kind of how you envision openZ being consumed, typically by a developer who wants to Bake zero trust into their applications and so they’re just going to be consuming openZ? Or is it more of an Ops oriented tool where infrastructure professionals are going to manage it?

[00:42:41.230] – Clint
Yeah, I think that it really depends on your particular use case. So one of the more stronger arguments that we have here at Foundry is actually from our lead site reliability engineer fellow. He’s also DevOps. I don’t know what term you guys prefer, but he goes by SRE, right? So he likes to say that ZD is a tool that he has adopted into his toolbox because it lets him do the things that he’s supposed to do and yet securely. As the fellow who’s DevOps related, Ops related, he’s oftentimes punching holes in firewalls, providing access to things that the business needs access to. But he has to do it through some sort of policy, some sort of rule somewhere that he has to maintain. It’s not one pane of glass oftentimes, and it’s on the open internet. So with ZD, he can just do all that and migrate people without them even knowing it. Because of the way the title apps will allow you to define any DNS name, you can effectively shadow a DNS and then deploy your tunnel, deploy your DNS name. Notice all the again, back to those dashboards and the pay for products.

[00:43:51.500] – Clint
Notice all the traffic is sending. I’m going to keep using Jenkins because this quintessential example, Clint is sending his traffic to Jenkins. I can see that Clint is accessing Jenkins. I can see that Bob is accessing Jenkins. Great, everybody’s accessing Jenkins. Now I can take Jenkins off the actual internet and nobody even notices. How many times have you migrated something and had the user base not noticed? He made a joke. He did a nice video, you can go find it on our YouTube channel. But he made a joke about how he turned off Access and expected the first two days to be just a firefight storm while he goes and fixes the things that were broken. But he said nobody complained. It was that point where he was himself totally sold that openZ was for Ops people. Now if you’re a developer like me, and I can tell you there have been times in my life when I have deployed an application and I thought I was listening to Port 443. But the Ops person decided that you were going to listen to Port 84. 43 and I’m going to expose Port 443 out in the cloud, right?

[00:44:51.860] – Clint
It’s like I was going to do a little Port switch and not tell you the developer. And then I had some bug in there where I was listening to my portfolio 43 or whatever. So my stuff wasn’t working. I can now declare my entire network. I’m in control as the developer. So one of the things we like to say is you will actually let you put the developer in control of those things, which is maybe scary because I’m a developer. I don’t know if I trust myself sometimes. I get that right. So it’s maybe scary. I can understand that. But once you see how it all hangs together, I think then you start to feel a little less concerned. Because really that application is deciding who is supposed to access their service. And so you only need to get that application the ability to modify whatever services it needs to modify or create an identity, if it needs to create an identity, once you’ve done that, then those applications that I’m deploying, it’s my job to get that strong identity down to them, or to enroll them, to actually make them do that, to create a strong identity.

[00:45:55.670] – Clint
And then all of a sudden it’s just easy for me. I can just say, hey, go take the app and use the app. And then you don’t need to worry about logging in. You don’t need to worry about the zero trust. So it really is for both. And it depends on your use case for developers, you can absolutely take this, automate the whole thing to the Nth degree, take total control. We have plenty of customers who actually do that. Both the openZ stuff and the NetFoundry APIs, they’re both all API driven.

[00:46:23.910] – Ethan
Clint, dude, my head is just exploding. There’s like another hour. There’s 6 hours worth of content to continue chatting about this. Clint, to thoroughly get into this, I think maybe we’ll just call it here for today help bulletize this conversation for us. There Azure, some takeaways, some things you want to leave with the audience as they think about openZ.

[00:46:47.790] – Clint
Yeah, well, first I’m happy to come back anytime. The kinds of takeaways that I think are really important to me. Application embedded Zero trust. I honestly do believe it is the future. You look around and you can’t go a day without a zero day exploit. You can’t go a day without seeing somebody else was breached. Right. Like getting that trust all the way into your application space, I think is really what openZ is all about and what people should really like. That’s the mind blower for me the first time when I realized that you could put that trust not only into the client either, but also into the server. Lots of times you think about that safe zone in the cloud, right? My VPC is safe. Nobody’s going to be able to attack me in here so I can get to the database. Oh yeah, but nobody is going to do that, right? There’s no such thing as log for shell. It’ll never happen. So application embedded. Zero trust, I think really is the future. And of course, since I’m working here for openZ and Netflix, openZ is free, right? Like you can go out and you can get Openzd today.

[00:47:56.230] – Clint
You don’t need to pay for it. The quick starts are out there. I was telling somebody on our discourse, I think it’s discourse where I said I wrote the quick starts. So if they’re no good, I want you to yell at me because I want to make those quick starts so easy that anybody can absolutely host it themselves and put it in the cloud. That’s the best place to put it. Because then wherever you stand up that edge router, it’s going to be out in the public and you can put it on whatever Port you want to put it on. Right? Like that quick start will take you through that sort of thing. If you’re a developer like me, openZ the SDKs themselves. They’re really approachable. So if you take a look, go out to our YouTube channel. Unfortunately, we don’t have a snazzy place. You can just go search for Open. Zd. Openzdi has pretty good SEO, which is nice. Don’t look for ZD, look for openZ because you’ll get lots of different stuff. If you just look for ZD, they get all kinds of recipes that you never thought that you wanted, but you’ll be like, man, why am I hungry now?

[00:48:52.360] – Clint
So if you go out to YouTube and you look for openZ, we get a bunch of videos on what it takes to actually adopt a zero trust SDK. I do a pretty good overview. We can go look at from Open Source 101 out there. I did it recently that takes you through what networking looks like now and how you can superpower. You can give your app superpowers by just adopting that SDK. And it literally is, I want to say ten lines of code for using the SDK itself. And then if you don’t want or you’re not able to Bake AKS the SDK into your applications, we have those things called tunnels, which you can start right now, right? Like you’ve got whatever that cool enterprise app is that bosses on your back about being exposed on the internet and you don’t want to get hacked. Well, you can add that extra layer of defense, take it all entirely offline, take it all totally dark and run those tunnelers. And then of course, we already hit on it before. But if you can’t or don’t want to host your own Open CD server, you don’t have to.

[00:49:57.830] – Clint
Since it is a sponsored podcast, I’ll put that in for you. You can always go and buy your own network from Netflix, but Netflix also provides a free tier, forever free. If you’re a maker and you want to expose ten devices on your home network, we’ll give you that for free already. So those are probably the main points that I have from the takeaways. Thanks, Clint.

[00:50:21.210] – Ethan
All good stuff. And I like that your point about application embedded zero trust really is the future. Insofar AKS I patched a server on Friday night and Monday morning there are 20 new patches, 18 of which are security patches. I am not making that up. That just happened to me this week and I’m forever annoyed and always chasing that stuff. Clint, if folks want to give them some landing pages, places they can go, and so on.

[00:50:46.820] – Clint
Yes. So Openzd GitHub. Io that’s the Doc’s page. That’s where you’ll find all those cool quick starts. I keep talking about that I wrote. And then there’s always GitHub. You can go to GitHub itself. When you go to Openzd, you’ll see a preponderance of projects in there. So many projects. The one that you really want to focus on is that ZD project. It’s pinned at the top, but that project is basically the root of all of the others. All the others are like libraries that go under it. The Openzl ZD project is really the one that I want you to focus on. And boy, I’d really love it for you to help us get the word out. Star that repo. Because if you start that repo, then other people will find this project. They’ll realize just how great it is to understand that application. Hopefully they’ll understand application embedded zero trust really is the future. Then we have Twitter handle. You can go and subtitle at Opens. And there’s also Open Ziggy. Who you’ll find he’s our mascot because as I say, every open source project must have a mascot. If you don’t have a mascot, then who are you?

[00:51:54.420] – Clint
Right? So we’ve got a little piece of pasta. He’s our buddy called Ziggy. You can follow him as well. And then every Friday I will do a YouTube TV. Sorry. Zd TV is what I call it where we’ll talk about something zero trust related Open ZD stuff. And sometimes it will be just like Golang. Like the other day we were talking about some developer stuff. So yeah, those are probably the best places to find us. You can find me out there too. But openZ is more important. You’ll find me if you name is not hard to find if you can spell it.

[00:52:28.770] – Ethan
If you want to find out more about the openZ project, all of the links that Clint just mentioned will be in the show notes at multicloud. Io or PacketPushers. Net where we host these shows and our thanks to Netflix for sponsoring today’s episode. We appreciate them. This is how Ned and I make our living. So virtual high fives to you for tuning in because if you weren’t listening out there, we wouldn’t be able to do what we do. Talking to the Cloud Ratty people like Clint for your education and information. And if you have suggestions for future shows, people you want us to interview, open source projects you want us to talk about, companies you want us to cover, et cetera. Let us know we are monitoring Twitter at day two cloud show and if you’re not a Twitter person, go to Ned’s fancy website and fill out the form there and let us know what you want us to cover on the next episode. If you like engineering oriented shows like this, just go to packetpushes netsubscribe for even more. All of the podcast, news, newsletters and websites are there. It’s all nerdy content like we covered today, all nerdy stuff designed for your professional career development.

[00:53:27.850] – Ethan
And until then, just remember Cloud is what happens while it is making other plans.

More from this show

Day Two Cloud 147: Google Cloud Is Not Just For Devs

Today on Day Two Cloud we peel back the curtains on Google Cloud with a GCP insider to find out how Google Cloud differentiates itself, its embrace of a multi-cloud approach, and more. Our guest is Richard Seroter, Director of Outbound Product Management...

Episode 142