Follow me:
Listen on:

Day Two Cloud 081: Abstractions Should Save Typing, Not Thinking

Episode 81

Play episode

Today’s Day Two Cloud episode is part one of a two-part show on abstractions. Hosts Ned Bellavance and Ethan Banks riff on the idea that “Abstractions are there to save you typing, not to save you thinking.”

The public cloud is a massive abstraction, and it’s often sold as a service that frees you up from having to think about stuff like power, cooling, running a data center or co-lo, and so on. But if you aren’t thinking about what you’re doing, design goes out the window. And then you run into issues with an Availability Zone and suddenly you’re faced with an outage.

The upshot? Abstractions don’t eliminate issues, they just move them someplace else. And that has repercussions for design, development, infrastructure, and operations.

We discuss:

  • The benefits and drawbacks of abstractions
  • Where abstractions exist in IT
  • The degree to which you need to understand the moving parts below the abstraction
  • Being pragmatic about abstractions
  • More

Sponsor: Onix

As an award-winning cloud solutions provider, Onix provides consulting services for cloud infrastructure, collaboration, devices, enterprise search and geospatial technology. For a limited time, Onix is offering your organization a FREE 6 Hour Cloud Data Strategy Workshop (normally valued at over $2,000). For more information on this special offer, visit

Tech Bytes: VMware vRealize True Visibility Suite

Stay tuned for a sponsored Tech Bytes conversation where we discuss True Visibility Suite, an add-on to the VMware vRealize Operations systems management tooling that helps you understand transactions from the physical up through application layers. Our guest from sponsor VMware is Apolak Borthakur.

Show Links:

The many lies about reducing complexity part 2: Cloud – R&A Enterprise Architecture

@Ned1313 – Ned Bellavance on Twitter

@ecbanks – Ethan Banks on Twitter

Ned In The Cloud – Ned’s Web site

Episode Transcript:

[00:00:00.970] – Ethan
[Ad] Day Two Cloud sponsor Onix is a premier Google cloud partner and an AWS advanced consulting partner their also an 11 time Google Cloud Award winner and was recently recognized as tech times number one for best cloud consulting services in 2020. For a limited time, Onix is offering a free six hour cloud data strategy workshop. That’s kind of a big deal. That’s normally a 2000 dollar service. For more information and to find out more about the six hour Cloud Data Strategy Workshop, visit

[00:00:38.430] Welcome to Day Two Cloud. And today you’ve got Ned and Ethan in a two part show talking about abstractions are there to save us typing not to save us thinking we’re going to riff on that because I heard this in some other podcast I was listening to and it really caught my attention, this whole notion of abstractions. And when we’re done that conversation, we’re going to chat with some folks at VMware about VR ops and then the true visibility suite in a tech bytes that will conclude the show today.

[00:01:09.270] So, Ned. Hey, buddy. How you doing? How’s it going, man?

[00:01:12.810] – Ned
That’s going. I’m doing good. I’m doing real good. And I got to say, our conversation is going to tie neatly into that tech bytes because their whole premise is you do have this complex layers of abstractions and you need to be able to drill down into those to understand what the heck is going on with your system. So that’s it makes less sense of whole thing.

[00:01:33.810] – Ethan
Hit me that the cloud is a massive abstraction on a variety of levels. So there’s if we talk IaaS, it’s a physical abstraction. You don’t see the hardware under there. It’s sort of kind of virtualization. You’re abstracted away from what’s going on underneath.

[00:01:48.360] You’re managing it via APIs or the clicky-clicky UI, you know, as the case may be, but you’re abstracted from a lot of what’s going on and the notion of abstractions that are there to save us typing not to save us thinking.

[00:02:03.390] You think and I think this is an issue a lot of companies have they come to cloud and cloud means because this is what they sold me. I don’t have to think about having a computer room, a data center, even a colo anymore. I have to worry about that stuff. They’re just going to do it all for me. And so design goes out the window and there’s some massive availability zone outage and all of a sudden their application is down.

[00:02:27.690] So what happened? The cloud was supposed to take care of everything and you weren’t supposed to not think about design. You still have to do a lot of robust design work. You’ve moved the problem. You haven’t done away with the problem of things like application availability.

[00:02:43.620] – Ned
Right. I think it ties well into the analogy of thinking about IT in terms of cars and trucks. As someone who drives a car, I don’t understand what goes on when I push the gas pedal like I have it. Kind of an idea, an abstract idea, one might say. But if you pressed me on it, I don’t know the nuts and bolts. I don’t know what’s actually going on. And the thing is, I don’t I don’t need to because what I do with the car doesn’t necessitate that level of understanding.

[00:03:15.510] But let’s say I want to get into drag racing. Oh, OK. Well, now it’s very important that I understand that lower level of abstraction because I’m asking for more out of a car than one would typically ask. And so I have to understand all these things about, I don’t know, pressure and valves and fuel oil mixtures.

[00:03:36.280] – Ethan
It’s funny you are you brought up the the gas pedal, which actually can vary widely how they work. Some of them are directly connected to a cable, a throttle cable, physically connected to the pedal. Some of them are drive by wire where it’s a computer that’s in between you and the gas pedal and how all that stuff works matters.

[00:03:56.100] You can actually, in some cars, change the throttle mapping and how it behaves so that when you hit the gas, how the engine reacts can be more aggressive or more sedate. All depending. It was a fast I don’t even know if you knew all that stuff if you’re a car guy or not. But that’s it’s really interesting that you chose the gas pedal, of all things, for abstractions.

[00:04:17.640] – Ned
Well, and that’s that’s kind of my point is I am not a car guy. I know very little about it. I’ve never even learned to change my own oil, which I mean is kind of like a knock on me. I’ll take that, take that lump. But like, I’m not a car guy, but for me, a car is an abstraction. It’s a way to get from point A to point B, and that’s all I want out of it.

[00:04:36.510] And I know I have to do some things over the course of car ownership to maintain that thing. Kind of like patching your operating system. But I don’t understand what’s going down in the guts and it doesn’t matter. And I think cloud can be the same way if you’re not pushing boundaries. If you don’t need exceptional performance from what you’ve deployed in the cloud, then you don’t need to know all those underlying complexities. But if you’re trying to build, say, I don’t know, a software defined storage array in the cloud, suddenly understanding the physical hardware and the way that they create storage that you’re going to be consuming, that becomes like pretty important.

[00:05:18.150] – Ethan
There’s scale, there’s performance, there’s availability and resiliency, and there are data governance concerns, all of that stuff you are still responsible for and that you said storage and boom, all that stuff immediately popped into my head is things you have to know when you’re building out your cloud plan.

[00:05:39.100] – Ned
Absolutely. The other thing that we probably need to get into a little bit is the idea of the shared responsibility model, which basically outlines what the cloud provider is responsible for and then what you’re responsible for. And the answer is you’re responsible for a lot more than the graphic that you see tells you. And I think, Ethan, you pointed me at a blog post that got into this is something I was aware of, but this really like shined a spotlight on how wrong that graphic is.

[00:06:09.980] – Ethan
OK, if you’re listening to this, you’ve got to see this article. We’re going to link to it in the show notes. But it’s from R&A Enterprise Architecture. And this article the author goes through in great detail with graphics showing you all the different parts of an application delivery stack. I’m actually going to pull up the article right now and and just give you a quick sense of what’s happening here. So, for example, you would have let’s start at the top of the stack and go down.

[00:06:36.940] You got applications at the top and then data underneath that, a runtime environment middleware, an operating system, virtualization, sitting under that and then add some physical server and there’s some storage and there’s some networking all the way at the bottom.

[00:06:50.590] If you were to do that on premises, you’d be responsible for everything in that stack.

[00:06:55.210] If you move to IaaS maybe or in theory only responsible for the upper part of this, that kind of the OS up, if it’s PaaS in theory, you’d only be responsible for the data and the applications and then SaaS you’re again, in theory, giving it all over to someone else. And each block in the stack is something that you are in theory, abstracting. But Ned, going back to your point, what’s really going on is this shared responsibility model for almost all of this stuff, networking?

[00:07:24.400] You’ve got to do a lot of networking. You’ve got to think about a lot of networking to get your IaaS environments stood up. You still got to think about storage. As your already pointed out, the server well, no, they have the server, but you still have to think about availability and design and so on and on and on it goes where it’s not you turning the keys over to the cloud provider and paying them a A set fees that you don’t have to worry about this stuff.

[00:07:50.020] Still going to worry about all this stuff.

[00:07:51.820] – Ned
Right. And as you go up the layers of abstraction, there’s something there’s less things you need to necessarily worry about, but new things crop up. Maybe you’re trying to reduce your administrative burden, but you still need to understand. I guess that’s kind of getting back to the core premise of this entire episode, which is abstractions are there to save you typing. It’s not there to save you thinking you still need to think about this stuff. For instance, let’s say that I am interested in running my application on an AMD based, a hypervisor and thus a virtual host to potentially save some money.

[00:08:26.290] – Ethan
You’re a hipster, AMD, so down with it, man. Wow.

[00:08:30.970] I’m down with AMD, you know me. So if I do want to do that and as someone who’s worked in VMware and had to have both Intel and AMD hardware, I know that there are some subtle differences in the instruction sets and it generally shouldn’t matter except when it does. So it’s like that’s an abstraction. I there should be a hardware abstraction layer between the hardware I’m consuming and the way that I consume it at the OS and application layer.

[00:09:01.960] But sometimes that abstraction layer breaks down and that’s what we call a leaky abstraction. It’s when the abstractions not clean. And so now you do need to understand the underlying layer you’re consuming because the impact of that is going to leak through. And I think there’s some really leaky abstractions when you get into the world of networking as well. Right

[00:09:21.710] – Ethan
All over the place. Of course, we use although I’m no longer in favor of it and using it habitually as I once did, the OSI model is a seven layer model to conceptualize networking that most folks are familiar with. Now, I would advocate the Reno model. That’s a topic for a different day. I’m going to build a presentation on it. But let’s do a really simple example. If we think of layer three in networking, we’re talking about routing the IP layer where there’s an address, it’s a dotted quad or it’s 128 bit IPV6 address.

[00:09:53.170] And then underneath that is some kind of actual transport, most commonly Ethernet.

[00:09:59.020] If the Ethernet broken does IP know? Not directly. It has no idea.

[00:10:04.870] The only connection really between IP and Ethernet might be, oh, an ether type bit and a header and ARP doing discovery mapping between an Ethernet Mac address and an IP address. But if the ethernets broken or there’s a topology loop causing a broadcast storm, meaning the Ethernet is down that the IP is trying to ride on top of, is there any abstracted, any knowledge between those two layers that there isn’t? It’s kind of broken. There’s better examples we could come up with.

[00:10:34.660] That’s the one that popped into my head as you go up and down the the networking stack Ned, Yeah, you’re exactly right, there’s leaky abstractions that pop up all over the place most of the time you don’t need to know except when you need to know.

[00:10:47.220] – Ned
And that’s the sort of thing that you just build up over time. You just learn, OK, where is it actually important for me to know how this that underlying piece functions? The other part where abstractions come in very heavily and this is something that I’m encountering more and more as I do more cloud work is with APIs.

[00:11:05.010] The idea behind an API is it’s a contract. I send you this request with this information and then you give me a response that’s, you know, you have codified somewhere in your API and and we understand and that’s a conversation that we can have, except when it doesn’t quite work, it doesn’t quite respect the request type that you’ve given. It gives you back data that doesn’t match the structure that you’re expecting. And then there’s versioning of APIs, and that’s when the abstraction starts to fall apart a little bit.

[00:11:36.060] The idea was, if I have a very clean API and it’s I have two services that are loosely coupled, if I want to swap out what services that API on the back end, I can do that as long as it honors the contents of the API. That’s the idea, right? Loosely coupled architectures, but sometimes it’s a little more tightly coupled inadvertently. And now when I try to swap out the back end with a new version of the service, suddenly my communication breaks down and I start seeing a bunch of friction between my services.

[00:12:09.480] And I have observed this first hand as software companies do try to swap out that back end with something newer and unexpected. Things break because that abstraction wasn’t as clean as you thought it was.

[00:12:22.050] – Ethan
OK, I want to dive into that for a minute. Let’s get an example now. I’m familiar with APIs. I’ve done plenty of API calls with like Python and so on to gather information and get structured data back.

[00:12:32.400] But I don’t live in the world like I think you do. Maybe since you’re heavy into into TerraForm and some other things that are going to be leveraging APIs and and such. One thing I want to know, if you’ve actually seen this where based on the API documentation, you’re expecting a particular structured chunk to come back to you. It’s it’s a JSON blob, let’s say. And these are the fields and this is how it’s structured hierarchically. This is what you’re going to get back.

[00:12:58.050] But it doesn’t give you that back. You’re saying that happens sometimes?

[00:13:00.930] – Ned
Yeah, that happens. And it’s usually something odd. It’s it’s a corner case where you’ve made a request for a certain amount of data and maybe you’ve asked for certain fields and it doesn’t return all of those fields because it either doesn’t have them or they’ve been deprecated. And so sometimes it’s because you don’t specify the API version that you are are requesting against. And so it’s giving you the newest version of that API that has deprecated some of the stuff that you’re asking for.

[00:13:29.640] – Ethan
You’re expecting the older version. That’s what you’ve been querying right along with your code is anticipating. They’ve updated to the new version. And so since either there’s no mechanism for it or you didn’t say, hey, this is the version I want to talk to, it says, oh, you must want the newer version. Here you go. Here’s the answer you’re not expecting.

[00:13:45.910] – Ned
Right. And sometimes there’s big leaps between the version of an API in the way that it functions. That’s certainly happened with the cloud providers more than once where they said, OK, we’ve got version one of our API and it works in this way. But now we’re upgrading to version two and we’re going to keep, you know, version one online and functional. But at some point we’re going to force you to cut over. And if you’re building, say, a new terraform configuration and you forget to specify the version of the provider and you’re working off of how the 1.0 worked, suddenly your whole configuration breaks because it’s pulling the 2.0 version of the provider that uses the version 2.0 of the API.

[00:14:26.880] And that that’s a lot of heartache for those of us who are trying to create those configurations.

[00:14:31.680] – Ethan
But there are always minor changes with just subtle tweaks is all you need, I’m sure. Right?

[00:14:35.580] – Ned
I’m sure. I will say that generally speaking, the cloud providers do provide a really good changelog of what’s been changed and they do signal what’s going to be deprecated pretty far in advance. Yeah. But sometimes they’re wrong, and that’s why you go over to their GitHub repo and you see thirteen hundred issues.

[00:14:57.180] – Ethan
Yeah, well, pretty far in advance is kind of a loaded term because pretty far in advance to them is probably a few months, you know, maybe six months, maybe even a year, although that sounds like an eternity in IT time. But yeah.

[00:15:10.410] No, but if we are using some tool that we wrote and we only use it once in a while to really think about it, otherwise, that’s not a lot of notice for us because it’d be like, oh crap, I wasn’t paying attention.

[00:15:22.860] And then they signaled they were making a change and now my script doesn’t work that I haven’t used for the last few months. I guess I’m going to have to update that, you know, painful for us, right?

[00:15:31.440] – Ned
Absolutely. And that’s actually in a completely separate context, has bit me because I wrote a Azure function that looks at a podcast feed and it will repost podcast episodes in my Twitter feed. And so it uses the buffer API and it uses some other libraries for the XML. Well, buffer change their API. I don’t check that very often. Yeah, it’s not something that I like. It’s not a professional thing. I’m not maintaining it. I just I’m using this abstraction that they provided to me and suddenly that API got versioned and the way I was invoking it stopped working.

[00:16:07.050] And it wasn’t until I happened to notice that the automatic posts stopped that it made me go back and dig through all of this to realize that my linkage was broken.

[00:16:16.260] – Ethan
So similarly, I had a this wasn’t an API, but it was a Web page that I was parsing so that I could log in. Embedded in the Web page was a dynamically generated key. That was a field that if you’re a browser, you can see it. It’s right there and know that you need to submit that dynamically generated key with the form along with your credentials.

[00:16:33.420] While I’ve been working on the script and chipping away at it, trying to get it to work, there is a lot of misery I won’t take you through to try to get it, you know, and I kind of got it going, you know, and I was able to finally get the token, the cookie that I needed as authentication and able to I was going to go from there.

[00:16:49.920] Then I had to shelf that project, got back to it some a couple of months later to start working on it again.

[00:16:56.460] Yeah, that none of that works anymore because they changed the whole login process on the form and it’s all different. And all that effort I had gone through to finally get that stupid authentication cookie stored in my script so that I could move ahead and retrieve some data I was looking for. It doesn’t work anymore, so.

[00:17:13.250] [Ad] Ned and I are stopping the show for a moment to talk about today’s sponsor, Onix. Onix is a cloud solutions provider and they’ve got the credentials and awards that tell you they know what they’re doing.

[00:17:23.300] From your Google Cloud partner, AWS, Advanced Consulting partner, 11 time Google Cloud Award winner, tech times number one for best cloud consulting services in 2020. You’re going I don’t care Ethan, what does Onix do?

[00:17:35.360] All right. Fair enough. Onix provides consulting services for cloud infrastructure, collaboration devices, enterprise search and geospatial technology. Let’s drill into this. Think about one of the recurring themes of Day Two Cloud. We have said for many episodes there are companies that do lift and shift of workloads to the cloud and they are sorry they did that. They pay dearly for that approach. They don’t automate well. The companies don’t leverage cloud data services. They don’t consider application delivery design like they should.

[00:18:03.410] And why do they have all these problems? Cloud brings a lot of new technologies into an I.T. shop. It is hard to get it all right the first time, even with training, even though you’re smart, even though you’re a very capable engineer, even if someone gives you the luxury of time that most of you, frankly, are not given by your senior management, well, OK. This is where the real value of Onix comes in. Onix is going to help you figure out what you really got.

[00:18:26.510] That’s the discovery process. And it matters because you kind of think, you know, everything you’ve got in your infrastructure that’s got to move to cloud, but you probably don’t. Onix is then once the discovery is done, going to help you construct a plan up. This is important. How do we move all the stuff we’ve got into the cloud?

[00:18:41.360] That plan is going to include a lot of the nitty gritty detail about getting that data moved to the right place and stored in the right way and then visualizing what’s actually happening in the cloud and more even stuff that maybe you haven’t considered before. Like machine learning. They have experienced cloud based machine learning nerds on staff at Onix. All right. Another Day Two Cloud topic that comes up a lot, automation. Onix can help you there, too. They have specialists with deep expertise and cloud tool sets and automating that tedious stuff that you’ve been screwing up, trying to do it all by hand.

[00:19:13.550] Been there. I’m right there with you. Well, Onix can build you a data pipeline, integrate it with the existing tools and databases that you already have, and then bring new functionality to your current data solution.

[00:19:24.260] You’re like, I don’t know, that sounds complicated. Maybe it is. But once they help you build all that fancy stuff, they’re going to train you on the solution. So it’s it’s good, right? You’re all good. Pretty cool stuff from Onix. And for a limited time, Onix is offering your organization a free six hour cloud, a data strategy workshop that’s a service they would normally sell for like two thousand dollars or more. But it’s free for people who respond to this offer.

[00:19:48.950] And if you want more information, visit One more time, that’s Let me spell Onix. Don’t get confused here O N I X slash Packet Pusher for more information about their six hour cloud data strategy workshop. And now back to today’s episode.

[00:20:12.230] – Ned
So it probably makes sense to take a step back here because we’re talking about abstractions and we’re relying on them to get work done. But as we’ve sort of communicated, abstractions break and things change. So when does it make sense to use an abstraction versus subverting that abstraction and going directly to the base layer? For instance, I remember there was this disk recovery software and the gentleman who wrote it talked about how he wrote the whole thing in assembly language. He was really proud that he wrote it in an assembly language that made small, powerful.

[00:20:49.760] And I remember thinking myself, you probably could have written it in C++ and it would have been fine, like but he was really he was all about that assembly language life. And I was like a more more power to you, if you can think that way.

[00:21:02.360] – Ethan
So tedious. I just I can’t I mean, I did a semester of that back in my CS program back in the day. I can’t imagine living there without a very specific reason, exactly what you said to write in C. You’re pretty close to things anyway.

[00:21:14.930] Right. But then there’s these other abstraction layers that we put on top, especially in programming, which I don’t do a ton of. But I’m aware that we have, say, JavaScript and then you have all these frameworks that live on top of JavaScript. Or if you’re using Python, you might be using something like Flask as a layer on top of that. Yeah. When does it make sense to use that additional layer, that sort of platform versus writing directly in JavaScript?

[00:21:40.530] What are your thoughts?

[00:21:42.380] – Ethan
It’s a dependency that you’re creating because of the problems that we’ve we’ve already highlighted. So to me, there is a you need to investigate what the abstraction layer really is and how well it’s supported. So you mentioned Flask.

[00:21:54.320] And as I listen to a lot of Python podcasts and stuff, Django comes up all the time and there’s a few other ones that are pretty routinely used, well-supported, lots of community action and movement, keeping that tooling alive.

[00:22:08.040] Sorry, if you’re listening to this podcast, there are builders right below me and I can’t get out of that just. There’s an impact screwdriver happening literally below my office where I record. Sorry about that.

[00:22:18.330] So when you have an abstraction that’s like that, very well supported, very well documented, popular, that feels pretty safe to use because breaking changes that may impact that abstraction are something that are going to be accounted for.

[00:22:32.490] And you’re probably not going to have to do a wholesale rewrite of whatever it is that is writing to that abstraction or reading from that abstraction. Another case here comes up a networking a lot. There are some providers out there where their vendors, I should say, where they’re part of their product value proposition is to abstract a lot of APIs away for you. You write to their API. They are an abstraction layer where they have committed to you as their customer that they’re going to keep up with how to abstract all the things underneath, all the different sort of network devices.

[00:23:06.750] So if you’re talking to a Cisco device and a Juniper device, you need to accomplish the same thing on both of those devices. The way you would do it at the CLI or the way you would do it, even with API calls are very different.

[00:23:19.020] Wouldn’t it be nice if you had this tool that would sit on top of the Cisco and the Juniper box boxes? You talk to the thing on top that abstracts the Cisco and Juniper boxes away. All you’ve got to do is write to the thing. The thing takes care of the rest for you and its vendor supported.

[00:23:34.290] OK, that feels pretty safe, I guess, although there’s the risk of the vendors saying, hey, we didn’t get enough sales for this thing, we’re going to kill it. Sorry, no more support for you. And then over time, the thing becomes increasingly less useful as the APIs as used to. Well, extract or abstract very well for you becomes useless as those APIs change.

[00:23:57.780] So I guess what I’m saying here Ned, as I’m walking through examples, to me it’s a educated risk calculation where you look ahead, you kind of look at the scenario of what this abstraction layer is and what it does, who’s making it, who’s supporting it, how well it’s kept up with and then decide it is worth the investment of time and operational resources to leverage this API for us, because we’re going to get these business wins, whatever they are or I don’t know, it’s never going to be much more in a science project.

[00:24:28.650] OK, fine. In the lab maybe. But do you want to roll that into production? Probably you don’t. You know, I think you could speak to, you know, my favorite abstraction layer to talk about, even though I’m not proficient in these tools yet. But but TerraForm comes up as a big one in my mind. That seems awfully useful if you considering the criteria that I just mentioned.

[00:24:46.890] – Ned
Right. There’s a there’s a cost to learning the abstraction and the benefit from learning that abstraction has to be greater than the cost of learning it. So one of the common examples, and I’ve had this conversation on this podcast actually with the with other folks is if you are only ever going to deploy in Microsoft Azure, you’re never going to use another cloud service. That’s that’s it. It’s your one and only then doing everything in ARM templates might make the most sense.

[00:25:16.050] Yep. You know, it’s it’s going to be as close to the API as possible because Microsoft is basically writing it and you already will learn it and it will be useful to you. And there’s no necessarily any benefit to using the terraform, which is a little bit removed. Although Microsoft provides really good support for the provider, it is maybe a step removed from some of the arm template stuff. So, OK, that makes sense. But if you think you’re going to end up working with more than one cloud, it may make sense to learn a tool like terraform that provides a little bit of abstraction.

[00:25:54.720] And I would say it’s a very thin layer of abstraction because unlike what you were talking about with the networking devices, where it takes all these different heterogeneous networking vendors and you push a config and it figures out what that looks like on each one, you have to go with the lowest common feature set for each of those devices.

[00:26:13.200] – Ethan
Yes, you are bounding yourself. You’re too constraining yourself in that way.

[00:26:16.560] – Ned
Yeah, absolutely. So you lose the specialization of those devices, in which case it probably makes sense to just start buying white box. Why are you spending the extra money? That’s the part where you can’t use. Right. So that is one way of abstracting things. TerraForm doesn’t necessarily do that. And there’s other stuff that also it doesn’t remove the specialization of the things that you’re working with. It provides a common language and interface for talking to all those things in the same way that you use the bash shell to accomplish a lot of stuff on Linux.

[00:26:51.240] And it does abstracts some things for you. It still gives you full access to everything that’s going on underneath. You’re not losing the specialization by using bash over just the vanilla shell. So I think that’s you have to make that cost benefit calculation for each abstraction that you want to consume. Am I getting more value by using it and. How much does it cost me to learn it and is there a penalty to learning it over, learning what’s going on underneath.

[00:27:20.150] – Ethan
Something like a hypervisor is kind of a classic thing. I mean, how many boxes do you build these days on bare metal as opposed to you layer on a hypervisor first?

[00:27:29.540] – Ned
I mean, I have four physical boxes, five physical boxes in my lab, and every single one is running a hypervisor. Wow. Why would I install a full OS? Because I would get it right.

[00:27:39.650] – Ethan
Why? Why would you do that? Unless it’s a different kind of hypervisor you’re running. But yeah, the points stands, why would you do that.

[00:27:48.350] On the other hand, I don’t know that cloud, you know, going to public cloud is always an obvious slam dunk for this stuff. And maybe that’s a bigger a bigger point to make in going back to that R&A enterprise architecture article we were referring to earlier.

[00:28:03.290] That’s a big part of the point of the article. You think you’re abstracting away all this stuff and the responsibility for taking care of all these blocks and the application stack become the public cloud providers problem, and you exchange that trade of responsibilities for money. You pay them and it’s their problem. Only it isn’t. It’s just not as simple as that. When we abstract things, it is not a trading of responsibilities. It is a shift in how you are executing the task you are ultimately trying to do.

[00:28:36.320] That still leaves all of the responsibility in your plate to get the job done, execute a proper design, make sure there’s the resiliency that is all there.

[00:28:46.460] It is not an abdication of responsibility, even though, particularly managers and executives kind of want it to be that way.

[00:28:53.510] I don’t want have to think about all this stuff. Amazon Uncle Jeff, he’ll take care of it all for me.

[00:28:58.820] – Ned
Like, if you don’t want to think about all this IT stuff, then get a different job. Yeah, yeah. If you’re the director of it, if you’re the CTO, it’s kind of your job. I mean, that’s just the way it is. If you don’t want to deal with technology, then don’t. But don’t don’t pretend that somebody else is going to take care of it for you unless you’re paying them to do that exact task.

[00:29:21.050] – Ethan
Another point made in that article, Ned, was if you don’t think about it in the right way, you will pay a penalty, a financial one. And this goes back to our lift and shift conversations that we’ve had on this podcast where we know companies that have done the lift and shift approach. We need to go to cloud. This is a thing we are doing. It is coming from on high. We shall go to cloud.

[00:29:44.750] And so the fastest way to get there is OK lift and shift to pick up the VM and effectively drop it in IaaS somewhere. That is not a cost effective way. And so you think your abstracting things, it’s going to be cheaper once you shut the computer room down, the data center down? No, it probably isn’t. You’re actually going to pay more to pick up your data center and move it ad hoc as opposed to leveraging the abstraction layers and the cloud native resources in the way they were meant to be consumed.

[00:30:15.840] – Ned
Absolutely. Absolutely. I think it all comes back to the fact that if you’re working in IT, you can’t avoid abstractions. You’re not going to go to a printed circuit board and manually flip registers in a CPU. You’re just not you have to add an abstraction layer. You need drivers and hardware, abstraction, layer and firmware. Well, that’s one level. And then above that, well, you need something to drive that and that’s going to be your operating system.

[00:30:45.200] But OK, you’ve got your operating system. That’s that’s one layer. But now you want your operating system to, like, run things like an application. So now you have your application. But that needs a runtime of some kind. So you need the runtime that sits above your operating system and then that runtime is going to run the application. But maybe that application is just creating a platform for an additional application. Let’s say you’re running Kubernetes. Why not?

[00:31:11.000] Let’s let’s do that. So Kubernetes now needs all of the constructs that Linux provides, as well as the container runtime environment, all that to schedule pods to run your application that’s running in containers, which may even form yet another layer of abstraction to the thing you want to actually accomplish. So here we are like it’s a seven layer dip of deliciousness and each layer of abstraction is a trade off. You’re trading the complexity or you’re hiding the complexity. But that’s it.

[00:31:46.580] – Ethan
Yeah, you’re hiding. Hiding. You’re not. It doesn’t go away? You’re merely hiding it and therefore you are exactly that.

[00:31:55.970] Now, Ned, I think as we’re winding this down here. There’s a point we got to make. We’re not arguing that abstractions are bad or the abstractions aren’t necessary. You should avoid abstractions because then you really are living in assembly language. That’s that’s not practical for any of us. You’d have to have abstractions of. But the it’s the point to me, it goes right back to the original quote, that we opened this thing with the idea that abstraction is there to save us typing, not to save us thinking, leverage the abstractions in a proper way, understanding what they’re doing for us and treating them with the respect, the wariness, the carefulness that they deserve.

[00:32:33.120] Can’t get you can’t do IT without abstractions.

[00:32:36.510] They’re everywhere. They’re absolutely critical.

[00:32:38.500] We’re not arguing against them.

[00:32:39.810] Just think about them in the right way.

[00:32:42.720] – Ned
Right. And understand that when you push the limits of an abstraction, it will break and understanding when it breaks. And what you have to do at that point is something that you just learn over time. And it’s it’s the big it’s the megascalers. It’s people who are pushing the limits of performance, specialization, availability. They’re the ones who discovered these limits of the abstraction. Generally speaking, you’re only going to bump up against one or two of these in your career.

[00:33:09.210] But people who are doing it at scale bump into them all the time. And I’m glad we have them around to to step on the Legos underfoot so we don’t have to.

[00:33:19.050] – Ned
Do presentations at Kubecon and and re:Invent for us.

[00:33:22.660] Yeah, yeah. All right. Well, that winds up this part of the discussion.

[00:33:26.970] Coming up next, Ned and I are going to chat with some folks at VMware. If you’re familiar with vRealize operations, they have got a new tool that you can add to your VR ops set up called True Visibility Suite. This whole thing is about visualizing right down to the physical wire and a switch right on all the way up through the application layer, exactly how an application is delivered so that when you are troubleshooting grey failure, not red light, green light, but like a gray area, something’s a little slow, something’s wonky.

[00:33:56.970] Transactions are getting dropped. On occasion.

[00:33:58.770] You can look at exactly how that application is being delivered across your infrastructure, across all layers, figure out where the problem is and get it solved quickly. So stay tuned for that in our Tech Bytes portion of today’s Day Two Cloud.

[00:34:18.560] – Ethan
Welcome to the Tech Bytes portion of today’s episode, where we are discussing true visibility suite, this is an add on to the VMware v. Realize operations systems management tool that helps you understand transactions from the physical all the way up through application layers. And I’ve seen a demo of this product. It is very interesting. Joining us today to talk about true visibility suite is Apolak Borthakur. Welcome to the Tech Bytes portion of Day Two Cloud. And man, let’s jump right in here.

[00:34:44.120] It just a sentence or two. We’re going to talk about your visibility suite, but we’ve got to start with vRealize operations because maybe not everybody that is familiar with VMware knows what we vRealize. Operations is all about

[00:34:54.530] – Apolak
Sure. So I will go back to the comment that I got when I first joined the group, which was about two years ago, and asked this person what what is vRops? And the answer he gave was, you know, this is the best thing since sliced bread for operations. And, you know, obviously that was a little bit of unbridled enthusiasm. But, you know, if you look at it, it is one of the leading products in the operations space.

[00:35:16.050] We got 10000 plus active customers. And the interesting part is a lot of really large customers have essentially outsourced their entire management operations to the vRops. And I still remember soon after I joined one of the customers that I was talking to, which is one of the largest U.S. airlines, basically said, you know, if your software goes down, planes don’t fly.

[00:35:39.410] But that’s the kind of mission critical operations that we support. And in terms of high level functionality, we have everything from monitoring, troubleshooting, root cost analysis, capacity optimization, workflow placement, all the things that you need to do to run your operations.

[00:35:53.810] – Ned
Right. So vRops is it’s more than just the thing that we’re talking about today. So this is in the context of a of a much larger suite. I’m curious about the true visibility portion of things. Is that something that VMware developed in-house or is that part of an acquisition that came in through the door?

[00:36:12.260] – Apolak
This was an acquisition. Now, of course, there’s ways that you can develop your own what we call management packs, which are integrations that bring data into the vRops. But TVS was an acquisition.

[00:36:25.010] And just to give you a little bit of the context behind why we acquired you know, soon after I joined, I talked to a lot of customers and I was explaining to them all the fancy root cause analysis and analytics that we do.

[00:36:38.960] But then when I at the end, when I asked him, what is some of the main gaps that you have in many, many of the cases, it’s about, you know, we need you to discover and help us monitor the Cisco switches or Juniper firewalls or what have you. So this is table stakes that was much more important for them. And that’s when it kind of hit me that, you know, this is what we need to focus on more.

[00:37:01.970] So when this acquisition proposal came my way, I’m like, yes, absolutely. This will help us light up, you know, some of the dark areas that they want to monitor and manage. So we went for it.

[00:37:13.010] – Ethan
Now, TVS, True Visibility Suite is an add on to vRops. So let’s say I got a pitch this to my less technical boss. You kind of give us a taste there a second ago of what TVS is all about. But but let’s say I got to do this pitch. How would I explain it simply to him?

[00:37:30.620] – Apolak
I would say that the best way of pitching it to your boss would be, hey, boss, this will help you expand your empire. You know, at the end of the day, it is always looking to expand what they do have more control over larger portions of the ideas estate. And this allows allows them to do that because fundamentally, vRops is one of the most sophisticated platforms for operations. And it’s over a thousand plus years of development that has gone in with a very large team, many hundreds of developers.

[00:38:01.010] But at the end of the day, if you’re not getting the data in, then you’re not able to leverage it to its full potential. And what what has happened in many of the customers that I talked to is, again, there are portions of the estate that they wanted to manage. But because there wasn’t a way to get all the information in, they weren’t able to manage it. So this is what enables you to really get in all of that information, you know, light up the darkness as one of their customers actually told me.

[00:38:25.640] And in effect, going back to your specific question, it really helps your boss have a much bigger say in terms of what they manage in their environment.

[00:38:35.030] – Ned
As someone who used VMware products. And still I still do use VMware products at one of the things that always seemed a little bit outside, was difficult to get integrated was my hardware. I would have some HP servers or I would have some. Like you said, Cisco switches are something. I want information from those as well as what’s happening in my virtual machines, because that’s all part of how my application is going to run. So I’m assuming that’s one of the features of TVS. Is this integration. Do you have an across the board integration with different hardware vendors or is it just specific ones that you’re working with?

[00:39:09.630] – Apolak
Yeah, so very good question. It is across the board integration with many hardware as well as software vendors. So one of the one of the value propositions is that we we are able to get all of the objects as well as the metrics all the way from app, all the way down to all the infra and hardware. So what this enables us to do is to have a full end to end picture all the way from app to infra. And so this enables a lot of interesting use cases.

[00:39:36.010] So, for example, right, if your application is slowing down, we know exactly what is the supporting infrastructure that essentially underpins that app. Now we’re able to do some intelligent analytics and say, hey, you know, the reason an application is slowing down is either because this particular hardware or software component that essentially supports that app is having some issues. And here’s the specific issue. So it really helps the the meantime to repair. And also, in some cases, it will be what we say is the meantime to innocence, which is if the issue is not in the infrastructure that I manage, we can also tell you that because OK, here, look at all of the infrastructure and all the underlying components underneath that app is all fine.

[00:40:17.260] So it’s something else. So just leave me alone, you know, so it helps to repair as well as meantime to innocence.

[00:40:23.340] – Ethan
What fascinated me is, as I saw the demo of this product, your visibility suite was the number of integrations there are. There are a ton of them. And it looks like you got a license, some of them. But but just to get this engineering audience, some ideas of what is out there. There’s support for Cisco, UCS, Dell, EMC, HPE Proliant. I mean, I guess from VMware, we kind of would expect all the Dell EMC stuff to be there.

[00:40:47.230] But then there’s things like if you’re a networking practitioner and you want to understand how the networking hardware infrastructure impacts your application delivery, there’s an Arista pack there, Cisco support, F5, Citrix, Palo Alto, Docker and and more. And then right on up through certain enterprise applications like Hadoop, MySQL, HANA. So in other words, you really can drill in very specifically to the components that you’re using to deliver an application. And it’s less about that operational tier versus the application developer people tier and so on.

[00:41:22.330] It really gives you this unified view of the application as it’s being delivered.

[00:41:27.580] – Apolak
Absolutely. And that is one of the key value-add. It’s it’s everything that is the full stack.

[00:41:33.370] And as you said, it’s a completely heterogeneous works across whether it’s Cisco Arista, you know, you call out, but there’s many more. Right. SAP, Hadoop, Tomcat, Hyper-V, Tanzu, the list is really long. And, you know, I was just talking to an analyst last week and what they’re seeing is specifically this heterogeneity problem. Right. A lot of large enterprises, they are going to be using a lot of different types of hardware, software, both on premise as well as public clouds.

[00:42:03.100] And they want to get the single point of view across the entire estate, because no matter what, unless you’re a very small startup, you can’t get past the heterogeneity. And this product allows us to manage the heterogeneous infrastructure from a single central console. And that’s a that’s a great value add for us.

[00:42:21.310] – Ned
I know one of the ways that I tend to interact with products these days is via the API or a command line, as opposed to consuming it through a console. So you mentioned the console. But I’m curious, is there a rich API that I can work with as well to integrate it with existing tool sets or something like a CI/CD pipeline?

[00:42:40.510] – Apolak
Absolutely, and as you guys know, right, a lot of the large customers nowadays, this is this is one of their table stakes demands. So I was talking to a few customers, you know, some of the largest banks, you know, JPM, CEO, Credit Suisse, as well as companies like Optum. And the interesting thing is all of these companies are using our products, but they have created their own portal because they have very specific needs and they want to do everything via the APIs.

[00:43:08.320] So it’s a it’s a it’s a it’s a fundamental tenet for us. Everything that we built and this is this is regardless of TVS or other parts of the VMware product suite has to have rich API. So and yes, TVS does have a rich API.

[00:43:20.650] – Ned
That’s fantastic to hear. I’m always surprised when people don’t do that, to be honest. I mean, in this day and age

[00:43:27.180] – Apolak
One of these tenets that we’re actually going with now is API first. So first build up the API and then build out the UI. Of course we want to have both because some customers prefer UIs, but it’s an API first approach that we’re adopting within the BU.

[00:43:40.850] – Ned
Excellent. Another thing you mentioned very briefly when we were going through all the integrations was cloud. And I’m curious because it might not necessarily have VMware in the cloud. How is this integrating into deployments? I might have in Azure or AWS.

[00:43:56.530] – Apolak
So cloud is very much a fundamental part of our strategy and that takes various different flavors. So, for example, you might have heard of VMC, which is VMware running on AWS and that is VMware SDDC all running within AWS and we’re very, very tight and partnership with AWS or essentially AWS provides the hardware and we.

[00:44:17.800] Essentially have our software running on top of it, and the great value that this provides is that it really enables customers to very easily switch to the cloud, because the other thing is normally when you switch to the cloud, you have to re architect your applications. And that’s a massive project. And customers don’t want to do it right. And with our cloud, you can literally VMotion it over. And because it’s the same underlying infrastructure, it’ll all work.

[00:44:41.140] Right. And and so in that context, all of the VMware SDDC infrastructure that is running on premise is also generally running on AWS. And all of the TVS management packs that work with that infrastructure and bring data in will be completely relevant. But outside of that, even for a customer that is running workloads on pure public cloud, let’s say on EC2 VMS or, you know, using some of the AWS services, there are management packs that what TVS had as well as we did some in-house that are specifically geared towards bringing all of the data in the cloud is very much a fundamental part of our strategy, whether it’s the VMware SDDC running on AWS or the or the other cloud or workloads running on pure public clouds.

[00:45:26.380] And one of the things that, in fact, I told the TVS team recently was that, you know, in some sense you almost have to get out of the business of creating these integrations if you want to be successful because the public clouds iterate so quickly.

[00:45:39.100] So right, and customers can’t wait for us to deliver the next rev of the management pack. And so one of the projects that we’re working on and Mike Langdon, who was the TVS person, is actually driving this is what can we do to make it so simple that customers themselves can create their own management packs? Because that’s the only way, you know, we will keep up with AWS. And so that’s an exciting project that the TVS team is working on, which would be very, very relevant towards how we win in pure public cloud workloads.

[00:46:09.790] – Ethan
Oh, that sounds scary.

[00:46:12.340] And also wonderful, both at the same time. But because you get that power, but then you’ve got to learn the tool and AWS does iterate fast and so on. But that sounds really interesting. I would be interested to see when you guys finally bring that featured a market for true visibility suite. What else is on the roadmap? Apolak?

[00:46:26.120] – Apolak
Also on the roadmap. We you know, we’re trying to follow an agile process. So it’s hard for me to give you a lot of specifics. But broadly, it’s our own cloud, what we call the hybrid cloud, which is VMware infrastructure running on AWS. And we continue to expand on that. We have a lot of acquisitions and we need to make sure that we bring all the data into the vRops and then pure public cloud workloads, expanding on the on the set of services that we integrate and work with.

[00:46:51.540] If I had to basically summarize broadly, it’s basically the hybrid and public cloud and making sure that we can get more and more information in those environments into the vRops. And that would be the focus of TVS.

[00:47:03.820] – Ethan
Gotcha. OK, so for people that have heard this, they’re in vRops. They’re interested in that or they want to learn more about true visibility suite. Is there a specific landing page or somewhere they should go or just kind of go up to VMware dot com and you can figure it out from there

[00:47:16.930] – Apolak And you can look at there’s a hybrid cloud section. And there’s also there’s a couple of different sections in there where you’ll get a lot of information on our products.

[00:47:28.840] – Ethan
Very good. Thank you very much for joining us today, Apolak.

[00:47:32.110] – Apolak
Thank you.

[00:47:33.010] – Ethan
Well, thanks to VMware for sponsoring the tech byte portion of today’s Day Two Cloud episode and Virtual High five to you, you awesome person for tuning in. If you have suggestions for future shows, we would love to hear them. You can hit us up on Twitter at @DayTwoCloudShow or fill out the form on Ned’s fancy website If you like engineering oriented shows like this one, visit All of our podcasts and there’s a bunch of them.

[00:47:57.970] All of our podcast newsletters and websites are there. It’s all nerdy content designed for your professional career development. Until then, just remember, cloud is what happens while IT is making other plans.

More from this show

Day Two Cloud 164: DevSecOps Is A Real Thing

Today on the Day Two Cloud podcast, we talk DevSecOps and how it's more than just a marketing term. We also discuss Infrastructure as Code (IaC) and IT as Code and what that actually means for operations folks. It doesn't mean you have to write code all...

Episode 81