Follow me:
Listen on:

Day Two Cloud 091: BGP School For Cloud People

BGP, the Border Gateway Protocol, is a routing protocol widely used across the Internet and private intranets. Routing devices speak BGP to another to exchange information about how to reach a destination. A familiarity with BGP and its many functions and manifestations is useful for infrastructure pros who set up, manage, and troubleshoot public cloud services.

On today’s Day Two Cloud, Ethan Banks and Ned Bellavance dive into essentials about BGP including why it’s so widely used, concepts such as the Autonomous System (AS), the AS number, loop prevention, BGP communities, IBGP, EBGP, and more.

Ethan and Ned also discuss BGP in relation to essential cloud services such as VPCs, gateways, and VPNs.

Sponsor: Packet Pushers Livestream With Alkira

The cloud era requires a clean-sheet approach to networking. Join the Packet Pushers’ inaugural Livestream event on April 22 to find out how the Alkira Network Cloud, our Livestream sponsor, lets you deploy and manage single and multi-cloud networks with built-in visibility, security, and governance–all delivered as-a-service. This free virtual event includes technical deep-dives, roundtable discussions hosted by the Packet Pushers, and use cases, deployment scenarios, and architectures. Go to to register.

Show Links:

AWS Direct Connect virtual interfaces – AWS

ExpressRoute circuits and peering – Azure

Routing policies and BGP communities – AWS

Creating a VPN gateway – Azure

About BGP with Azure VPN Gateway – Azure – to comment or follow up on this and other podcasts – Join the Packet Pushers Slack channel



[00:00:00.120] – Ethan
[AD] The packet pushers are live streaming with cloud networking vendor Alkira on April 22nd, 2021. Sign up, attend live, get your questions about clean sheet networking for the Cloud era answered. Visit Packet pusher’s dot net slash live stream to register. That’s packet pusher’s dot net slash lives tream. [/AD] [00:00:24.080] – Ned
Welcome to Day Two Cloud. You know, there’s nothing that quite strikes terror into my heart, like the words b, g, p, my goodness, I have no idea what this thing is.

[00:00:36.320] I’m pretty sure I have screwed it up countless times and to school me on all things BGP and a little bit of additional routing. We have our own co-host, Mr. Ethan Banks, joining me on this show. Ethan, hello. How are you? Are you ready to talk networking nerdiness?

[00:00:53.570] – Ethan
Yeah, buddy, let’s do this. I think it’s funny that you said all things BGP, because here’s the joke about BGP. It is such a huge topic. Many books, hundreds of pages in length have been written and many of those are topical, like narrowly scoped about some particular function within BGP.

[00:01:12.140] It is a massive project, a product, I guess whatever, with lots and lots and lots of aspects to it and rabbit trails you can go down and RFCs that have been written related to it and stuff. So, you know, I’m not going to school you on all the things. We’re going to talk about some of the things you know.

[00:01:29.840] – Ned
No, we’re doing it all. Strap in listeners have been driving this is a sixteen podcast. Yes, OK, maybe not. Maybe we won’t do that. So the reason I bring this up is as someone who worked on virtualization systems a lot, worked on a little bit of networking, but never got outside of the data center really. Or maybe, you know, just the office space. Routing protocols have always been a little mysterious to me and especially BGP, because even when I did encounter routing protocols, it was real simple stuff like RIP and I was trying to get my MCSE.

[00:02:03.670] That’s that’s when I encountered this stuff. Once I moved onto cloud technologies and started having to create the VPN tunnels between on Prem and the Cloud or trying to link different cloud networks together. Suddenly this term started coming up BGP and asking for my AS number and I’m like, I sure I don’t know. And it had a suggested value and like, good, I will go with that, but didn’t know what I was doing that I was kind of bothered me a little bit. So maybe we can start at the highest level here and just kind of define what BGP is and then work our way down to my specific concerns.

[00:02:40.930] – Ethan
So at the top level, BGP is a routing protocol, as you said, and it is a language to routing devices, speak to one another, to exchange destinations they know about. I know how to get to this IP prefix, OK? And I know how to get to this IP prefix.

[00:02:59.800] They’ll exchange that information and learn about how the other router knows how to get somewhere and build a routing table and then hop by hop you can get from where you are to where you’re trying to get to via all these routers that I’ve learned about each other and where they know how to get using Border Gateway Protocol BGP.

[00:03:21.070] – Ned
OK, so that the border would be a network of some kind if we’re trying to figure out and map it to what I already know.

[00:03:29.380] – Ethan
All right. So let’s, let’s dig into BGP then specifically because there’s lots of routing protocols, as you mentioned. So BGP is the Internet routing protocol. It scales globally. There are hundreds of thousands and we’re approaching a million routes before too awful long in the Internet routing table. And the border would be that of what BGP would call an autonomous system. So an AS and you might have heard the term autonomous system AS or an ASN an autonomous system number. So when you are going from one place to another on the Internet, you’re doing doing it by going through a variety of autonomous systems.

[00:04:07.210] So this is a cloud show, well Amazon has their own autonomous system. Now, I forget what it is, but they’ve they’ve got one. And your service provider that you’re connected to at your house, they’ve got an autonomous system number, or two or three. All depending on how they grew over the years. And you hop through autonomous systems. An AS, an autonomous system, it’s a group of routers, Ned it’s group of routers. Think of them as like a cluster, an island of routers that are interconnected and you go from system to system.

[00:04:36.070] BGP doesn’t know every router in the AS. It just looks at the AS as a thing, as an object that it has to traverse to get from point A to point B. And you’re dealing with borders. Yeah, at the edge of an autonomous system. You could think of it as it was a gateway to get you from one place to another, not to bury you in terminology. But there’s a, you know, another point here worth making, which is you’ve got external BGP and internal BGP.

[00:05:04.690] Inside my autonomous system, I’m talking iBGP internal BGP to all the routers inside that AS. If I’m exchanging data with a different autonomous system, then I’m speaking eBGP or external border gateway protocol so did any of that help?

[00:05:23.170] – Ned
Yeah, that helped a lot. I think so the autonomous system portion of it, that’s going to be a bunch of routers that are all managed by the same entity and.

[00:05:31.840] – Ethan
They all share a common number. That number was uniquely assigned to them. If they’re on the public Internet or if they’re a private something you’re doing internal to your company, you can use a private AS number that doesn’t get shared out. Kind of like you can have private IP numbers and public IP numbers and and. Yeah, so those AS numbers are a big deal.

[00:05:51.700] – Ned
OK, so every router in that AS has an AS number that it knows it’s part of this autonomous system. Does that also then imply that they’re all using the same IP address space, they’re all part of the same contiguous block or does not imply that at all?

[00:06:09.610] – Ethan
From a design perspective, it probably means that. Yeah. So when you as say you’re on the on the Internet as a service provider, you’ve got some big block of IP addresses that we were assigned to you. When you advertise your block to your to other autonomous systems, yeah. You probably have a contiguous block no matter how you’ve broken those out into little subnets inside your domain. You just advertise the one big block to everybody else.

[00:06:36.370] – Ned
Right. If you want to get to Ned’s Taco Hut, dot com and I or my taco service, let’s say this is a service provider now. So Ned’s taco service provider, if you want to get to that. Here’s the. This is my AS number, this is the range of public IP addresses that I own, send all that traffic to that public IP address range to my AS number, and I’ll do some routing internally to get it where it needs to go beyond that.

[00:07:03.810] – Ethan
Yeah, you just you just nailed it. So you’re going to advertise the big block. You know, Ned’s Taco Hut service is one IP address inside of that big block. But that’s good enough. I know I need to get to the big block, which is inside that autonomous system, so long as I get it in there. Now that that IP packet is inside that autonomous system, it’s up to the AS to then deliver it to the actual server that’s got your service living on it.

[00:07:26.760] And service providers probably have just to be clear, not just one block, but probably a bunch of a bunch of IPv4 blocks. They’ll have a bunch of IPv6 blocks that they announce, but they’ll do it as tightly, as efficiently as possible with those super blocks like we were talking about.

[00:07:42.420] – Ned
OK, I think I’ve kind of got a general idea of where BGP lives, and it lives above the layer that I typically worked at, which was sort of the layer two a little bit of layer three configuring VLANs. But once the traffic left my network, that’s that’s all I knew it. I had a default gateway. That’s where my traffic goes. If it’s not going to stay on the internal network and once it leaves, I don’t need to understand what happens to it.

[00:08:10.980] – Ethan
You don’t the vast majority of networks like that don’t care about BGP at all because of what you just said. You have a default gateway, you send it to the service provider and the service provider takes care of all the rest of that magic for you. They’re for sure Speaking BGP because you don’t get around the Internet otherwise, but it doesn’t matter to you. That said, you could order an Internet service provider circuit to come into your on prem facility and request BGP service.

[00:08:38.460] So a scenario where I’ve done this, even with smaller companies, is I would get a service from, say, I don’t know, Verizon and one from AT&T and I would have them send me BGP feeds and then I would let my BGP router sitting at my network determine should I send this traffic to Verizon or should I send it to AT&T because I was accepting BGP feeds and then letting my BGP router decide which traffic link were excuse me, which link was faster to send the traffic down. So you can do that. You can accept BGP as a end user.

[00:09:14.970] – Ned
OK, now the situation that I encountered BGP initially is when I was setting up a connection from AWS, the VPN gateway there down to on Prem system, actually in this case it was an Azure VM.

[00:09:30.420] But let’s just pretend it was on Prem and when I was doing the configuration, there was a BGP field and it said you could just use a number that’s in the sixty five thousand range, doesn’t matter. You just plug whatever number you wanted in and left me scratching my head like, OK, assuming I do fill out that number, what actual information is going to be exchanged between the VPC and the VPN gateway that I have in AWS and then my on prem system? How do they exchange information and what’s contained in that information?

[00:10:02.730] – Ethan
So we just jumped over about 16 different things. OK, so one point to make and in our conversation about the Internet and BGP, we kind of implied this is for the Internet.

[00:10:15.480] But as you just said, bring up this specific use case question. Hey, I can use BGP anywhere and you absolutely can. You can use BGP anywhere. Just because it was designed for and is used to get traffic around the Internet doesn’t mean we mere mortals can’t use it to connect up our cloud resources and so on. So and you’re coming up with a scenario there where you can because you were asked you can you can do that. So so let’s dig into that question of yours.

[00:10:41.310] They said to you, hey Ned you just put in something sixty five thousand or bigger any any autonomous system number that you want, because they would have been asking you for an autonomous system, right? Well, that range and I forget exactly where the breakpoint is, but there’s a range they’re very high in that list where it’s a private autonomous system number and you can genuinely pick anything you want. But what’s the point of it? Well, as we described earlier, you have to have an autonomous system because that’s how BGP thinks.

[00:11:07.920] It thinks in the terms of autonomous systems to autonomous systems talking to one another. So you have to have one. And then what’s happening is you would configure your side of the BGP, the BGP peer relationship to. Was this Azure AWS we were talking about?

[00:11:25.380] – Ned
AWS was the one side of it, the VPN gateway. And then for argument’s sake, I had an on Prem network with a firewall that I’m configuring.

[00:11:33.270] – Ethan
Yes. So so the side you were configuring, you’d need to tell it what the AWS AS number was the AWS side would use whatever that private sixty five thousand and higher number that you had chosen. That becomes part of the defining of the peer relationship. O the speaker I’m speaking to is coming from this autonomous system number. And you would expect to see the same autonomous system number coming at you for whatever their autonomous system was. If you don’t have that, it’s kind of like, oh, this isn’t who I was expecting to be talking to.

[00:12:09.060] This isn’t right. I have a mismatch and the end of the peer relationship doesn’t come up. In other words, those are that’s a characteristic of the relationship you have to have. It genuinely doesn’t matter what your private AS number that you chose was. If they said you could pick any one. Yeah, you can pick any one, you would you would pick one. Probably that means something to you if you’re running BGP and a lot of places in your own company, you’d pick something that’s like makes sense for your scheme, right?

[00:12:37.250] They kind of don’t care. They’re not announcing it to anybody else. They’re not using it anywhere else. They just want to know what it is you picked that they know how to set up their side of the configuration to talk to you.

[00:12:46.930] – Ned
OK, so this would be the same as if I’m using well, obviously I’m probably using private IP address space in my internal network and I either have some sort of IPAM system or I have a spreadsheet, probably a spreadsheet, I’m using to track all these private IP address allocations. In the same way, I want to track my AS my private AS number allocations to make sure I don’t use the same number in two different spots because that is bad.

[00:13:10.330] – Ethan
It could be, especially if you were using BGP a lot internally. One of the things that BGP uses autonomous systems for is for loop prevention. So if it looks in its AS path that it sees coming from a neighbor and sees its own AS number coming back at it, it goes, Oh, I must have a loop somewhere. I’m going to reject that path. That’s not good.

[00:13:29.990] And so if you you can inadvertently create that kind of a situation where you might not have a loop, but you use the same AS number twice, even two different AS with the same number. And so BGP doesn’t know doesn’t know your topology. So you do need to be careful and track that stuff is the point.

[00:13:48.760] – Ned
Gotcha. Gotcha. OK, so assuming that I’ve picked my two numbers, one for my on prem network and one for the VPC, I have in AWS and I’ve exchanged I’ve put those numbers in the proper fields in my configuration. Now how does one tell the other what networks or routes are available in that peering relationship?

[00:14:13.560] – Ethan
So we’re going to we’re going to get the peer relationship established, right, and then, yeah, now the next thing is exactly what you asked. How do I how do I exchange networks? How do I exchange, you know, the routes that I want these two routers to know about? So if we were to take a Cisco router, you would use a network statement and say, hey, BGP router, I want you to tell your neighbor about these networks and you would put those networks in. As long as that device can actually get to those, you can’t just advertise anything. But if those networks are actually in the routing table of that device, BGP will announce those routes to its neighbor.

[00:14:52.590] And I would assume there’s some kind of configuration for you to do there with on the cloud side where they’re saying, hey, you got you got a list, what networks are going to be advertised or announce something like that or they’re going to tell you, hey, we’re going to announce these routes to you over this link, all depending on what kind of a tunnel you’re setting up, I suppose.

[00:15:15.000] – Ned
Right. Yeah, I believe that was the case where AWS will just automatically advertise those routes to the VPC or what networks you’re using on your VPC. And then it’s on you to configure your on prem device to advertise the routes, advertise the networks that are on the local side up to AWS. That makes sense. So when I want to send a packet, if we’re playing follow the packet here. If I want to send something from an EC2 instance on a subnet in my VPC and it knows about those additional routes because of BGP when it hits the gateway on the VPC, it’ll get sent down that VPN tunnel to the other side and make its way to whatever system it was destined for.

[00:15:58.200] – Ethan
Well, and that’s going to be part of the equation, too, is actually two components going on. You got the VPN tunnel with its crypto associations and then you’ve got the BGP or I shouldn’t say BGP. I should just say the routing table itself so that when traffic shows up, it knows ah this traffic goes over the VPN tunnel and the VPN tunnel has been configured so that it knows, ah I need to encrypt this traffic going between these IP pairs. So you would have presumably configured the VPN tunnel in that way as well.

[00:16:28.170] Hey, these these traffic, if this traffic shows up, this source going to this destination, encrypt it and then the routing is actually a separate thing. Hey, if this comes in, I know to send it over the VPN tunnel, I think this is likely a couple of steps there.

[00:16:43.170] – Ned
OK, yeah, OK. That I think that clears it up for me a little bit. Now, like I said, in previous lives, I would use something like maybe RIP or even just adding static routes because that is very easy to set up.

[00:16:56.160] Static routes you just put in. You know, here’s where you send that traffic and you’re done. You’re very happy. Why would I use why is BGP more suitable than something like RIP or more complicated protocols like OSPF?

[00:17:14.700] – Ethan
Did you just say OSPF was more complicated than BGP or.

[00:17:17.520] – Ned
I have no idea. That’s why I’m talking to you, Ethan. You know these things. I don’t. I just knew the acronym. Come on.

[00:17:25.170] – Ethan
They’re both complicated in different ways. OK, so the answer to that question is really. A matter of complexity, what does your network look like if you have one point of entry and exit from your network, like, say, you have a small network, everything funnels down to one firewall and all traffic that leaves your network and heads out to the Internet, hits that firewall, that same firewall is where your VPN tunnel going up to your VPC terminates.

[00:17:52.170] Do you need BGP? Uh no? Because everything is funneling down to that firewall anyway. So what would adding a routing protocol do for you? You know that traffic is going to show up at the firewall and you know that because you’ve got the VPN tunnel configured on that firewall, the firewall policy will know, oh I got to send this traffic to the tunnel up to the VPC. So on that side, you don’t need it. On the other side, if you just could add a few static routes that the VPC knows to send traffic down the tunnel to your firewall, if you’ve got that side covered, that would be fine. Simple.

[00:18:26.760] Why add BGP? You’re not solving any problems that way. You’re just adding BGP for some reason doesn’t really seem to be fixing anything. So in that scenario, I think static routes are fine. So, OK, the opposite scenario, Ned, that is you don’t have a simple network, you got six ways off and on your network, you’ve got here’s a scenario. You’ve got that simple network we described before. Plus you’ve got an MPLS WAN let’s say and that provider gives you an off ramp from their MPLS cloud into VPC via some kind of a direct connect service.

[00:19:02.920] That way, those that’s a service that exists. You could do that. OK, and maybe you’ve also got a colo connected somewhere. You’re an Equinix and they are giving you a direct connect up to the VPC. That’s another way to get to a VPC. All right.

[00:19:20.200] – Ned
And now I’ve got at least the VPN tunnel and the direct connect from my ISP and a direct connect from my colo on Equinix, and it sounds like you’re going to add another thing to this.

[00:19:31.480] Oh I’m going to add SDWAN the mix, because, of course, of course, you have a building an SDWAN fabric out Ned and stuck an SDWAN endpoint node to extend your fabric up into the cloud, into your VPC. So that’s yet another way you could get up there. So now you’ve got this complex set of ways that you could get traffic into and out of your VPC. OK, that’s probably more complex than it needs to be. To be fair, you probably wouldn’t you wouldn’t have something quite that complex, but you could.

[00:20:01.240] Now we’re in a situation where we need a dynamic routing protocol and why well, things change on a network like this. Circuits come up and go down and you need to be able to react to that. You don’t want to be reacting to it with with you changing a static route. You know, static routes mean it’s in the name Ned static. They don’t they don’t do good and dynamic situation.

[00:20:24.670] – Ned
Boo we don’t like anything static, we’re dynamic baby.

[00:20:28.300] – Ethan
So so routing protocol like BGP can help you route around changes that are in the network. BGP, also, if you really get into it, you can do a lot of complex tweaks and set a lot of policy. I don’t know, on the cloud side how much flexibility they’re going to give you that way.

[00:20:45.340] But certainly if you’re controlling your own BGP destiny, you’ve got your own BGP routers and stuff you can get nuts with which your favorite exit point from the network is going to be and you know, and so on by tracking your BGP policy. So we don’t need to get into all that especially. But you have all sorts of complicated flexibility that you can build into your routing policy using BGP.

[00:21:11.530] – Ned
Well, I think that brings up a good point about what a real world environment would probably look like. And let’s let’s run with that example. I have some workloads that are running in my colo in Equinix, and then I’ve got, you know, some people who want to get to this VPC from their branch office, which might be using that direct connect connection on my MPLS network. So how would I let devices in the data center know, you should send your trafficup the Equinix Direct Connect, whereas devices and people out of my branch offices, you should use that MPLS connection.

[00:21:48.730] – Ethan
This could come into your autonomous system design because you’re using two different connections there. And so you’d be setting up two different BGP relationships, one from your VPC to the one router, one from your VPC to a different router. That’s in a different part, as you were describing two different areas where you were needed needing to be able to send traffic through. When what BGP will determine is forwarding based on the closest AS path it’s not going to go two AS hops away, if it can only go one.

[00:22:25.350] – Ned

[00:22:26.360] – Ethan
So in that scenario, whoever’s whichever the closest connection is to you, there’s there’s more to this routing stuff, you could get beat, be a little careful and pay attention to what you’re doing. But essentially what you’re doing is whoever you’re going to end up going through the connections that is closest to you on the assumption you’re announcing that the routes coming from your BGP connection into your Interior Gateway routing protocol probably OSPF. In the right way.

[00:22:54.650] – Ned
OK, so in that way, I have maybe an AS number that’s associated with my data center, my colo, and one that’s associated with that MPLS connection.

[00:23:06.020] Yep. And so if something’s going to the branch offices, it’s going to know that’s one hop through the AS associated with that. Whereas if it sends it down through Equinix and then over to my branch offices, that’s two hops, that’s two AWS numbers it has to go through. So it’s going to prefer the first one in this scenario.

[00:23:25.730] – Ethan
There are there are network nerds listening to this going, why would you be connecting both of those BGP sessions Banks? No, I probably wouldn’t, actually. So in honesty, Ned, they might not even know about each other as such, that there’s a lot of it depends here. So let’s back up a step and think about this. You probably in your data center and on your internal network, on Prem, you’re probably not running BGP except at the edge.

[00:23:51.320] So think of BGP in that scenario is like a like a like a bridge, you know, like a bit of glue that is getting the VPC routes into your network. And you’re announcing selected routes from your on prem into the VPC just a few routes, just a few routes to get them there. Well, how does the BGP router let’s talk about the VPC routes there now have made it to the BGP router that’s on Prem. How do you announce them to the rest of your network?

[00:24:16.280] You don’t run BGP anywhere else. You’re probably running OSPF or maybe you’re a Cisco shop running EIGRP. We’re going to redistribute those routes from the BGP process into your Interior Gateway routing protocol. OSPF or EIGRP, now OSPF let’s just say, doesn’t really know anything about BGP metrics or ASPath or anything like that. So this is why I say it gets complicated and it depends, you know, a lot because you lose some of the knowledge that BGP has about that path as soon as you redistribute it.

[00:24:51.290] Now OSPF looking at it going, OK, I talked to that guy who was a BGP redistribution router, doesn’t know anything about BGP, doesn’t know it’s a redistribution router, just knows I go to him. If I want to go to that route that lives up in Amazon at that VPC, that’s all he really knows.

[00:25:09.140] So if you’re doing that redistribution at multiple points coming from multiple BGP processes, you have to make sure that the distance, the metric that OSPF computes is going to converge on the closest one, which isn’t too hard to do. But it just you know, it does add a little more complexity, design and thought to the process to make it happen the way you want, OK.

[00:25:31.500] – Ethan
[AD] Pausing the episode for a quick ad spot about something cool we packet pushers are doing on April 22, 2021, we’re live streaming with Alkira.

[00:25:39.870] Alkira is a cloud networking vendor, and to them cloud networking isn’t just connecting your users to the cloud. It is also about end-to-end governance and policy management. Transitioning to Multi-cloud, supporting data Center Migration’s, security delivered by a cloud firewall and zero trust posture that Alkira feature set. That means they think they’ve got an alternative to SDWAN and MPLS. Woo that’s a big claim. So is Alkira really all that? Well, that’s what we’re going to talk through in the livestream, help make this discussion great by showing up and asking your questions.

[00:26:12.870] Now, you need to register for the events. You can participate. But the way we do it, no one will follow up with you unless you opt in to be contacted. To register for the livestream. With the packet pusher’s and Alkira happening April 22nd, hit packet pusher’s dot net slash live stream. That’s packet pusher’s dot net slash live stream that will redirect you to a Zoom webinar reg page. And from there you know what to do. Thank you for being a part of our community and we hope to see you virtually on April 22nd. [/AD] [00:26:41.240] – Ned
Let’s simplify our example a little bit and go with something that actually is fairly common, which is having a direct connect connection from your colo and then also having a separate VPN connection from your colo in case something happens to your direct connect circuit, you don’t lose all connectivity. Now, if I because I have two paths and they’re essentially going to this from the same VPC or I should say same AWS region to the same data center, what would the configuration look like there to make sure that the preferred path is going to be that direct connect unless it is not available?

[00:27:15.290] – Ethan
You can assign metadata of various kinds to your BGP routes. And I forget which happens on which end because I don’t do BGP every day Ned. But the bottom line here is you would set so that the VPC routes announced over Direct Connect are preferred to BGP over the VPN ones. So you’d have both paths there, BGP would know about both ways to get to the VPC one through direct connect, one through the VPN. You would set your BGP policy to prefer the direct connect. Direct Connect falls over that other path, is still there and is now going to be the only one left will percolate to the top and be used at that point. And that’s that is a pretty common scenario to have parallel links like that with one preferred and the other not. I just forget the exact BGP mechanism that we want to use.

[00:28:05.960] – Ned
OK, but we’re talking about a policy. So this is beyond just basic routing. Now we’re we’re constructing a policy on how we want it to handle some some routing decisions.

[00:28:15.470] – Ethan

[00:28:16.770] – Ned
OK. OK, now another term that I’ve heard thrown around is BGP communities, and I don’t think we’ve touched on that yet. So is a BGP communities. Is that like a bunch of yerts out in a village somewhere or. I feel like there’s something different going on there.

[00:28:33.690] – Ethan
BGP communities, just metadata again. OK, so when when the creators of BGP, I don’t know, I want to go back that far in time. But essentially there are some standard communities, standard bits of metadata that are meaningful to BGP the world over. All BGP processes understand these communities to mean these things. Community does sound like a group of things and it doesn’t really. Dude It’s just a it’s just a number you can attach to the BGP routing update. It’s just something that’s in there. And based on what that number is, you can do a thing, you can make a decision, you can have routing, routing policy behave in a certain way.

[00:29:17.940] Some of these standard ones are like one that pops to mind is the NOEXPORT community. What is no export do? It means, hey, BGP receiver. When you get this route that’s got the no export community bit of metadata attached to it, don’t distribute that route outside of your autonomous system. At least I think that’s what it means. Yeah, some of this stuff is buried in nuance. Dude, I read for three hours, getting ready for the show. Some of the stuff fell back out of my brain.

[00:29:44.340] But the the point of the communities. So if you got no export and a few other standard communities, bits of metadata that tell BGP to behave in a certain way, but again, it’s a field that’s just a number and you can stick any number in there you want. It’s pretty customary with BGP communities now where it’s a two number field separated by a colon. At least that’s how it’s represented to you when you look at the config. And most commonly it’s the origin AS that’s sending you the community, colon and then some number.

[00:30:19.140] Some shops use this to like I want to have this particular community, my AS number colon and then some other number represent the geographic region that this route originated from. And then when I receive that community, I will know, OK, this route originated from this geography. And I am going to therefore handle the route in some way. And you build policy around that community that you’ve received.

[00:30:46.060] – Ned
OK, so it sounds like there’s a lot of metadata being added to just the basic routing information because we’ve got the AS number and then like the advertised routes associated with that AS number. But then it sounds like there’s a whole bunch of metadata you can tack onto that and do custom things, whatever you want.

[00:31:06.310] – Ethan
You want me to blow your mind? The route is metadata as far as as far as BGP is concerned, really because because what BGP really is, it is an exchange of network layer reach ability information. It exchanges NLRIs. And inside of this NLRI this is the thing you’re actually exchanging between BGP speakers an NLRI.

[00:31:30.860] That thing that you’re exchanging has a bunch of bunch of path attributes, all the metadata about the NLRI, of which prefixes are a thing, and AS numbers are a thing, the AS path is a thing and the communities are a thing and and so on and so on.

[00:31:47.540] If you go to a router and do a show like a show BGP, and it starts dumping you, all the things that are in its database every NLRI you see gives you all kinds of crazy information. So much stuff going on in there. But, but to go back to your point. Yeah, exactly. You now have this flexibility to do all kinds of crazy stuff with your routing, using all of these different capabilities that BGP has, part of which is enabled by the crazy amount of metadata that gets sent along inside of all of these NLRIs.

[00:32:25.250] – Ned
So I have to back up a little bit here. What you’re saying is BGP is not a routing protocol. It is an information exchange system. You’ve got all these different independent systems running BGP that are just exchanging information. And yes, the central idea behind it is to exchange routes to figure out networking stuff, but you can use it to exchange any kind of information and process it as you will.

[00:32:52.850] – Ethan
Yeah. And in fact, some people have described BGP as just a kind of a fancy message bus, you know, a way to put information to and from a couple of different databases. And if you look at some of the efforts lately, what the network computer networking community has been doing with BGP, that is kind of how it’s getting used. We’re stuffing the kitchen sink into BGP, which I don’t personally think is a great idea. But, you know, just to get some other examples, some of which are good ideas and some of which may be I don’t know what I think, but you can do Mac addresses like like switches with bridging tables. Mac addresses Ethernet addresses. Right?

[00:33:32.900] Well you can exchange that information using BGP EVPN. And in those NLRIs, you’ve got Mac addresses, you’ve got tunnel end points that you can reach those Mac addresses from since you’re encapsulating an Ethernet frame inside of a VXLAN packet and other stuff. Other bits of metadata there to which you look at that NLRI, it looks a little different from something where you’d see IPv4 IPv6 prefixes.

[00:33:58.670] You’ve got BGP LS, which is linked state where it’s not exchanging route reachability information at all. It’s exchanging from BGP speaker to BGP speaker what the underlying inter the the internal gateway protocol thinks about how to get from point A to point B just so that BGP can give that off to a computation engine that’s going to then figure out how to do traffic engineering.

[00:34:29.450] It’s saying, hey, here’s the underlying link state information here, PCE, Path Computation Engine. Use that information to figure out how you want to forward traffic. Based on what I gave you, did the BGP tell it how to forward? Nope. Were there any routes exchanged? I mean, kind of, but not for forwarding purposes. It’s just a message bus to carry information about the links state routing protocol, BGP riding on top of and give it over to an engine. So BGP’s weird man is so much stuff going on there.

[00:34:59.690] – Ned
Yeah, it really, it really sounds like it and it sounds like the sky’s the limit. Now when I think about data exchange systems and especially because this is kind of like a key value system to a certain degree, I think of other distributed key value systems out there and there are varying efficiencies in what they do. Is BGP especially efficient in the way that exchanges information, considering how old it is.

[00:35:23.890] – Ethan
Efficient. Boy, that’s a loaded question. I don’t I don’t feel confident answering that question, especially I can say. Like, if you look at it at Internet scale, what’s happening with BGP is that it never actually converges anymore. BGP is always converging because it can no longer, at Internet scale, keep up with the constant amount of ads and withdraws of of IP routing information that are being added and removed from the Internet routing tables at any given time. Is that an efficiency problem or is that just how busy the Internet is because of the thousands and thousands and thousands of routers that are out there? I don’t know. I don’t know the answer to that question.

[00:36:08.730] It is old and it has gotten some criticism because if you could do it all over again, create an Internet scale routing system, would it look like BGP? I’m not sure that’s not my specialty. But you got to look and go, wow, it is working. It’s not like the Internet breaks. It’s not because it’s not that the Internet breaks because, you know, BGP itself as a database distribution message bus system is broken.

[00:36:35.670] That’s not the part that’s broken. There are other things that are broken about BGP, but not that aspect of it. So I don’t know. I don’t know how efficient it is, man. I’m not sure.

[00:36:43.920] – Ned
And I didn’t know if you know the answer, but it just it popped into my head because I know that’s something people like to to think about. What’s the most efficient data exchange mechanism for the type of information I’m trying to distribute? And BGP was kind of ahead of its time, it sounds like a little bit in the fact that it’s decentralized and distributed as opposed to having a central store of these are all the routes to everything in the Internet stored in one convenient place? No, it is like you said, it’s it’s constantly converging towards that, but it never gets there.

[00:37:14.490] – Ethan
Not anymore. Now, there was a time, but not anymore, that decentralization is a blessing and a curse. Because if we want to talk about some of BGP problems, kind of stepping back from cloud and looking at the Internet again, it’s possible for anybody, and this this happens from time to time to announce IP routes that they don’t own and all of a sudden BGP starts converging in their direction. When you hear I’ve heard about some big outages like, oh, I couldn’t get to Google Cloud today or I couldn’t get to, you know, a bunch of SaaS services that all live in this particular AWS region or whatever it is. Very often the problem is somebody, some service provider somewhere announced, usually accidentally a route block they did not own that really belong to Google or AWS.

[00:38:02.060] Some region of the planet converged in that direction. And so and basically the traffic was black holed. Sent it to a service provider that couldn’t actually service the traffic because they don’t own the block, that those IP actually live somewhere else. But they announced that they could. That’s me, you know.

[00:38:20.550] – Ned
So so the traffic hit their routers and their routers went, I don’t have that network, I got nothing. Yeah. And I’m not going to pass it through. So.

[00:38:29.010] – Ethan
Or it would loop out, maybe all depending on, you know, how the announcement, where it came from and and how their internal AS’s looked and so on, and that traffic would die. You’d send it somewhere that had no way to service it and no way to get it to where it needed to go.

[00:38:45.160] That’s a problem with BGP and there are solutions on the Internet that are trying to deal with that where so one of them would be origin where you are. I believe it’s RPKI Origin where basically you’ve signed this route and that is in a centralized registry of some sort and everybody subscribes and agrees that, yup, that route came from who it should have came come from. It’s signed appropriately, digitally signed. I will accept this route and trust that the person announcing this to me, it’s coming from an appropriate place. I’m seeing what I should see.

[00:39:23.390] And BGP security, which I did not review in preparation for this podcast, to be honest, is another day, another tool in their arsenal. They’re trying to improve that. But because these are breaking changes, you can’t just like, oh, I’m going to check the box and now I’ve got RPKI Origin verification turned on or I’m signing all of my routes appropriately or I’ve got BGP security turned on. Yeah, no, it’s hard, hard to do that and to do it well.

[00:39:47.320] And there’s never been a flag day for that, that at least that not that Internet wide. We’ve all rallied around and tried to make happen and so BGP still broken and still, you know, careful in that sense. Now if you’re sitting at the edge of the Internet like, you know, me little company, hey, I want to get BGP routes from from you, Verizon or AT&T, like we were talking about earlier, they’re going to clamp down on me.

[00:40:13.880] They’re not just going to accept any route I want to announce to them, no, they’re not going to do that. They’re going to say, what routes do we accept from you? I’m going to send you this one, OK? Prove to us you own that. OK, here’s how here’s see, these are mine. I got these from my my Internet registry here and they’re mine. See, OK, we’re going to accept that one, but no others.

[00:40:35.750] If you send us any other routes, we’re going to drop them. And if you keep sending them, we’re going to turn your circuit down because naughty. That’s how they’ll treat you when you’re just a little peon sitting out at the edge of the Internet Service Providers network. So it’s not.

[00:40:49.410] But you can’t really do that when you’re in the middle of, like the Internet backbone with the big, big routers that are exchanging and millions of routes, or hundreds of thousands of routes and carrying terabytes of data at any given time because the Internet is changing and traffic moves around to different circuits.

[00:41:10.340] And there’s all kinds of peering relationships between service providers that change all the time. So you couldn’t actually filter as a service provider what another service provider sending to you necessarily. I’ve never worked at a servic provider, so I don’t know all the details there, but it’s not the kind of thing where you can just stop accepting routes because you might actually break something else in that way to to solve that Internet. It’s a route leak. They call it a route leak, BGP route leaking problem.

[00:41:37.910] – Ned
OK, yes, I’ve certainly heard of those. And I distinctly remember one that ended up being like an ISP in Pittsburgh where they’d accepted a route from a steel mill or something and they shouldn’t have and then advertised it, which they shouldn’t have. And then it ended up black holing all the traffic for like AWS or some portion of AWS for a few hours. And it was they fixed it relatively quickly. But but yeah, like you said, it’s not instantaneous.

[00:42:06.500] – Ethan
It’s not instantaneous. And it’s the kind of thing that that shouldn’t happen. And it sucks that it’s so easy, even when you’re careful and conscientious that such a thing can happen and propagate out through the Internet. It does blow and it is what it is.

[00:42:18.830] – Ned
Yeah. So if I were to to summarize up BGP as it applies to the cloud architect. Right. Because that’s that’s the perspective I’m coming from. I’m not going to be engineering an ISP network. I’m not going to be running some sort of large WAN implementation. What I do need to know is how to connect my cloud architectures together and also connect them down to on Prem. And it sounds like when I’m creating those direct connect connections or if I’m setting up a VPN tunnel, then BGP is going to be useful to me. But I’m mostly going to be using the sort of internal version of BGP with those private AS numbers that.

[00:42:58.130] – Ethan
Yeah, a lot of the stuff we were talking about with, you know, Internet scale stuff is like fun sidebars to talk about, but doesn’t really apply to nailing up a connection to the cloud. Because exactly what you’re saying, what’s happening with the cloud is BGP is just like it’s like a least common denominator. Your routing device that you’ve got in your shop speaks BGP. They can speak BGP at the cloud provider, there’s a million different BGP Daemons out there. You can run it on Linux. There’s a bunch of different versions of it.

[00:43:24.590] If you can fire a BGP daemon, you don’t even have to have a super special router from Juniper Networks or whoever to run BGP, it’s like anybody can run it on anything. And because of that, it’s just an easy way to. Well, it’s a common way to exchange routes. The message, I would say is keep it simple, keep your BGP connection as simple as possible, even though we talked about the fact that you can get nuts with policies and manipulating routes and you might have to redistribute routes.

[00:44:00.300] You know, if you’ve got a bigger enterprise network, you’ve probably got a network engineer or three on staff that can help, you know, get that stuff done. If all you’re trying to do is just get routes to and from a single VPC, you don’t even need BGP Ned you could for fun.

[00:44:15.720] But static routes will probably work in that situation. But keep your BGP design if you do end up going to BGP direction as simple as you can and you’ll probably be fine.

[00:44:28.080] – Ned
Right, that’s great. So I think my ultimate goal here is when I did talk to networking people in the past and I would try to talk about this sort of stuff, they would look at me like I was an idiot. And that’s probably becuase I was saying, things that made no sense.

[00:44:42.630] But now I feel like through this conversation, I know just enough where I can hold my own in the conversation and correctly tell them the things that I need and not have them look at me like I’ve got three heads, because I think that was I didn’t know how to express what I needed in the language they would understand. And sometimes and I don’t know if you know this, Ethan, networking folks can be a little prickly.

[00:45:05.730] – Ethan
A lot.

[00:45:05.730] – Ned
They can be a little hostile, especially to some guy who just runs servers somewhere.

[00:45:14.760] – Ethan
We’re under a lot of pressure man, you got to understand, is a lot of pressure. As a network engineer.

[00:45:20.250] – Ned
Being able to come to them and speak their language enough to get a point across is incredibly important. I learned that early on with stuff like VLANs and all that kind of stuff. Now I feel like I know it a little bit with BGP and hey, listener out there. Hopefully you now feel a little more comfortable talking about BGP as well. I want to say thank you, Ethan, so much for taking some time out, doing some remedial reading. I’m sure it was super enjoyable to educate me a little bit on on what’s going on with BGP.

[00:45:50.910] – Ethan
And for you people out there that are listening. I know a lot of you are packet pusher’s listeners and you’re also listening to Day Two Cloud and you’re like Banks got this wrong, woah Banks got that wrong. Sorry, there are some probably some little details and things where I may be misspoke along the way. Feel free to beat me up in the comments or DM me in the Slack group or something like that, if you want. I apologize because I don’t live in the BGP world all the time. And if one of you were like super BGP cloud nerds and you want to come on a show and we do a like a reboot of good BGP cloud designs and you know, you’ve had experiences or you’re bearing the scars. We can have that conversation on the podcast. Just a ping Ned and I on the show.

[00:46:30.090] – Ned
Yeah, we’d love that. And anything else that you are interested in talking about hit us up. You can hit us up on Twitter. It’s at Day Two Cloud show. And you know, Packet Pusher’s does have a Slack channel, so you should definitely check that out as well. I believe you can find that on the packet pusher’s website. Is that correct Ethan?

[00:46:45.420] – Ethan
Packet Pusher’s dot net slash Slack.

[00:46:48.700] – Ned
Awesome. Hey, cirtual high fives to you for tuning in and listening to this entire episode, because I know you enjoyed it. If you like engineering oriented shows like this one, you can visit Packet Pusher’s dot net slash Subscribe all of our podcast newsletters and websites are there. It’s all nerdy content just like this designed for your professional career development. I feel like I developed a little bit on this one until next time. Just remember clouds, what happens while it is making other plans.

Episode 91