Follow me:
Listen on:

Day Two Cloud 132: What Web3 Means For Infrastructure Engineers

Episode 132

Play episode

Web3 is the term for an emerging technology movement that aims to create a more decentralized Internet and put more ownership in the hands of individual users and consumers. As of early 2022, Web3 is most closely associated with faddish-seeming cryptocurrencies and NFTs, but it’s worth understanding the technological underpinnings of Web3, particularly blockchain and its broader applications. We also discuss business models driving Web3.

Our guest to guide us through Web3 is Josh Neuroth, Head of Product at Ankr.

We discuss:

  • Web3 terminology, including dApps, blockchain, and nodes
  • Web3 as cultural and technological phenomena
  • Blockchain/distributed ledger fundamentals
  • Proof of work vs. proof of stake
  • Smart contracts
  • Enterprise use cases for blockchain
  • Blockchain software stacks
  • Operations and management challenges of running highly distributed systems
  • More


  1. Web3 can feel daunting. Get started with a wallet. Metamask in the Chrome web store.
  2. Buy $20 worth of crypto on Coinbase and put it in your wallet. Start connecting to dApps.
  3. Get a cheap web server and launch a node. Start playing around with it. Or if you don’t want to build a node, query Ankr’s public (rate-limited) RPC API to connect to the blockchain.

Show Links:

How Infrastructure Providers Can Enable Web3 – The New Stack

@0xJosh.eth – Josh Neuroth on Twitter

Josh Neuroth on LinkedIn


[00:00:04.250] – Ned
Welcome to Day Two Cloud. Today’s it’s going to be an interesting conversation. It might be a hot button topic. We’re talking about web3, but not just web3 in the abstract sense. We’re going to get down to what it actually means to the Infrastructure Engineer. And we have a special guest, Josh Neuroth. He’s a head of product at Ankr, who is a service provider for hosting nodes that run blockchain. So he might know a thing or two about a thing or two, right, Ethan?

[00:00:30.340] – Ethan
He knows more than a thing or two about a thing or two. He knows a lot he knows a lot about the culture, the environment, use cases, system requirements, and so on, along with all the terminology that if you’ve been reading about web3 and your head’s swimming because all the jargon is just kind of drowning you, Josh can clarify all of that for you.

[00:00:47.800] – Ned
Yeah, and I like jargon, but my goodness, there’s a lot of jargon. So if web3 is piquing in your interest, enjoy this episode with Josh Neuroth, head of product at ankr.

[00:00:58.290] – Ned
Well, Josh Neuroth, welcome to Day Two Cloud. Before we get into the topic at hand, why don’t you introduce yourself to the fine audience out there?

[00:01:07.980] – Josh
Thank you. Yeah. My name is Josh Neuroth. I lead the product team over at ankr, which is a web3 infrastructure company. And I’ve spent most of my career in product management last seven or eight years kind of geeking out in infrastructure, working at Infrastructure as a service companies and building solutions for some of the world’s leading networks and companies.

[00:01:31.210] – Ned
Okay. Well, that dovetails nicely into the conversation we want to have because we wanted to talk about the infrastructure side of web3 and understand from an engineering standpoint what’s required to run web3 components. So let’s get started with some fundamentals. Ethan and I are mostly web3 Newbies. We don’t know that much. We’ve done a little bit of reading, but terminology is very important here. So what does web3 actually mean to you?

[00:01:59.320] – Josh
Yeah, absolutely. So I would think about web3 at this point in time in early 2022 is not just a technical system. It’s also kind of becoming a cultural movement, as we’ve seen athletes and celebrities talking about it. The NFTs are everywhere now. So on the cultural side of web3, I would simply say that web3 is a way to organize or coordinate large groups of people in a way that is fair and meaningful to them. And then on the technical side, web3 is really using blockchain technology to build a distributed and a decentralized system that is essentially everywhere in the world.

[00:02:44.350] – Ned
Okay. And I think we’re going to focus more on the technical aspects. The cultural aspects are a little fraught at the moment.

[00:02:50.210] – Josh
Yeah. To say the least.

[00:02:53.430] – Ned
But Interestingly, the first thing that you picked out was decentralized. But the other two big terms that I tend to hear associated with web3 are blockchain and cryptocurrency. Are those necessarily tied to web3 or are they all kind of one package?

[00:03:11.610] – Josh
Well, yeah, web3 is definitely using the blockchain. I’d say cryptocurrency is just a small sliver of what web3 is. I think the database, the decentralized database is really at the heart of it. And so essentially what we have in web3 is we have something called the distributed ledger. Think about it as a database that is maintained by nodes all across the network, servers that are running the database. And that database is kind of public to whoever anyone can view at any point. And there’s applications and different use cases that can be built on that database. And so if you think about it, where does crypto come into that? It’s very similar. Like when you log into your checking account at your bank and you see the balance in your checking account, there’s some database that the bank is running that says you have X amount of money. Right. And then every time you use your debit card that database subtracts some amount from that. Right. And that’s effectively how crypto is using the blockchain.

[00:04:23.470] – Josh
Right. And so web3 is something that is kind of living on the blockchain. Right. But it can or cannot include crypto. Right. Like, that’s just a piece of it.

[00:04:35.350] – Ethan
Now, you said the blockchain several times, but in reality, there is more than one blockchain that could be participating in. Right?

[00:04:43.570] – Josh
That’s absolutely right. Today we see there’s dozens of blockchains. There’s probably hundreds if you go down the long tail and look at all the ones that are out there. But some of the most common ones that you may have heard of in the news are like Solana, Ethereum. Bitcoin is a blockchain, too, but Bitcoin is a lot different from some of these other chains.

[00:05:07.730] – Ned
Okay, yeah. I think Ethereum is probably the one that people are most familiar with because I think that’s kind of what the whole NFT Craze was mostly using. Right.

[00:05:17.050] – Josh
Yeah. So let me dive into a little bit about the technical details here. So Bitcoin was essentially the world’s first blockchain. It uses something called proof of work, which you guys are probably familiar with the concept of miners that are using all sorts of electricity to process the next block. Right. We’re using like 1% of the world’s electricity or more now for Bitcoin, but Bitcoin is kind of I’m going to offend some people by saying, but it’s kind of a dumb blockchain in a sense, like it’s just used for storing basically, who has the bitcoin, what wallets contain Bitcoin? Now, Ethereum at the moment is still using the proof of work. It still has the mining infrastructure, but it’s moving to something called proof of stake. And what makes Ethereum different from Bitcoin is the concept of a smart contract. Right. So intelligence that essentially lives out there in the network.

[00:06:17.170] – Ned
Can you expand on that? So what do you mean by it lives on the network?

[00:06:21.100] – Josh
Yeah. So in Ethereum and these newer proof of stake based blockchains, it’s almost like a program, a script can live out there as part of the database. And when certain conditions are met in the network, that script will auto execute. Right. And so that’s what makes some people refer to Ethereum as a global supercomputer because it’s not just about who has Ethereum and who’s spending Ethereum on something or the cryptocurrency. It’s also about programmatically executing code or logic in the network when some condition is met.

[00:06:58.990] – Ned
Oh, interesting. Okay, so that would kind of be like when I had SQL triggers back in the day on my SQL database, something happens and that would have a trigger run to execute a program. Well, I guess the program kind of also does live in the database, doesn’t it? So, yeah, in that way, the two are very similar.

[00:07:18.440] – Josh
Yes, exactly. So if you think about it, like the if/then logic, you could write a smart contract and you upload the smart contract, you pay some gas fee or some Ethereum to upload the smart contract. And then that smart contract lives across a distributed ledger. The distributed database, and when its conditions are met that are programmed into it will execute something. So people have found lots of ways to use the smart contract. But it’s really interesting from my point of view, what kind of makes me excited about is this logic that lives out there in the network. Right. And so people can see they can do smart contract audits. They can see what’s about to happen on the network when certain conditions are met. And there’s just so many use cases that can kind of branch off from that.

[00:08:09.190] – Ned
Right. Because a pretty important condition and this is something that you brought up earlier is that the blockchain is public.

[00:08:15.410] – Josh
That’s right.

[00:08:16.160] – Ned
Everybody can read and parse through the items that are stored on the blockchain.

[00:08:20.290] – Josh
Yeah, exactly. So, I mean, people are using smart contracts now to auto invest digital assets or to buy digital assets. So I said earlier that in the cultural side of things, people are excited about things being fair. And so what I meant by that is when you have a smart contract that is auditable by anyone, anyone can read it and see what’s about to happen. Say you promised your community that they’d get 10% of ownership of something that can be written into a smart contract. So it’s auto executed. Right. And that’s why I think people are excited is because it brings a level of transparency that we really haven’t seen before in the Internet.

[00:09:06.830] – Ned
Okay. All right. My brain is already lighting up. So another term that I’ve heard is a dApp.

[00:09:16.480] – Josh

[00:09:17.610] – Ned
What is the dApp? Can you maybe explain a little bit and then give a few potential use cases for that type of application.

[00:09:25.290] – Josh
Sure. A dApp. So the term itself really means a decentralized application that is a little bit misleading. And it’s from a generic definition, because many dApps use a traditional web stack to run their applications, they’re browser based web applications. They’re using JavaScript. They’re using. Maybe they’re even hosting on Amazon Web Services or Vercel. Or they’re using a standard it’s a standard web application. What makes it a dApp, at least from a technical standpoint, is that the user authentication is being done by the blockchain or by a blockchain. So you’ll see in the dApp that there is instead of a login button where you log in, maybe with your Google user account or Twitter user account or email and password, you connect your wallet. And so this is something that is really interesting, I think is a defining factor of the web3 movement is this login with wallet system that’s being built. And so you guys are familiar with logging into a server with your SSH key. And with a wallet, you have a public and a private key pair. So I have a public wallet address, and my Ethereum wallet starts with zero X, and then the whole hash.

[00:10:48.780] – Josh
Right. And then I have my private key, my private seed key, which no one else knows. And so if you think about the wallet, then it’s almost like SSH for all these applications, right?

[00:11:02.040] – Ned
Okay. Rather than having an email address and a password that’s stored somewhere in that system, all it has is the public key. And then if I send it, I won’t send it the private key, but I would somehow match the two up.

[00:11:18.100] – Josh
That’s right.

[00:11:18.740] – Ned
Proof of my identity. And the dApp doesn’t actually have any additional information about me and can’t leak my password by accident, correct?

[00:11:28.790] – Josh
Exactly. So you authenticate through your wallet and then that dApp is going to know from my public key that I’ve authenticated to their application. Right. And then there’s information that that dApp can see about me, maybe a list of the transactions I’ve completed in the past. What NFT’s I own things like that. But effectively, a dApp is just a way to say there’s an application that’s using this decentralized identity in it, and that’s the user authentication.

[00:12:02.670] – Ethan
So there’s nothing necessarily magical about the application itself. If we use the client server architecture, it sounds like it’s still an executable or microservices or whatever it is running somewhere. And the whole magic and why we call it a dApp, truly it’s just tied to wallet based authentication.

[00:12:19.590] – Josh
Yes, in most cases now there are people working on decentralized cloud infrastructure where they’re kind of pooling together compute resources to build like a user ran and operated version of AWS. I guess you could say that you could build a dApp on that as well. But for the most dApps that you see, at least today, the dApps are running a standard web architecture with microservices and just the user authentication is happening through a blockchain and in that decentralized authentication.

[00:12:57.210] – Ned
Right. And part of that, I think, correct me if I’m wrong, but because of the decentralized nature of the blockchain and also how long it takes to execute a transaction that’s not really suitable as a general purpose database, it’s really just for very specific types of transactions.

[00:13:14.150] – Josh
Yeah. That’s I think a criticism of where the technology is at right now. So in proof of stake, every time you do a write to the database, to the distributed ledger, you need to pay something called a gas fee. Right. The gas fee is what is the economic mechanism that maintains the blockchain and the system. And so in Ethereum at the moment, gas fees are very expensive. To send a transaction. You’re looking at a $20 to $30 gas fee at the moment. And that fluctuates up. That goes up and down right now on other blockchains, like Solana, the gas fee is like less than a penny. So you have different blockchains that are addressing the gas fees. And some of the blockchains, the cost is approaching zero. So if you think about all the costs that go into running a standard database, you need servers, you need DevOps, you need standard infrastructure, you’re effectively getting to leverage all of that. The blockchain is running just by paying the gas fees. So I think as gas fees goes down, there’s a lot of people working on solving that problem. We’ll see greater enterprise and user adoption of these systems.

[00:14:31.980] – Ethan
When you talk about those gas fees, you couch them as actual dollars versus cryptocurrency denominated.

[00:14:39.780] – Josh

[00:14:40.730] – Ethan
Is that what you meant or did you mean it’s some fraction of proof of work or something that you’re more or less penalized for?

[00:14:48.930] – Josh
Well, I think the gas fees are in the native token of the network. But from because of those, there’s so many blockchains now, people still use the US dollar as a way to kind of comparing and relating the economic costs of doing transactions. Right. And so you’re right. The gas fee is like point one Ethereum, or whatever it is today. And that fluctuates every day. But most people what is meaningful to most people is what it’s costing them in their Fiat, what’s costing their bank account.

[00:15:31.180] – Ethan
Right. Which fluctuates wildly depending on what’s happening in the markets on any given week.

[00:15:35.870] – Josh
Yes, exactly.

[00:15:37.170] – Ethan
All right, Josh, so we have an idea of infrastructure and what’s happening here, the decentralized nature and such. But man, I have all this functionality in Web 2.0. I mean, this I can do today. Pretty much. There’s centralized banks behind it and there’s centralized major cloud services, the big three and many other companies coming up with their cloud infrastructure. What am I doing here? Is this going to fundamentally change how I interact with the web? If I go the web3 way, I haven’t got the thing yet. That’s like, yes, I’m excited. I want to go web3!

[00:16:16.610] – Josh
Sure. Absolutely. And I think there’s a lot of people in that boat. Right. That even have a high degree of skepticism for where this is going to go. Let’s talk a little bit about, I think, some of the themes that are kind of driving maybe the excitement that’s out there. So in Web 2.0, there’s frankly, two different business models that have worked. Right. You have your advertising driven business model, which is what Google and Facebook use, and then you have your subscription model, pay as you go, usage based or standard subscription, Netflix, AWS, LinkedIn Premium. Right. You have these different subscriptions. And early on in the Web, there was other business models that were kind of tried, but there wasn’t really the technology to allow those business models to work. And so what web3 is doing, I think primarily is it’s bringing a new business model to the table, one where your users of your applications can share in the ownership. Right. And there’s effectively the Internet of money that can be paid to them for doing different things. Right. So if you think about it, you have credit card infrastructure is predominantly used in Web 2.0, but there’s just not the best integration.

[00:17:39.050] – Josh
Think about this. If you’re a contributor to a website like Wikipedia. Right. Are you going to send them out a payout to their bank account every time they contribute something, or is it volunteer based? There hasn’t been like a native way in the Web from an economic standpoint to pay people. And that’s really what web3 is bringing to the table is the economic system.

[00:18:07.830] – Ethan
So you’re saying someone can pay in cryptocurrency, we’re participating in the same blockchain, and web3 infrastructure is going to make it easier for me to pay Ned in some crypto because I really like that blog post and learned a lot.

[00:18:23.470] – Josh
Yes. But I think it goes a lot further. So there isn’t really a way in Web 2.0, for machines to pay other machines. Right. And so that is a really interesting use case that I think is very early right now in web3. But when we think about systems being able to incentivize other systems economically and read the entire database, that is a really interesting thing. I don’t know. Maybe there’s an API integration that’s happening between two different systems today that would be almost impossible to have machines manage that you have to connect to someone’s bank account, you have to read it. There’s so many different things there that make it very difficult. But when everything is out there on the blockchain and these machines can read it and like I said earlier, with smart contracts execute code based on certain conditions, you have a system where machines can incentivize other machines without human involvement.

[00:19:29.330] – Ethan
When you say incentivize, I interpret that to mean from my very nascent reading on web3 that we’re talking about participating in the blockchain and doing work or whatever the version of work is for that blockchain to earn tokens. And the more tokens you have, that’s your incentive, right?

[00:19:44.540] – Josh
Yes, absolutely. You can earn a fraction of a token. Like, you don’t have to have, like, one Bitcoin or one Etherium. You can have 0.001 of that. Right. And so there’s ways to do that. So I think we’re very early in that. But people have been talking about this idea of kind of a connected world, a fourth industrial revolution where we have sensors, we have machines connecting and talking to each other, whether it’s self driving cars or data. And what’s kind of held that back is that those systems can’t really talk to each other today unless there’s a lot of manual integration of these systems. And that’s where I see blockchain networks helping with that.

[00:20:32.050] – Ethan
You’re talking about a protocol that’s going to facilitate easy communications machine to machine here. But I think there’s another challenge, Josh, which is just simply the number of blockchains that are out there, the number of token systems that you can participate in.

[00:20:46.820] – Ethan
Yes. Ethereum, yes. Bitcoin, Solana. These are the big ones. But every system is unique. Just like we have a number of global currencies distributed by various countries. We have a similar challenge today. So it feels like almost like you got to place your bet on something. But there’s what are we up to? Hundreds of cryptocurrencies? I don’t think it’s thousands. Maybe it’s even thousands. I don’t know.

[00:21:11.320] – Josh
Yeah, it is thousands. Now, most people are only probably exposed to the top 50 of the top 100. But, yes, we are in the thousands at this point.

[00:21:20.310] – Ethan
So if I want to be incentivised to earn a token on a particular blockchain, I got to place a bet and hope that I’m earning tokens in a system that is going to be ultimately globally recognized so that I’m earning value in the right system, that enough people have put their trust in, that it has value. So they can do exchanges one to another.

[00:21:42.810] – Josh
Yes. And platforms like ankr are working on making that what we call a multi chain interoperability easier. Right.

[00:21:55.360] – Josh
And so there are solutions that programmatically, you can take payment in one token and then trade it or swap it for something else and track all of that. Again, very early days, I think, on these. But we do see that most of the adoption is happening around 15 different blockchains right now, which is a lot still. But you have Ethereum, Solana, Binance, smart chain. I’ve seen Phantom, Polygon, kind of the name brands that most applications and dApps have really started to integrate with. Right. And so there’s a long tail of all these blockchains. Think about it as like a programming language where JavaScript and Python are probably the most common languages now used out there in the web. But in other days, there were different ones. Cobalt, C++, and it’s very similar with blockchain. I think as a developer in this space, you can pick the chain which works best for your community. You can pick the chain that works best for your economic mechanism and go from there, potentially integrate others as you need them later on.

[00:23:12.700] – Ethan
But there’s nothing stopping the big guys from getting in and starting their own blockchains and trying to capitalize on this, which I mean, Google just announced this. They’re hiring people, they got a blockchain division or whatever they called it. So what does that mean for the culture of decentralization?

[00:23:30.270] – Josh
Yeah, absolutely. Well, every blockchain has a trade offs, so Solana has gotten some criticism because it’s a little bit more centralized than Ethereum, but it also has a lot higher performance. And so I think everyone can pick those that really care about the decentralized movement. You can pick something like Ethereum, of course, there’s higher gas fees, it’s a slower network, but you do have this very decentralized system. And so being that blockchain is a distributed ledger, if there are enterprise blockchains that are starting to emerge and many of them are centrally controlled. But I think a lot of those same tenants are still prevalent in those like the transparency, anyone being able to authenticate transactions and see what’s happening. So, frankly, there’s going to be a scale of very decentralized systems and very centralized systems, and they’re going to leverage a lot of the same technology. But it really ends up being like, who’s in control and where is it going?

[00:24:31.770] – Ethan
Describing a world, Josh, where the principle is the same across all the blockchains. And I, as a consumer would have my wallet and that wallet. I wouldn’t have 15 wallets, one for each blockchain I’m participating in. Or would I? It sounds like I could have one wallet that participates in multiple blockchains.

[00:24:49.950] – Josh
Yes. So predominantly now we have something called on the Ethereum network, we have something called the Ethereum Virtual Machine. It’s frankly, the software that runs the network and all these other blockchains, like Polygon Arbitram Phantom, are what’s called EVM compatible or Ethereum Virtual Machine compatible, which means I can have one wallet address that I can use any of these blockchains. Now, Solana at the moment is not EVM compatible, so I need a separate wallet out for Solana. Right. But there is a lot of work going into making these compatible and essentially having like a super wallet app where I have one wallet and can use any blockchain.

[00:25:32.310] – Ned
So, yeah, you have that explosion of initial creativity, and then the standardization starts rolling in.

[00:25:37.370] – Josh
That’s right.

[00:25:38.790] – Ned
And that’s when I think enterprises start to become more comfortable with a concept or an idea is when standardization rolls in the second generation, because right now, when I think about web3, it seems very bleeding edge. It seems where like when people are talking about Kubernetes five years ago, it was the first movers. It was the bleeding edge folks who were doing it. To what degree are you seeing enterprises exploring the web3?

[00:26:05.850] – Josh
Yeah, well, I can tell you from our work at ankr. We’re an infrastructure provider and we have RFPs from almost every major financial institution. So they’re building just like you said earlier, like Google’s building a web3 team, the payment providers, the credit card providers, the standard US based banks, they all have a web3 strategy. They’re tasking their infrastructure teams and their software engineering teams, figuring out what to do. How do we get into this? How is it going to disrupt us? And they’re still early days. In fact, I’d say the last six months has really the second half of 2021 was really when we started seeing that traction. But they don’t want to get too far behind in this because they see it as a disruptor to them.

[00:27:00.390] – Ned
Okay, so, yeah, it’s still early days, but I’m sure they’re spinning up some kind of pilot project or skunkworks in one of their development groups.

[00:27:08.150] – Josh
Yeah, absolutely. And gaining the resources on their team so they know how to work with this technology.

[00:27:14.250] – Ethan
Now we’re talking about enterprises that are putting RFPs out there because they’re interested in this. Josh, if I am an infrastructure architect or cloud engineer working for one of these companies, what should I know about web3? So that I’m properly informed, participating in the conferences appropriately?

[00:27:34.470] – Josh
Yeah. So a couple of things. If you’re an infrastructure professional, what we see predominantly is you need to connect to the blockchain. You need to start being able to query and look at information. And to do that, you either need to run your own node, which the node is just an application that runs on a server that connects and operates the blockchain. Or you need to work with a node as a service provider like ankr.

[00:28:06.930] – Ethan
Would miner be another name for node?

[00:28:09.880] – Josh
No proof of remember, we talked a little bit about the consensus mechanism for different blockchains. So Bitcoin, actually, everyone is familiar with Bitcoin miners, I think, but lesser known is that if you want to process a transaction in Bitcoin, you need something called a Bitcoin full node. The same goes with Ethereum, where Ethereum has in version one of Ethereum, we have miners. Version two, we have validators that are the block creators. But you need something called a full node to query information on the blockchain to send transactions, any kind of development you’re doing. You need a node that can pull information off the blockchain. And that node is just a standard application that runs on a web server. You can launch one yourself with the open source documentation that’s out there. Or you can use a provider like ankr to do that for you.

[00:29:05.010] – Ned
Okay, so we’ve got a few different node types that are participating in the network. Size of a node am I talking about here? Am I running this on a Raspberry Pi, or do I have to buy like a $3,000 graphics processing engine to run it all?

[00:29:19.920] – Josh
The requirements change depending on the blockchain. Let’s talk about Ethereum. Ethereum is you could run a node on a Raspberry Pi. The performance is going to be probably terrible. It’s possible if you’re trying to do something yourself as a hobbyist, you definitely could do that. For us at ankr, we use kind of a standard configuration of a standard enterprise server. We use an AMD Epyc as our CPU, 128GB of Ram, and then NVMe storage. And you’re looking at to run at least a full node, you definitely will need at least a terabyte of storage to store the latest transactions. Now if you want to get everything and store a complete copy of the blockchain on your node, you’re going to turn your node into something called archive mode and then you need to have a lot of storage. So we have up to 30 terabytes of NVMe per server to do that. We’re using a lot of NVMe storage now just to get that speed, that fast read and write. But again, think of it as a database server. So you need the fast IO is what really matters. I can talk to you all day about our servers.

[00:30:35.950] – Ethan
Blockchain is essentially a math problem. And so at the end of the day you need something with some heft. Putting things in memory is going to help. Reducing IO swaps is going to help. But ultimately, even if it’s just a node and you’re not mining, there’s math that’s got to be done to validate what you’re seeing on the blockchain. And again, Josh, you mentioned archive mode. If I want a copy of the entire blockchain, I’m needing terabytes. And that storage requirement is going to increase the more transactions that are committed to the blockchain.

[00:31:07.760] – Josh
That’s absolutely right. Yes. So we’ve seen the Ethereum archive at the moment is around. I think it’s around nine terabytes as of this month. And that is going to just keep increasing over time. Because think about this database is like a stack of Lego bricks with a new block. That block contains transactions ever increasing. So the requirements do go up all the time.

[00:31:31.330] – Ned
Is that a long term problem that could be solved with a new version of a blockchain? Because I’m thinking if Ethereum sticks around for ten more years, are we going to have enough storage to even hold the and Joe Average is not going to have enough storage to hold.

[00:31:46.730] – Josh
Correct. This is a big topic in the community right now around what to do. There’s protocols like Graph Protocol that are pulling this data and indexing it, and you can load up Graph QL and quickly query information. There’s node providers like ankr that are addressing this too. But yeah, there are many solutions out there. Some are open source that are being worked on to make these requirements less. But I think this kind of goes back to as usage grows. Right. The requirements are definitely going to grow. And so you have, like, Solana. Let’s talk about some of the trade offs. So Ethereum, it’s global network of Ethereum at the moment is limited to 20 transactions per second across the whole world. All the users. So that doesn’t sound like a lot. Right. Very slow. Right. But you have the node requirements are, like you said, you Colo run one on a Raspberry Pi. And so most average user does not need a complete copy of the blockchain. They’re going to use, like a service provider or a protocol like Graph protocol to kind of pull that information. But to speed up the blockchain to get that performance, those node requirements are going to increase.

[00:33:08.550] – Josh
So with Solana, we see you need a very powerful server. I mean, I’m buying servers with 512GB of Ram right now for Solana. Right. And so the average person is not going to be able to afford that on a monthly basis to run a Solana node.

[00:33:24.920] – Ethan
The problem here is consensus, Josh? That is all the nodes need to come to consensus that this block is valid. And so there’s a lot of math that’s got to happen. You’re running it on our Raspberry Pi. It’s going to take a lot longer to do that math than if you’re running out on some beastly machine.

[00:33:40.470] – Josh
Yeah, that’s a simple that’s a high level way of describing it, essentially. But you’re right.

[00:33:47.270] – Ethan
That’s alright man. You can call me simple. I got this.

[00:33:48.130] – Ned
That’s what I heard.

[00:33:52.530] – Josh
What I do want to point out is we’ve seen the best performance on bare metal. So early days ankr, we started using virtualization, virtual machines. We were running nodes in VMs, and as of now, we’ve almost completely cut over to running on single tenant bare metal, which we’ve seen the best performance on for sure.

[00:34:13.610] – Ned
Okay. And is that partially because of the software itself? It’s able to run multi threaded across the whole processor. So some of the gains you would normally have from Virtualization don’t really apply.

[00:34:25.150] – Josh
Yeah. I mean, the thing about blockchain is the blockchain itself is redundant. So if you cut out a node, the blockchain itself is fine. So as long as I have what we do at ankr is we have a load balancer with many nodes. If one of those nodes fails, it’s fine. We just load balance the traffic or the read queries across other nodes because the resiliency and the redundancy is built into the network itself. You don’t need all of the traditional enterprise level redundancy that you get in virtual machines. Right.

[00:35:06.440] – Ned
So the thing that I would do, like taking snapshots of the virtual machine for backup and recovery, or even replicating between the nodes, well blockchain is already replicating.

[00:35:15.940] – Josh

[00:35:16.510] – Ned
That’s already taken care of for me.

[00:35:19.230] – Josh
Exactly. So we’ve tried to cut out all unnecessary redundancy in our server configurations and just basically spend that trade off on performance. So a lot of enterprise grade servers have like two power supplies or even two CPUs or redundant RAID-1 or some kind of configuration like that. We put our disk our nVME disk in RAID-0, which obviously, if one of the drives goes down, it corrupts the volume. But that’s fine for us. We want that performance, and we can do that because of what you just said. Because the redundancy is built into the network.

[00:36:00.540] – Ethan
It’s better to spend money on more nodes and scale out rather than make sure that this one server is so precious it can never go down. Sure, it can go down. You’ve lost that compute capacity, but you’re saving the money and throwing it at more nodes out there.

[00:36:16.470] – Josh
Yeah, exactly. Now, there is an exception to this, and that’s when you are running what’s called a validator, most engineering teams that are tasked with looking at web3 strategy are going to need to use the full nodes, which is what I just said. Well, you don’t need that redundancy. But if you are a financial company and you’re tasked with running some validator, which most financial companies wouldn’t run them themselves. But the validator is the block creator. And why is that? Why would a traditional financial enterprise be tasked with that? So in the proof of stake system, you essentially put a security deposit onto the validator, the block creator, and then you have passive earnings that the network pays out to you. Right. And so you can get five, six, 7% APR from the network by running a validator. And so the validator, you want the redundancy. You want it to always be online because you’re earning some kind of financial asset for running it. And in that case, you do probably want the virtualization, you want the redundancy, and all the traditional, I guess, enterprise requirements for security and redundancy apply in that case.

[00:37:35.970] – Ethan
You said you’re getting the performance out of bare metal. Okay. In a world of containers and virtual machines and Kubernetes, it sounds like web3 doesn’t really play there. Or does it? Do you see cloud native and Kubernetes kind of infrastructure employed at all?

[00:37:51.020] – Josh
We’ve tried it. We’ve built a whole Kubernetes Docker, container based deployment process for our full nodes. And again, we just didn’t get the performance that we needed out of that system. And so we switched to running our nodes directly on Ubuntu, on bare metal. We have seen by far the best performance using AMD, AMD CPUs on that.

[00:38:19.740] – Ethan
Do you remember what the penalty was that you paid for running it? Well, basically, abstracted in a container.

[00:38:29.130] – Josh
More than 30 or 40%. The thing is that when you get into these in blockchain, the most important requirement in most of the chains. I’m taking a very generalized rule here is your disk IO Ops, right. In our Binance smart chain configuration, we’ve consistently needed more than 10,000 IO Ops. Right. And so we’ve tried very fast network attached storage did not get performance. The virtualization sometimes decreases the IO Ops as well. So we ended up just getting NVMe on the actual box and just putting single tenant nodes, one server, one node with its own NVMe. And that was the best solution for us.

[00:39:24.410] – Ned
It’s one of those cases where it kind of reminds me of high performance computing, where they take a lot of the same approaches because they know that there’s a performance penalty for every abstraction you layer on to things.

[00:39:36.440] – Josh
That’s right.

[00:39:37.120] – Ned
The closer you get to bare metal, the better, the performance. Disk IO especially is the case because it doesn’t have to time slice the IO if there’s really just like one process hogging it all up.

[00:39:48.920] – Josh
Yeah, absolutely. So we’ve been so happy with our NVMe based storage, but the thing is, we had to go build our own server because no one was selling 30 terabytes of NVMe on a box. Right. Like, we went to Equinix, we went to all the bare metal providers, and no one had that. So we’ve had to kind of custom order that and build in some cases. Now we’re doing our own colocation as well.

[00:40:20.270] – Ned
Interesting. I would imagine that the newer form factors for SSDs and NVMe has paid off for you. That sort of ruler format that intel introduced, because that really lets you pack in possibly a petabyte of storage in 1U if you want to spend the money for it.

[00:40:37.800] – Josh
Yeah, we haven’t had to go that far yet. And again, those solutions are a little overkill at the moment for us, but we’ve had great success with just the super micro boards. Put the AMD Epyc in there, add some fast RAM, and then your NVMe drives got you.

[00:40:54.800] – Ethan
You guys didn’t try fiber channel for storage? That’s actually a serious question.

[00:41:02.250] – Josh
We’ve tried so many different things. Like I said, the best trade off between price and cost and performance was just standard NVMe. Finding a server chassis that had ten drives or so and put those in RAID-0.

[00:41:19.970] – Ethan
Running a fiber channel network would add a lot of complexity and cost to the solution, for sure. But I think you would potentially get the throughput guarantees you’re looking for with, again, the trade off of complexity and cost. But NVMe, I understand exactly why you are where you are. It’s just you’re in this world now of needing to manually rack constantly and then add the node to the network and all the rest of that. So it feels like architecturally a step backwards from what we’ve been doing all these years, where we’ve extracted away the hardware as much as possible, repackaged things into containers, and just made it. I don’t care about the hardware, just rack it, add it to a Kubernetes cluster, and boom, throw the workload on it, and we’re done. Walk away, move on to the next step.

[00:42:00.420] – Josh
Yeah. So another thing, too, is that there’s something very important called block height. So when you run a node, your node has to stay in sync in real time with all the other nodes on the network in the peer to peer system. And so the term for that is called block height. And so you need enough CPU, you need enough RAM, and then this speed to stay at block height. And that’s the most important thing. Right. So you’re constantly streaming information to the other nodes. Now, most networks, the actual network requirements really aren’t that significant. You’re looking at. I think the average I see is around 50 megabits per second per node. So you don’t need like tons of bandwidth to run this stuff. However, right now all the full nodes are completely based on layer three using Internet protocol to connect to each other. But we’re starting to see a few interesting use cases for building kind of like layer two based VLANs between our nodes and whatnot.

[00:43:06.150] – Ned
Yeah, I could see where at least in terms of you do need that extra network capacity. For some reason that would be helpful, but because these are all single tenant machines, you don’t really need the network isolation that you would typically use a VLAN for.

[00:43:19.970] – Josh

[00:43:21.150] – Ned
Now, I did have a question in here about using the public cloud to host nodes for web3, but based off of what you’re telling me, it sounds like that would be cost prohibitive for most cases.

[00:43:34.750] – Josh
Well, I think there’s a trade off of the easy button, verse doing it yourself. Right. So a lot of our competitors do host on Google Cloud and Amazon Web Services almost exclusively. There’s like an altruistic reason, I think, for us to not host in the public cloud to help with the decentralization. A lot of our community members are running our nodes, and we want to do that on the bare metal or in the Colo environment for that reason. But I will say we get a lot higher performance for a fraction of the cost of bare metal. And so from a technical point of view, it makes a lot of sense for us to stay on bare metal.

[00:44:18.400] – Ethan
So ankr, you guys are crypto bros, and those people using the cloud, they’re all the button down suits.

[00:44:29.570] – Josh
A lot of people still love the cloud, and I think there’s been a lot of speculation in our space around who’s building like the AWS of crypto. And to me, it’s like AWS is going to become the AWS of crypto. Eventually you’re going to have a button on AWS or a script on AWS where you can just launch a node, and that’s already starting to manifest over there. So Cloudflare has a web3 strategy as well. As you mentioned earlier, Google does as well. And so eventually spinning up a blockchain note is going to be as simple as just clicking a button in AWS.

[00:45:02.690] – Ned
Right. Azure announced a blockchain. I want to say it was like four years ago, blockchain service on Azure, and then they kind of shut it down after a couple of years, I think, because it wasn’t ready yet.

[00:45:13.510] – Josh
Correct. Yeah.

[00:45:14.740] – Ned
And I got to imagine once bitten, twice shy, they might wait a little while before they have their entree onto.

[00:45:22.390] – Josh
Yeah. The market is still relatively small. I would expect this year to see a lot of announcements from the big cloud providers around their strategies in this. We’re kind of at that point, that critical tipping point of mass adoption now. And the fact that their customers are probably asking if they’re sending us RFPs are definitely sending the cloud hosting providers those RFPs, too. So we’ll see you follow the money. It’s going to happen soon.

[00:45:49.790] – Ned
All right, Josh, let’s bring this in for a landing. If folks have been listening to the podcast, what are some key takeaways or actionable items they could have moving forward?

[00:46:01.550] – Josh
Well, I think the first is that web3 can feel very daunting when you’re starting out. And I think the first thing that the average listener should do is create a wallet. So go get there’s an open source wallet called MetaMask. It’s out there in the Chrome Web store. Metamask is just a wallet client. You can go create a wallet in a matter of minutes. You get your private key. You should save that. You can save a paper version of that, too, to put it in your safe. You can use a password manager, whatever it is, but get a wallet. I would recommend the thing that I did early on to just get started is go buy like $20 worth of crypto on Coinbase or something like that, transfer it to your MetaMask wallet and then start playing around in space. Start connecting your wallet to some of these dApps out there. There’s many dApp directories now of all the different applications, there’s games starting to use wallets. And I think just start playing around with the technology, maybe mint an NFT or something like that. And then if you’re a sysadmin and you want to play around with a node, go get a cheap web server and launch a node and start playing around with that.

[00:47:22.610] – Ned
Yeah. Sounds like at least the hardware requirements for some of the blockchains are not that prohibitive. So you could start with one of those that has the lower requirements and just get a feel for it.

[00:47:33.160] – Josh
Yeah, absolutely. If you don’t want to run your own server, but you just want to start curling commands in your CLI to connect to the blockchain, you can do that with our ankr. At ankr, we run something called a public RPC, which is just a public API endpoint that connects to the blockchain. It’s rate limited, so you’re not going to be able to build like a full on production application, but you can find those on our website and connect directly to blockchain and start playing around.

[00:47:59.860] – Ned
Amazing. So if folks want to know more about you, if they want to connect up to you. Do you have a website, a blog or Twitter?

[00:48:06.800] – Josh
Yeah, my Twitter is probably the best place. My DMs are open. I love talking to people in the community. It’s just at Josh Neuroth and I’m pretty responsive on there also I’m on LinkedIn. Will do LinkedIn private messages too. Not everyone likes that but I’m available there as well.

[00:48:22.570] – Ned
Awesome so we’ll include those links in the show. Notes Josh Neuroth thank you so much for being a guest today on day two Cloud and hey virtual high fives to you out there for tuning in. If you have suggestions for future shows, we would love to hear them. You can hit either of us up on Twitter at day two Cloud show or you can fill out the form on my fancy website until next time, just remember Cloud is what happens while IT is making other plans.

More from this show

Day Two Cloud 153: IaC With GPPL Or DSL? IDK

On Day Two Cloud we’ve had a lot of conversations about using infrastructure as code. We’ve looked at solutions like Ansible, Terraform, the AWS CDK, and Pulumi. Which begs the question, which IaC solution should you learn? A Domain Specific Language...

Episode 132