A bug in top Ethereum software could be catastrophic — but that’s fast changing

A spontaneous public awareness campaign aimed at some of the largest players in Ethereum may be bearing fruit.

Over the past several weeks, the market share of software called Go-ethereum has fallen from roughly 84% to roughly 72%, according to one estimate.

The sudden drop suggests that several companies have made good on recent commitments to wean themselves off the software, better known as Geth.

Activists are now looking to Coinbase, the largest centralised crypto exchange in the US. If Coinbase were to follow suit, the market share of Geth could fall below a critical threshold, staving off a doomsday scenario that some fear would irreparably damage Ethereum.

“To date, Geth is the only execution layer client that has met our technical requirements,” a spokesperson for Coinbase told DL News. “We are actively assessing alternative execution clients, and we will share an update on that process by the end of February.”

The episode has prompted debate over an extreme measure Ethereum developers took to ensure the blockchain remains decentralised.

It also put an awkward spotlight on the power that corporations and Ethereum’s community — its so-called social layer — have over the blockchain, which was designed, in part, to counter centralised power and operate with minimal human intervention.

Go-ethereum and client diversity

For most of its history, Ethereum, like Bitcoin, was secured by a distributed network of powerful computers called miners. Miners would race to guess a number generated by the Ethereum network; the first to guess that number was awarded the privilege of appending the newest block of transactions to the chain and of claiming newly created Ether.

Since an upgrade that went live in September 2022, Ethereum replaced this system, called proof-of-work, with a so-called proof-of-stake model that derives its security from users willing to stake, or lock up, their Ether in exchange for modest annual yield. The greater the stake, the greater the odds of being selected to append the newest block and to claim new Ether.

The proof-of-stake model relies on hardware operators running “client” and “validator” software to keep track of the chain, to let users interact with the Ethereum network, and to append new blocks. Major Geth competitors include Nethermind, Besu, and Erigon.

But Ethereum was envisaged as a world computer without a single point of failure, and its developers took an extreme measure to ensure decentralisation.

To discourage overreliance on a single piece of software, Ethereum was designed such that it would split in two if a client used by more than two-thirds of Ethereum’s validators went offline.

Furthermore, the stakers running the supermajority client would lose their Ether.

“Having many independently developed and maintained clients is vital for the health of a decentralised network,” Ethereum’s community-run website reads.

That diversity could limit the damage to Ethereum in the event a large client goes offline due to a bug or hack.

Nevertheless, Geth has long dominated the market for execution clients, which handle transactions as well as the deployment and execution of smart contracts.

On January 22, 84% of validators were using Geth, according to an estimate using data gathered by pseudonymous developer Sonic.

Geth largely benefitted from first-mover advantage, according to sources who spoke to DL News. It was the first execution client at launch, and for a long time faced little competition.

Additionally, it long held a slight technical advantage over its competitors.

“When it comes to big stakers, this 2% or 3% or even 1% could amount to a lot of money,” Ahmad Bitar, a developer at Nethermind, told DL News.

But those competitors recently closed the gap, according to Nixo, the pseudonymous head of EthStaker, an educational resource for Ethereum stakers.

Some of Ethereum’s most devoted users, developers, and investors have long warned that overreliance on Geth poses an existential threat to the world’s second-largest blockchain.

“The impact when one client is in a 2/3 supermajority is quite catastrophic, and it is also a relatively likely scenario,” Dankrad Feist, a researcher at the Ethereum Foundation, wrote in a 2022 blog post titled, “Ethereum Merge: Run the majority client at your own peril!”

That warning took on new urgency last month, after a pair of competitors, Besu and Nethermind, suffered bugs that temporarily forced them offline.

“When the first one went offline, everybody was like, ‘Oh, a huge, client-wide bug, what an anomaly,’” Nixo told DL News. “And then the second went down, and it starts to look like falling dominoes. And if that third domino is Geth … that is a huge fucking deal if in another month, Geth goes down.”

The fear isn’t unreasonable: Bugs have been found in Geth as well.

In 2020, several major DeFi applications went offline due to a fork triggered by developers at Optimism, an Ethereum-affiliated blockchain. The developers intentionally triggered a Geth bug that had been patched several months earlier, not realising many validators hadn’t updated their software since the fix. Those included validators run by Infura.

In 2021, software developer Guido Vranken found a critical bug in Geth, which was promptly patched.

But Geth bugs have yet to be exploited by bad actors, according to Justin Drake, a researcher at the Ethereum Foundation.

“The good news is that all the issues so far have been disclosed ahead of time, by a white hat,” Drake told DL News, using an industry term for law-abiding, civic-minded hackers.

After the Nethermind bug on January 21, several crypto influencers took to social media to call on organisations to ditch Geth before it too suffered a bug or went offline due to an exploit.

Bailouts and the social layer

If a bug — rare though it may be — were to knock Geth offline, Ethereum could split into two blockchains: one running on Geth, and another running on its competitors, such as Nethermind and Besu.

“Long story short, the [Geth] stakers would lose the vast majority of their Ether,” Drake said.

Some estimates have put the total loss at 20% of all Ether in circulation, worth about $59 billion at Friday’s price.

In the event of an Ethereum fork, people using the Geth chain would have a choice: forfeit billions of dollars worth of Ether, or push to make their chain the canonical version of Ethereum.

“It would be undoubtedly a catastrophe,” Matt Kunke, a research analyst at GSR Markets, told DL News.

With billions of dollars at stake, experts expect a heated debate that could irreparably damage Ethereum’s reputation.

The debate would pit idealism against pragmatism, and Ethereum core developers against billion-dollar entities like Coinbase and Lido, according to Kunke.

“It doesn’t matter what the outcome of that [debate] would be,” Nixo said. “There would be so much equivocation and so much arguing in the Ethereum community that it would absolutely trash the credibility of the blockchain.”

Drake fears the punishment is so severe it almost guarantees that stakers using Geth will get bailed out.

“It’s a non-credible threat, or at least there’s a credible possibility that the social layer would intervene,” he said.

It’s a massive problem for a technology born out of the 2008 financial crisis and the subsequent bailout of big banks.

If the financial system were transparent and its rules inviolable, the fraud that blew up the world economy in 2008 would have been impossible.

Blockchain technology was considered a potential solution in that it could counter centralised power and limit human intervention.

“We should really be minimising the intervention from the social layer,” Drake said. “At least that’s my personal opinion.”

Penalty caps

Geth’s dominance has prompted debate as to whether it was wise to include such a severe punishment for overreliance on a single client.

“That’s like deterring shoplifters by putting Claymore mines in front of the entrance to the store, and they’ll blow [it] up if they catch you shoplifting,” Doug Colkitt, founder of decentralised exchange Ambient Finance, told DL News.

“Okay, yeah, that’ll stop shoplifting, but it might blow up the store, too.”

Developers at the Ethereum Foundation have discussed a change to Ethereum itself, according to Drake: capping the penalties for overreliance on a single client.

Validators must stake Ether in increments of 32. Currently, validators running Geth risk losing all 32 in the event Geth goes offline. Drake supports capping the penalty at one-eighth of a validator’s stake, or 4 Ether.

“Suddenly the threat becomes much more credible, because I don’t think there’s any way the social layer would bailout for just [one-eighth],” he said. “I’ve become a little bit more bullish on this penalties capping idea, as a way to make people understand that the threat is really credible.”

‘Vague promises’

But that would take years, Nixo said. In the meantime, stakers who use Geth can transition to competitors such as Nethermind, Besu, and Erigon.

A social media campaign to convince stakers to change has had some success.

Allnodes, P2P, and Ether.Fi have all transitioned from Geth, the entities announced on X.

Matthew Howells-Barby, Kraken’s vice president of growth, said January 23 the exchange is “currently exploring” using alternative clients.

Staking operator Stakefish said it would transition from complete reliance on Geth and run alternative clients with at least half of its hardware, though it didn’t provide a timeline.

Lido, the largest protocol in decentralised finance, reiterated a previous commitment to reduce its reliance on Geth, which has fallen from 93% to 67% since a major Ethereum upgrade in September 2022.

Major holdouts include Binance and Bitcoin Suisse, which rely entirely on Geth and have not announced plans to change.

There is a major challenge to tracking the situation, however: Data on the proportion of validators running Geth and its competitors is incomplete, self-reported, and unreliable.

Almost half of Ethereum’s 923,000 validators haven’t reported which execution client they are using, according to data gathered by Sonic. As such, the “most important metric” — the share of validators known to be running on Geth alternatives — is a mere 15%, well below the critical 33% threshold that would make a fork impossible.

Nevertheless, their shows a sharp drop in the proportion of validators running Geth. As of Friday, it comprised an estimated 72% of known clients.

Coinbase controls almost 15% of Ethereum validators, according to Rated, a company that gathers Ethereum staking data.

If the exchange were to wean itself off Geth, many believe the latter’s marketshare could drop below the critical two-thirds threshold, making a fork impossible.

“Coinbase has made some vague promises, but I do believe they’re making a solid effort toward it based on some private conversations with their staking team,” Nixo said.

Nethermind has received a grant from the Ethereum Foundation to research methods of automating the collection of client diversity data without revealing specific entities’ client usage.

“The goal is also to collect this data anonymously,” Bitar said. “Because, you know, some people, some institutions might not want people to know what they’re running, because they can be publicly shamed.”

Miguel Prada, a co-founder of Diva Staking, was one of many people who lost money during an earlier controversy that tested the limits of Ethereum’s social layer — the 2016 hack of The DAO, an early, Ethereum-based investment cooperative.

“People were super YOLO into smart contracts at the time, and many ICOs,” he said.

When The DAO was hacked, the fallout was so severe Ethereum developers initiated a fork that created a new chain in which the hack had never happened.

The stolen money was effectively returned to people who had deposited crypto in The DAO, but the affair tempered the you-only-live-once mentality that fueled the mad rush to untested protocols.

“I think this is going to be similar,” he said. “In security, people don’t realise the issues until they face them.”

Aleks Gilbert is DL News’ New York DeFi correspondent. Have a tip? Contact him at aleks@dlnews.com.