DeFi Infrastructure Providers Find It Hard to Keep up with Surging User Demand
The recent dramatic rise in the demand for decentralized finance (DeFi) apps, courtesy of the “yield farming” frenzy, proved too much to handle for The Graph – a blockchain infrastructure firm that processes the complex DeFi data transactions.
Demand too High to Handle
DeFi is all the rage in the crypto industry today.
Blockchain transaction processing firm called The Graph recently released a report titled “DeFi is Eating the World: Outage Post-Mortem” that sheds light on the unprecedented user demand for DeFi applications that caused an outage on company’s database on June 24.
We’ve been experiencing truly insane load on our hosted service and are seeing high levels of failed requests. We’ve maxed out the largest DB size on Google cloud. Yesterday we had 50% head room.
Sincere apologies to the teams affected. Our engineers are hard at work on a fix.
— The Graph (@graphprotocol) June 24, 2020
The outage caused due to excessive DeFi transactions load on The Graph’s system subsequently caused several frontend issues for popular DeFi applications, including lending protocols such as Aave and Balancer.
We are experiencing shortages in our front end service, and we are working to bring back the service as soon as possible. The protocol remains fully accessible on-chain and using alternative services as @1inchExchange or @paraswap
— Aave (@AaveAave) June 24, 2020
The magnitude of the pressure caused by DeFi users was so huge that it caused an eleven hours-long outage for The Graph, starting from June 24, 12:00 pm PST until 11:10 am PST. This outage was followed by another, although shorter, outage that lasted for about an hour.
In its report, The Graph state that over the past two weeks, the firm has witnessed query volume grow from 25m queries/day to about 45m queries/day. The Graph posits that the exponential rise in the number of queries could be attributed to the recent growth in the adoption of DeFi apps; including product launches, liquidity mining programs, and rise in the price of crypto-assets.
The Graph added that on June 23, their database CPU maxed out at 100 percent capacity, causing requests to fail. Notably, according to the data processing firm, under normal circumstances, its CPU runs with 50 percent unused capacity.
Focus on Decentralizing the Infrastructure
To ensure that such an unfortunate chain of events does not unfold again, The Graph announced that they are working toward making their infrastructure completely decentralized.
In the past couple of days, The Graph has taken several steps aimed at preventing expensive queries from running on the database.
The firm identified a list of improvements that it seeks to implement and roll out over the next few days. These improvements include introducing more sophisticated query complexity costing, optimizing query processing with better caching, setting more aggressive triggers and alerts, and selective rate-limiting, among others.
The team is also looking to hire an additional engineer in Europe or Asia to give itself greater geographical coverage.
Although the outage might have been irritating for DeFi enthusiasts looking to make money by harvesting their yields or by liquidating their time-sensitive orders, it has, at least temporarily, proved that users are willing to experiment with novel finance protocols that value decentralization. Whether such protocols are able to scale appropriately to match the rising user demand remains to be seen.
