Bitfinex “experiencing data corruption issues” and Technical Difficulties
Bitfinex, one of the world’s largest bitcoin exchanges by trading volume, has been experiencing technical difficulties and data corruption issues on their bitcoin exchange.
The Bitfinex trading engine was offline several times before the exchange finally reported the problem on Twitter, at 12:28 AM – 25 Aug 2015. The Bitfinex team told its users: “We are experiencing data corruption issues and will be offline until corrected. Sorry for the inconvenience.”
The exchange was restored and the trades began to be processed at around 4:00 AM EST. Over the course of a minute, over 730 BTC were traded on the platform.
During the four hours of the exchange’s discontinuation, the Bitfinex team stated, “No hack has taken place, all of these issues have been related to trade processing,”
Bitfinex Flash Crash
Prior to its problems with the database, Bitfinex cited issues with its post-trade processing. Its users were provided with a 15-minute window to cancel their orders. The major cause of this problem was the technical glitches of the integration of AlphaPoint software.
“Earlier today, at 5:27 UTC, we encountered an issue with post-trade processing, upon which we decided to halt trading to ensure consistency in the order book,” the Bitfinex team stated.
Serious Lag Issues
Over the last few months, Bitfinex users have also experienced serious lag issues, some losing large chunks of their bitcoins in marginal trades. Bitfinex has restored the BTCs to the users that reported them missing. However, users were frustrated at the inefficiency of the exchange’s AlphaPoint integration.
One user stated, “I had a few hundred trades executed in my account within a couple of minutes. At one point I was long over 2000 BTC, a minute later short 707 BTC. Currently long 808BTC. My account started with only 6.5BTC and is currently reporting losses of $15,000.”
A few weeks after the complaints and reports on missing bitcoins in marginal trades, the Bitfinex team finally provided an answer and solution to the problems.
Director of Community & Product Development Zane Tackett explained on Reddit:
“1) Our margin check system has 2 filters in place to check the collateral of a position for ever user placing an order. The first takes place when the trader places an order and the second when the order is executed.
2) Yesterday night, while working on the AlphaPoint integration the second filter was lifted in order to expedite the process. Our dev team believed that the first filter was sufficient in preventing users from over-leveraging their positions.
3) At around 9:45 PM GMT while resuming trading on our existing matching engine we had a spike in volume that resulted in some lag. This lag induced some panic among some traders causing them to place multiple orders (hit submit/cancel multiple times). The issue lies in the fact that the second filter did not catch these orders and led to the issues that were experienced by some users.
4) Upon discovering this issue the dev team immediately activated both filters remedying the problem at hand. We are currently in the process of examining what went wrong with the first filter to allow this problem to happen. Since we now know that we cannot rely solely on the first filter–for any period of time–we are moving forward with both filters activated at all times. Thus, we are 100% sure that this problem has been fixed.”