Stellar news delivered weekly:

TL;DR StellarExpert is restored. All services are functioning normally. New features added and more to come.

During the last few weeks, I received more than 60 emails, DMs and comments with the same question, "When StellarExpert will be restored?" I apologize for not answering all of your requests, my focus was fully on coding itself. I'll try to answer each and every one of you in the upcoming week. Many thanks to Stellar community for the understanding and kind words.

And now when everything got back to normal, I'd like to share my thought regarding the future of StellarExpert and a brief retrospective of things that were done during the last month.

It was a tough month for me, it took approximately 250-280 hours to rewrite from the scratch the aggregation backend of the service, completely changing the data collection approach and implementing my own ledger parsing engine. From the very beginning, I underestimated the complexity of the problem, and it was my huge mistake. If only I knew that it would take that long, I would, of course, restored the existing architecture on a new server before starting a fundamental refactoring. Service restoration could take more than a week, but at least we'd have operational StellarExpert two weeks ago.

You see, 98% of the planned functionality was ready by July 18. And then my little personal nightmare began. There was no time to write comprehensive tests for the ingestion engine, so I had to experiment with a ledger, fixing numerous problems one by one. And each time I found a new tiny problem, I had to change a few lines of code and re-ingest the whole ledger starting from the 2015 year. That's 140 000 000 operations, including fee charges and failed transactions (did you know that failed transactions are also included in the ledger archives?).

The structure of SDF archives is well described in the official documentation, but I faced hundreds of the minor problems. For example, transactions in tx sets are not always properly ordered, ledger results not always contain all required information to reconstruct current ledger state etc. Every time I thought that everything is ready for the deployment, I was checking the data and it always was some new problem, either with account balance, account state, or statistics. Of course, I could not roll out the half-working service.

However, it's something that had to be done anyway. Horizon DB (PostgreSQL) can no longer provide reasonable performance for my complex aggregation queries. The database size (including custom tables and materialized view) effectively doubled over the last 5 months. My new engine provides almost unlimited possibilities for the ledger analysis. For instance, I will be able to reconstruct any given asset orderbook or precise account balance at any given time, see the number of transactions discarded by SCP, view protocol upgrades history, view historical price and precise trade volumes for custom assets in fiat currencies (USD, EUR).

And here we come to a pleasant part, new features. Here is a short list of already implemented improvements:

1. Precise historical XLM balance for all accounts. The previous version ignored transaction fees for the historical price chart, so actual account balance was slightly different.
2. Accurate trading volumes for all assets. In the new version, all trades are taken into account for the volume calculation, even trades between custom assets, like BTC-CNY pair. In this particular example, historical BTC and CNY prices are used to calculate estimated trade volume in Stellar lumens.
3. Operations instead of effects in account/asset history. Let's imagine that you have an offer placed on Stellar DEX, and someone is matching your order with, say, path payment operation. You'll see all operation details in your transaction history without a need to search for parent operation by effects it produced.
4. In-place effects exploration. Just click an icon beside an operation in explorer or asset/account history to reveal all ledger modification effects yielded by the operation.
5. Search by operation and transaction ID (not only transaction hash) now available.
6. Built-in data caching for API. Vitally important for overall service performance.

And some more minor fixes and improvements, like grouped path payments, signers weights on the account page etc.

The new engine also gives way for many long anticipated features:

• Precise retrospective trading price and volumes in USD (EUR, GBP etc) for all assets.
• Retrospective chart for XLM in circulation instead of a total number of all lumens in the network.
• Embedded mode (live widgets), a feature requested by a few anchors. Any chart will be available for sharing in a form of dynamically updated widgets (the same way you share YouTube videos or Twitter posts). The same with asset stats (activity indicators, price tickers), you will be able to place a fine looking badge for your beloved asset directly on your website. DM me if you have some specific requirements on this functionality, I'd love to hear thoughts directly from potential end-users.
• A whole lot of new charts and statistics: protocol versioning history, inflations history, cumulative failed transactions counter, asset stability index, ledger closing and capacity retrospective charts, account relations graphs and more.

Since the engine is new, new bugs will definitely come into sight. Please let me know if you spot any problems or data inconsistency with new StellarExpert. Post an issue on the bug tracker or drop me a DM.

I'm going to write a series of short articles on Stellar internals, mainly targeting Stellar developers. Follow me on Twitter, I will publish links there alongside with interesting Stellar metrics and analytical articles for a wider audience.

Thanks for your patience and moral support.

Cheers!