On June 1st, 2024, the Saturn node provider program was concluded.

The goal of this project is to increase the transparency of 🪐 Filecoin Saturn network – the fastest growing, community-run distributed content delivery network (CDN) for Web3.

We use historical data about the network gathered by 🌖 Saturn Moonlet to prepare and publish analytical reports on Saturn Observatory website every month. Raw data that is used to generate the reports is available on IPFS and Filecoin.

Saturn Observatory compliments official tools like Saturn Node Dashboard and Saturn Explorer, aiming to provide better insights into the network state and performance. To highlight a few:

Head over to Saturn Observatory website to see the full report. If you're curious, you can see how the project is made.

Saturn Observatory is neither affiliated with the Filecoin Saturn project nor the Protocol Labs organization. However, it was built during the Open Data Hack powered by Filecoin and received a Grand Prize from Saturn 🏆 and Honorable mention from Lilypad 🏅

To create reports we export historical data about the Saturn network from a running Saturn Moonlet long-term storage into a set of CSV files. We run a bunch of analytical SQL queries against these files and output results into CSVs. Finally, we use the results to generate plots and publish them on the Saturn Observatory website.

The following is a more detailed explanation of each step.

1. Everything starts with exporting Saturn metrics into CSV:

   EXPORTER_PASSWORD=secret ./moonlet/export-csv.sh 2023-09-01 2023-10-01
   

   This shell script uses VictoriaMetrics export API for some metrics and a small Python script for others. This is because we need to apply a rollup function to some metrics (e.g., increase(saturn_node_retrievals_total)).

2. Now we have raw CSV data that we upload and pin to IPFS and store on Filecoin using web3.storage:

   make web3-storage-upload-inputs
   

   Before running this command for the first time, we need to set our web3.storage token: make web3-storage-token.

3. To perform the actual analysis we run a Bacalhau job that uses DuckDB to execute a bunch of SQL queries on the input CSV data that we previously pinned to IPFS:

   make bacalhau-analytics cid=bafybeiarymtc6w32n2ud6w27vbhjqf2seax65l2rrfsfyucxc4gjutugni
   

   We wanted to use Lilypad to run the analysis and made a couple of PRs (one, two, three) with a custom DuckDB module. But, by the time of writing the module is not yet available on Lilypad testnet. This was mostly blocked on Lilypad's team.

4. Once the Bacalhau job finishes we download its results and make sure that these results are adequate (check for outliers, empty values, compare with the previous month, etc.). Then store the results in IPFS and Filecoin using web3.storage:

   make web3-storage-upload-outputs
   

5. Finally, we plug the CID that we get from the previous step into dataUrl function of web/main.js. We commit and push this change to the project's repo and the Saturn Observatory website gets published using GitHub pages.

6. When you open the Saturn Observatory website, CSV data from step 4 is fetched using the Saturn browser client and then plotted using PlotlyJS.

This project is licensed under the MIT License - see the LICENSE file for details.