README.md

NineChronicles DataProvider

• NineChronicles.DataProvider is an off-chain service that stores NineChronicles game action data to a database mainly for game analysis.
• Currently, this service only supports MySQL database.

Table of Contents

• Pre-requisite
• Run
• Development Guide
• Current Table Descriptions
• Migrating Past Chain Data to MySQL Database

Pre-requisite

• MySQL and Entity Framework Core tools should be installed in the local machine.

Run

• Before running the program, please refer to the option values in the latest official 9c-launcher-config.json and fill out the variables in appsettings.json.
• In appsettings.json, AppProtocolVersionToken, StorePath, PeerStrings, MySqlConnectionString properties MUST be filled to run the program.
• To setup the NineChronicles mainnet blockchain store to use in the StorePath, download and extract the latest mainnet snapshot to a desired location.

$ dotnet run --project ./NineChronicles.DataProvider.Executable/ -- 

Development Guide

• This section lays out the steps in how to log a new action in the database.
• The TransferAsset action is used as an example in this guide.

1. Setup Database

• To setup the database, navigate to NineChronicles.DataProvider/NineChronicles.DataProvider.Executable directory on your terminal and run the following migration command.

dotnet ef database update -- [Connection String]
- Connection String example: "server=localhost;database=data_provider;port=3306;uid=root;pwd=root;"

2. Create Model

• In NineChronicles.DataProvider/NineChronicles.DataProvider/Store/Models directory, create a model file called TransferAssetModel.cs.
• In general, TxId, BlockIndex, Date, and Timestamp are useful to add as default properties in a model because these values will help with query speed when table size increases.

namespace NineChronicles.DataProvider.Store.Models
{
    using System;
    using System.ComponentModel.DataAnnotations;

    public class TransferAssetModel
    {
        [Key]
        public string? TxId { get; set; }

        public long BlockIndex { get; set; }

        public string? Sender { get; set; }

        public string? Recipient { get; set; }

        public decimal Amount { get; set; }

        public DateOnly Date { get; set; }

        public DateTimeOffset TimeStamp { get; set; }
    }
}

• In NineChronicles.DataProvider/NineChronicles.DataProvider/Store/NineChroniclesContext.cs, add a DbSet called TransferAssets and its description for reference.

// Table for storing TransferAsset actions
public DbSet<TransferAssetModel> TransferAssets => Set<TransferAssetModel>();

2. Create Store Method

• In NineChronicles.DataProvider/NineChronicles.DataProvider/Store/MySqlStore.cs, add a following method that stores the TransferAsset data into MySQL.

public void StoreTransferAsset(TransferAssetModel model)
{
    using NineChroniclesContext ctx = _dbContextFactory.CreateDbContext();
    TransferAssetModel? prevModel =
        ctx.TransferAssets.FirstOrDefault(r => r.TxId == model.TxId);
    if (prevModel is null)
    {
        ctx.TransferAssets.Add(model);
    }
    else
    {
        prevModel.BlockIndex = model.BlockIndex;
        prevModel.Sender = model.Sender;
        prevModel.Recipient = model.Recipient;
        prevModel.Amount = model..Amount;
        prevModel.Date = model.Date;
        prevModel.TimeStamp = model.TimeStamp;
        ctx.TransferAssets.Update(prevModel);
    }

    ctx.SaveChanges();
}

3. (Optional) Add Data Getter

In some cases, you need to handle state to get data to make model.
To do this easily, you can make your own data getter inside NineChronicles.DataProvider/DataRendering/.

4. Render & Store Action Data

• In NineChronicles.DataProvider/NineChronicles.DataProvider/RenderSubscriber.cs, add a following render code

_actionRenderer.EveryRender<TransferAsset>()
    .Subscribe(ev =>
    {
        try
        {
            if (ev.Exception is null && ev.Action is { } transferAsset)
            {
                var model = new TransferAssetModel()
                {
                    TxId = transferAsset.TxId,
                    BlockIndex = transferAsset.BlockIndex,
                    Sender = transferAsset.Sender,
                    Recipient = transferAsset.Recipient,
                    Amount = Convert.ToDecimal(transferAsset.Amount.GetQuantityString()),
                    Date = DateOnly.FromDateTime(_blockTimeOffset.DateTime),
                    TimeStamp = _blockTimeOffset,
                };
                MySqlStore.StoreTransferAsset(model);
            }
        }
        catch (Exception e)
        {
            Console.WriteLine(e);
        }

5. Add Database Migration

• Navigate to NineChronicles.DataProvider/NineChronicles.DataProvider.Executable directory on your terminal and run the following migration command

dotnet ef migrations add AddTransferAsset -- [Connection String]
- Connection String example: "server=localhost;database=data_provider;port=3306;uid=root;pwd=root;"

Current Table Descriptions

• Tables that NineChronicles.DataProvider stores data into are listed in NineChroniclesContext.cs.
• Please refer to each DbSet's comment in NineChroniclesContext.cs for table descriptions.

Migrating Past Chain Data to MySQL Database

Migrate All Action Data

• This command migrates all action data based on DataRendering to the designated MySQL database.
• Options such as offset and limit are provided to specify which block data to migrate.
• IMPORTANT) This migration tool requires you to have the necessary state data in the blocks you want to migrate (If your chain store lacks the state data, this tool will not work).

Usage: NineChronicles.DataProvider.Executable mysql-migration [--store-path <String>] [--mysql-server <String>] [--mysql-port <UInt32>] [--mysql-username <String>] [--mysql-password <String>] [--mysql-database <String>] [--offset <I
nt32>] [--limit <Int32>] [--help]

Migrate action data in rocksdb store to mysql db.

Options:
  -o, --store-path <String>    Rocksdb path to migrate. (Required)
  --mysql-server <String>      A hostname of MySQL server. (Required)
  --mysql-port <UInt32>        A port of MySQL server. (Required)
  --mysql-username <String>    The name of MySQL user. (Required)
  --mysql-password <String>    The password of MySQL user. (Required)
  --mysql-database <String>    The name of MySQL database to use. (Required)
  --offset <Int32>             offset of block index (no entry will migrate from the genesis block).
  --limit <Int32>              limit of block count (no entry will migrate to the chain tip).
  -h, --help                   Show help message

Migrate Battle Arena Ranking Data

• This command calculates the battle arena ranking of all participants at a specific block index (migration-block-index) and inserts the data to a designated mysql database.

Usage: NineChronicles.DataProvider.Executable battle-arena-ranking-migration [--store-path <String>] [--mysql-server <String>] [--mysql-port <UInt32>] [--mysql-username <String>] [--mysql-password <String>] [--mysql-database <Stri
ng>] [--migration-block-index <Int64>] [--help]

Migrate battle arena ranking data at a specific block index to a mysql database.

Options:
  -o, --store-path <String>          RocksDB store path to migrate. (Required)
  --mysql-server <String>            Hostname of MySQL server. (Required)
  --mysql-port <UInt32>              Port of MySQL server. (Required)
  --mysql-username <String>          Name of MySQL user. (Required)
  --mysql-password <String>          Password of MySQL user. (Required)
  --mysql-database <String>          Name of MySQL database. (Required)
  --migration-block-index <Int64>    Block index to migrate.
  -h, --help                         Show help message

Migrate User Staking Data

• This command calculates the NCG staking amount of all participants at a specific block index (migration-block-index) and inserts the data to a designated mysql database.
• Currently, the slack-token and slack-channel options for sending the data in csv format are required, however, these will be changed to optional in the near future.

Usage: NineChronicles.DataProvider.Executable user-staking-migration [--store-path <String>] [--mysql-server <String>] [--mysql-port <UInt32>] [--mysql-username <String>] [--mysql-password <String>] [--mysql-database <String>] [--
slack-token <String>] [--slack-channel <String>] [--migration-block-index <Int64>] [--help]

Migrate staking amounts of users at a specific block index to a mysql database.

Options:
  -o, --store-path <String>          Rocksdb store path to migrate. (Required)
  --mysql-server <String>            Hostname of MySQL server. (Required)
  --mysql-port <UInt32>              Port of MySQL server. (Required)
  --mysql-username <String>          Name of MySQL user. (Required)
  --mysql-password <String>          Password of MySQL user. (Required)
  --mysql-database <String>          Name of MySQL database to use. (Required)
  --slack-token <String>             slack token to send the migration data. (Required)
  --slack-channel <String>           slack channel that receives the migration data. (Required)
  --migration-block-index <Int64>    Block index to migrate.
  -h, --help                         Show help message