Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement new REST API: GET /colors/ #33

Merged
merged 5 commits into from
Apr 22, 2024
Merged

Implement new REST API: GET /colors/ #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ef4a407
Implement new REST API: GET /colors/
Yamaguchi Mar 13, 2024
4f17079
API:GET /colors should return color_id and block_height pairs sorted …
Yamaguchi Mar 17, 2024
3aa81f5
Store block height as big endian for ColorIdRow and ColoredTxHistoryRow
Yamaguchi Mar 25, 2024
7c71bb8
Add stats fields for GET /colors API
Yamaguchi Mar 25, 2024
5b40145
Add descriptions for chain_stats and mempool_stats
Yamaguchi Mar 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,337 changes: 730 additions & 607 deletions Cargo.lock
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ page_size = "0.4.2"
prometheus = "0.11.0"
openassets-tapyrus = "^0.2.4"
rayon = "1.5.0"
rocksdb = { version = "0.15.0", optional = true }
rocksdb-oldcpu = { version = "0.12.4", optional = true, package = "rocksdb" }
rocksdb = { version = "^0.22.0", optional = true }
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

最新のmacOS環境でビルドが通らなかったため、アップデート

rocksdb-oldcpu = { version = "^0.22.0", optional = true, package = "rocksdb" }
rust-crypto = "0.2"
serde = { version = "1.0.118", features = ["derive"] }
serde_derive = "1.0.118"
Expand Down
17 changes: 17 additions & 0 deletions doc/API.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,23 @@ For example: `{ "1": 87.882, "2": 87.882, "3": 87.882, "4": 87.882, "5": 81.129,

## Tokens

### `GET /colors[/:last_seen_color_id]`

Returns a list of the 25 objects with:
- `color_id`
- `block_height`
- `chain_stats` and `mempool_stats`, each contains an object with:
- `tx_count`
- `issued_tx_count`
- `transferred_tx_count`
- `burned_tx_count`
- `issued_sum`
- `transferred_sum`
- `burned_sum`

if `last_seen_color_id` specified, returns a list of the next 25 objects after specified `last_seen_color_id`


### `GET /color/:color_id`

Get information about a colored coin.
Expand Down
3 changes: 3 additions & 0 deletions doc/schema.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@ Each spending input (except the coinbase) results in the following new rows (`S`

* `"S{funding-txid:vout}{spending-txid:vin}" → ""`

If transaction output include colored coins, it results in the following new row:
* `"B{block-height}c{color-id}" → ""` (block_height is latest block height which colored coin used)

colored coin's issuances results in the following new rows (`I` is for issuing):

* `"C{color-id}{issuance-height}I{issuing-txid}{value}" → ""`
Expand Down
101 changes: 71 additions & 30 deletions src/new_index/color.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ use crate::new_index::db::DBRow;
use crate::new_index::schema::FullHash;
use crate::util::{full_hash, Bytes};

use super::schema::ColorIdRow;

#[derive(Serialize, Deserialize, Debug, PartialEq)]
pub struct ColoredTxHistoryKey {
pub color_id: ColorIdentifier,
Expand Down Expand Up @@ -38,11 +40,11 @@ impl ColoredTxHistoryRow {
}

pub fn prefix_height(color_id: &ColorIdentifier, height: u32) -> Bytes {
bincode::serialize(&(b'C', &serialize_color_id(color_id), height)).unwrap()
bincode::serialize(&(b'C', &serialize_color_id(color_id), height.to_be_bytes())).unwrap()
}

pub fn prefix_end(color_id: &ColorIdentifier) -> Bytes {
bincode::serialize(&(b'C', &serialize_color_id(color_id), std::u32::MAX)).unwrap()
bincode::serialize(&(b'C', &serialize_color_id(color_id), std::u32::MAX.to_be_bytes())).unwrap()
}

pub fn get_txid(&self) -> Txid {
Expand All @@ -54,7 +56,7 @@ impl ColoredTxHistoryRow {
key: bincode::serialize(&(
b'C',
&serialize_color_id(&self.key.color_id),
self.key.confirmed_height,
self.key.confirmed_height.to_be_bytes(),
self.key.txinfo,
))
.unwrap(),
Expand All @@ -67,13 +69,13 @@ impl ColoredTxHistoryRow {
u8,
u8,
[u8; 32],
u32,
[u8; 4],
ColoredTxHistoryInfo,
) = bincode::deserialize(&row.key).unwrap();
ColoredTxHistoryRow {
key: ColoredTxHistoryKey {
color_id: deserialize_color_id(token_type, payload),
confirmed_height,
confirmed_height: u32::from_be_bytes(confirmed_height),
txinfo,
},
}
Expand Down Expand Up @@ -173,7 +175,7 @@ impl ColoredStatsCacheRow {
}
}

#[derive(Serialize, Deserialize, Debug)]
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct ColoredStats {
pub color_id: ColorIdentifier,
pub tx_count: usize,
Expand Down Expand Up @@ -208,11 +210,10 @@ pub fn index_confirmed_colored_tx(
) {
let history = colored_tx_history(tx, previous_txos_map);

rows.extend(
history.into_iter().map(|(color_id, info)| {
colored_history_row(&color_id, confirmed_height, info).into_row()
}),
);
history.into_iter().for_each(|(color_id, info)| {
rows.push(colored_history_row(&color_id, confirmed_height, info).into_row());
rows.push(ColorIdRow::new(confirmed_height, &color_id).into_row());
});
}

fn colored_history_row(
Expand Down Expand Up @@ -526,55 +527,95 @@ mod tests {
let mut rows = vec![];
index_confirmed_colored_tx(&tx, 10, &previous_txos_map, &mut rows);

assert_eq!(rows.len(), 4);
assert_eq!(rows.len(), 8);

rows.sort_by(|a, b| a.key.cmp(&b.key));
let row0 = rows.get(0).unwrap();
let hex = hex::encode::<Vec<u8>>(row0.key.iter().cloned().collect());
// field | size | value |
//---------------------|------|-----------------------------------------------------------------------|
// prefix | 1 | 'B'(0x42) |
// height | 4 | 10(0x0000000a) (big endian) |
// code | 1 | 'c'(0x63) |
// color_id | 33 | c12dceb0cedd7c372c838fea8d46ae863a3c47b2ad0fb950e90ac9d531583ad35e |
assert_eq!(hex, "420000000a63c12dceb0cedd7c372c838fea8d46ae863a3c47b2ad0fb950e90ac9d531583ad35e");

let row1 = rows.get(1).unwrap();
let hex = hex::encode::<Vec<u8>>(row1.key.iter().cloned().collect());
// field | size | value |
//---------------------|------|-----------------------------------------------------------------------|
// prefix | 1 | 'B'(0x42) |
// height | 4 | 10(0x0000000a) (big endian) |
// code | 1 | 'c'(0x63) |
// color_id | 33 | c12dceb0cedd7c372c838fea8d46ae863a3c47b2ad0fb950e90ac9d531583ad35e |
assert_eq!(hex, "420000000a63c12dceb0cedd7c372c838fea8d46ae863a3c47b2ad0fb950e90ac9d531583ad35e");

let row2 = rows.get(2).unwrap();
let hex = hex::encode::<Vec<u8>>(row2.key.iter().cloned().collect());
// field | size | value |
//---------------------|------|-----------------------------------------------------------------------|
// prefix | 1 | 'B'(0x42) |
// height | 4 | 10(0x0000000a) (big endian) |
// code | 1 | 'c'(0x63) |
// color_id | 33 | c271c99cc3bc21757feed5b712744ebb0f770d5c41d99189f9457495747bf11050 |
assert_eq!(hex, "420000000a63c271c99cc3bc21757feed5b712744ebb0f770d5c41d99189f9457495747bf11050");

let row3 = rows.get(3).unwrap();
let hex = hex::encode::<Vec<u8>>(row3.key.iter().cloned().collect());
// field | size | value |
//---------------------|------|-----------------------------------------------------------------------|
// prefix | 1 | 'B'(0x42) |
// height | 4 | 10(0x0000000a) (big endian) |
// code | 1 | 'c'(0x63) |
// color_id | 33 | c271c99cc3bc21757feed5b712744ebb0f770d5c41d99189f9457495747bf11050 |
assert_eq!(hex, "420000000a63c271c99cc3bc21757feed5b712744ebb0f770d5c41d99189f9457495747bf11050");

let row4 = rows.get(4).unwrap();
let hex = hex::encode::<Vec<u8>>(row4.key.iter().cloned().collect());
// field | size | value |
//---------------------|------|-----------------------------------------------------------------------|
// prefix | 1 | 'C'(0x43) |
// color_id | 33 | c12dceb0cedd7c372c838fea8d46ae863a3c47b2ad0fb950e90ac9d531583ad35e |
// height | 4 | 10(0x0a000000) |
// height | 4 | 10(0x0000000a) (big endian) |
// Issue/Transfer/Burn | 4 | 'Transfer'(0x010000000) |
// txid | 32 | 59abe954f5636c86484e5e2817d29b915e7f9a9f0294e87c438fd060694a8b1c |
// value | 8 | 100(0x64) |
assert_eq!(hex, "43c12dceb0cedd7c372c838fea8d46ae863a3c47b2ad0fb950e90ac9d531583ad35e0a0000000100000059abe954f5636c86484e5e2817d29b915e7f9a9f0294e87c438fd060694a8b1c6400000000000000");
// value | 8 | 100(0x6400000000000000) |
assert_eq!(hex, "43c12dceb0cedd7c372c838fea8d46ae863a3c47b2ad0fb950e90ac9d531583ad35e0000000a0100000059abe954f5636c86484e5e2817d29b915e7f9a9f0294e87c438fd060694a8b1c6400000000000000");

let row1 = rows.get(1).unwrap();
let hex = hex::encode::<Vec<u8>>(row1.key.iter().cloned().collect());
let row5 = rows.get(5).unwrap();
let hex = hex::encode::<Vec<u8>>(row5.key.iter().cloned().collect());
// field | size | value |
//---------------------|------|-----------------------------------------------------------------------|
// prefix | 1 | 'C'(0x43) |
// color_id | 33 | c12dceb0cedd7c372c838fea8d46ae863a3c47b2ad0fb950e90ac9d531583ad35e |
// height | 4 | 10(0x0a000000) |
// height | 4 | 10(0x0000000a) (big endian) |
// Issue/Transfer/Burn | 4 | 'Burn'(0x02000000) |
// txid | 32 | 59abe954f5636c86484e5e2817d29b915e7f9a9f0294e87c438fd060694a8b1c |
// value | 8 | 100(0x64) |
assert_eq!(hex, "43c12dceb0cedd7c372c838fea8d46ae863a3c47b2ad0fb950e90ac9d531583ad35e0a0000000200000059abe954f5636c86484e5e2817d29b915e7f9a9f0294e87c438fd060694a8b1c6400000000000000");
// value | 8 | 100(0x6400000000000000) |
assert_eq!(hex, "43c12dceb0cedd7c372c838fea8d46ae863a3c47b2ad0fb950e90ac9d531583ad35e0000000a0200000059abe954f5636c86484e5e2817d29b915e7f9a9f0294e87c438fd060694a8b1c6400000000000000");

let row2 = rows.get(2).unwrap();
let hex = hex::encode::<Vec<u8>>(row2.key.iter().cloned().collect());
let row6 = rows.get(6).unwrap();
let hex = hex::encode::<Vec<u8>>(row6.key.iter().cloned().collect());
// field | size | value |
//---------------------|------|-----------------------------------------------------------------------|
// prefix | 1 | 'C'(0x43) |
// color_id | 33 | c271c99cc3bc21757feed5b712744ebb0f770d5c41d99189f9457495747bf11050 |
// height | 4 | 10(0x0a000000) |
// height | 4 | 10(0x0000000a) (big endian) |
// Issue/Transfer/Burn | 4 | 'Issue'(0x00000000) |
// txid | 32 | 59abe954f5636c86484e5e2817d29b915e7f9a9f0294e87c438fd060694a8b1c |
// value | 8 | 100(0x64) |
assert_eq!(hex, "43c271c99cc3bc21757feed5b712744ebb0f770d5c41d99189f9457495747bf110500a0000000000000059abe954f5636c86484e5e2817d29b915e7f9a9f0294e87c438fd060694a8b1c6400000000000000");
// value | 8 | 100(0x6400000000000000) |
assert_eq!(hex, "43c271c99cc3bc21757feed5b712744ebb0f770d5c41d99189f9457495747bf110500000000a0000000059abe954f5636c86484e5e2817d29b915e7f9a9f0294e87c438fd060694a8b1c6400000000000000");

let row3 = rows.get(3).unwrap();
let hex = hex::encode::<Vec<u8>>(row3.key.iter().cloned().collect());
let row7 = rows.get(7).unwrap();
let hex = hex::encode::<Vec<u8>>(row7.key.iter().cloned().collect());
// field | size | value |
//---------------------|------|-----------------------------------------------------------------------|
// prefix | 1 | 'C'(0x43) |
// color_id | 33 | c271c99cc3bc21757feed5b712744ebb0f770d5c41d99189f9457495747bf11050 |
// height | 4 | 10(0x0a000000) |
// height | 4 | 10(0x0000000a) (big endian) |
// Issue/Transfer/Burn | 4 | 'Transfer'(0x01000000) |
// txid | 32 | 59abe954f5636c86484e5e2817d29b915e7f9a9f0294e87c438fd060694a8b1c |
// value | 8 | 200(0xc8) |
assert_eq!(hex, "43c271c99cc3bc21757feed5b712744ebb0f770d5c41d99189f9457495747bf110500a0000000100000059abe954f5636c86484e5e2817d29b915e7f9a9f0294e87c438fd060694a8b1cc800000000000000");
// value | 8 | 200(0xc800000000000000) |
assert_eq!(hex, "43c271c99cc3bc21757feed5b712744ebb0f770d5c41d99189f9457495747bf110500000000a0100000059abe954f5636c86484e5e2817d29b915e7f9a9f0294e87c438fd060694a8b1cc800000000000000");
}
}
2 changes: 1 addition & 1 deletion src/new_index/db.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ impl<'a> Iterator for ScanIterator<'a> {
if self.done {
return None;
}
let (key, value) = self.iter.next()?;
let (key, value) = self.iter.next()?.ok()?;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

最新のmacOS環境(rust: 1.76.0)でビルドが通らなかったため、修正

if !key.starts_with(&self.prefix) {
self.done = true;
return None;
Expand Down
17 changes: 17 additions & 0 deletions src/new_index/query.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,23 @@ impl Query {
Ok(map)
}

pub fn get_colors(&self, last_seen_color_id: Option<ColorIdentifier>, limit: usize) -> Vec<(ColorIdentifier, u32)> {
let (block_height, color_id) = if let Some(color_id) = last_seen_color_id {
match self.chain.get_height_by_color_id(&color_id) {
Some(height) => (height, Some(color_id)),
_ => (self.daemon.getblockchaininfo().ok().unwrap().blocks, None),
}
} else {
(self.daemon.getblockchaininfo().ok().unwrap().blocks, None)
};

let colors = self
.chain
.get_colors(block_height, &color_id, limit)
.expect("failed to get colors");
colors
}

pub fn get_colored_stats(&self, color_id: &ColorIdentifier) -> (ColoredStats, ColoredStats) {
(
self.chain
Expand Down
82 changes: 82 additions & 0 deletions src/new_index/schema.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ use crate::util::{
use crate::new_index::db::{DBFlush, DBRow, ReverseScanIterator, ScanIterator, DB};
use crate::new_index::fetch::{start_fetcher, BlockEntry, FetchFrom};

use super::color::{deserialize_color_id, serialize_color_id};

const MIN_HISTORY_ITEMS_TO_CACHE: usize = 100;

pub struct Store {
Expand Down Expand Up @@ -674,6 +676,29 @@ impl ChainQuery {
update_stats(init_stats, &histories)
}

pub fn get_colors(&self, block_height: u32, last_seen_color_id: &Option<ColorIdentifier>, limit: usize) -> Result<Vec<(ColorIdentifier, u32)>> {
let colors = self.store.history_db().iter_scan_reverse(
&ColorIdRow::prefix(),
&ColorIdRow::filter_end(block_height, last_seen_color_id)
).filter_map(|row|{
let row = ColorIdRow::from_row(row);
let result = self.get_height_by_color_id(&row.key.color_id);
// Ignore color_id, block_height pair if the block_height is not the latest
match result {
Some(block_height) if (block_height <= row.key.block_height) => Some((row.key.color_id, row.key.block_height)),
_ => None,
}
})
.skip(1) // Ignore the first element, which is the last_seen_color_id itself
.take(limit)
.collect::<Vec<_>>();
Ok(colors)
}

pub fn get_height_by_color_id(&self, color_id: &ColorIdentifier) -> Option<u32> {
self.colored_history_iter_scan_reverse(color_id).map(|c| ColoredTxHistoryRow::from_row(c).key.confirmed_height).next()
}

pub fn get_colored_stats(&self, color_id: &ColorIdentifier) -> Result<ColoredStats> {
let cache: Option<(ColoredStats, usize)> = self
.store
Expand Down Expand Up @@ -1330,6 +1355,63 @@ impl TxOutRow {
}
}

#[derive(Serialize, Deserialize)]
pub struct ColorIdKey {
block_height: u32,// MUST be serialized as big-endian.
color_id: ColorIdentifier,
}
// keys is "B{block-height}c{color-id}" (block_height is latest block height which colored coin used)
// value is ""
pub struct ColorIdRow {
key: ColorIdKey,
}

impl ColorIdRow {
pub fn new(block_height: u32, color_id: &ColorIdentifier) -> ColorIdRow {
ColorIdRow {
key: ColorIdKey {
block_height: block_height,
color_id: color_id.clone()
}
}
}

fn prefix() -> Bytes {
bincode::serialize(&(b'B'))
.unwrap()
}

fn filter_end(block_height: u32, color_id: &Option<ColorIdentifier>) -> Bytes {
let filter = if let Some(color_id) = color_id {
bincode::serialize(&(b'B', block_height.to_be_bytes(), b'c', &serialize_color_id(&color_id)))
.unwrap()
} else {
bincode::serialize(&(b'B', (block_height + 1).to_be_bytes(), b'c'))
.unwrap()
};
filter
}

pub fn into_row(self) -> DBRow {
DBRow {
key: bincode::serialize(&(b'B', self.key.block_height.to_be_bytes(), b'c', &serialize_color_id(&self.key.color_id))).unwrap(),
value: vec![],
}
}

pub fn from_row(row: DBRow) -> Self {
let (_prefix, block_height, _prefix2, token_type, payload): (u8, [u8; 4] , u8, u8, [u8; 32]) =
bincode::deserialize(&row.key).expect("failed to deserialize ColorIdRow");
ColorIdRow {
key: ColorIdKey {
block_height: u32::from_be_bytes(block_height),
color_id: deserialize_color_id(token_type, payload),
}
}
}
}


#[derive(Serialize, Deserialize)]
struct BlockKey {
code: u8,
Expand Down
Loading
Loading