feat: tokenlists resolver

[mktcode]

Closes: #24

Tests

The (e2e) tests check previously generated fingerprints of the returned images. Of courtse that will make tests fail, when one of these images changes. The images we test should be chosen accordingly. Correct invalidation of changed/updated images should probably be a test of its own.

The tests could be far more comprehensive but I didn't want to cause too much refactoring. The fingerprinting should help to test more things already and reduce manual review.

Implementation

The tokenlists data is aggregated into a map, containing all found image URLs for a <chainId>-<address> pair.

stamp/src/helpers/tokenlists.ts

Line 9 in 0ee529e

type AggregatedTokenList = Map<string, string[]>;

The URLs are sorted based on keywords they contain (in their path).

stamp/src/helpers/tokenlists.ts

Lines 85 to 126 in 0ee529e

	const sizeKeywords = [
	'xxl',
	'xl',
	'large',
	'lg',
	'big',
	'medium',
	'md',
	'small',
	'sm',
	'thumb',
	'icon',
	'xs',
	'xxs'
	];
	/**
	* Sorts URIs by the size keyword in the URI. The order in the array above is the order of the sort.
	*/
	export function sortByKeywordMatch(a: string, b: string) {
	try {
	const aPath = new URL(a).pathname;
	const bPath = new URL(b).pathname;
	const keywordRegex = new RegExp(`\\b(${sizeKeywords.join('|')})\\b`);
	const aMatch = aPath.match(keywordRegex);
	const bMatch = bPath.match(keywordRegex);
	if (aMatch && bMatch) {
	return sizeKeywords.indexOf(aMatch[1]) - sizeKeywords.indexOf(bMatch[1]);
	} else if (aMatch) {
	return -1;
	} else if (bMatch) {
	return 1;
	} else {
	return a.localeCompare(b);
	}
	} catch (e) {
	return 0;
	}
	}

Before that, known URL patterns related to image size, are replaced. Like for coingecko, we know we can replace small and thumb with large to get a larger image.

stamp/src/helpers/tokenlists.ts

Lines 71 to 85 in c0f8a9c

	const REPLACE_SIZE_REGEXES: { pattern: RegExp; replacement: string }[] = [
	{
	pattern: /assets.coingecko.com\/coins\/images\/(\d+)\/(thumb|small)/,
	replacement: 'assets.coingecko.com/coins/images/$1/large'
	}
	];
	// TODO: Since we do the sorting by keyword match, we should probably not change the URLs in place but add the "large" version to the list of URIs.
	// Might be better for fallback mechanisms, instead of overwriting the version that was fetched.
	export function replaceURIPatterns(uri: string) {
	for (const { pattern, replacement } of REPLACE_SIZE_REGEXES) {
	uri = uri.replace(pattern, replacement);
	}
	return uri;
	}

When the resolver runs for the first time, it waits for the tokenlists data to be fetched and aggregated.

stamp/src/resolvers/tokenlists.ts

Lines 6 to 10 in c0f8a9c

	export default async function resolve(address: string, chainId: string) {
	try {
	await updateExpiredAggregatedTokenList();
	const url = findImageUrl(address, chainId);
	const image = await fetchHttpImage(url);

stamp/src/helpers/tokenlists.ts

Lines 135 to 140 in 51bff9d

	export async function updateExpiredAggregatedTokenList() {
	if (!isExpired() || isUpdating) {
	return;
	}
	isUpdating = true;

When the cache expires, an update is triggered by the first incoming request. This request waits for the update to complete. Other concurrent requests will use the existing list, which may be empty after a server restart. While they could wait for the first request to finish updating, I'm still struggling to understand the rationale behind this approach.

I'm curious about the reasons for not running updates at regular intervals, starting with the server initialization. Additionally, I'm wondering why it's necessary for the resolver to determine the TTL starting timestamp.
Personally, I might consider a different approach, such as running a daily or hourly GitHub workflow to update the data in the Redis cache. This way, the application wouldn't need to handle these updates directly. The dependabot-like approach I suggested earlier is not even necessary.

Btw, I didn't have the time to look into sentry integration yet.

@bonustrack I'm working on another project now, so, like I said, I won't have a lot of time anymore to work on this one. But I will check back in a couple of weeks/months. Let me know when release day get's closer.

[mktcode]

@bonustrack I moved everything to the resolver and made sure the fetches run in parallel.

The little test I wrote still passes but you didn't comment on the extra headers yet. This needs improvement but I think it is a good way of testing more things. We can decide per environment what headers are being sent. Some headers can be exclusively for testing.

[bonustrack]

Shouldn't the change on cache be in another PR? I don't see how it's related to tokenlists

[mktcode]

Oh sry, I didn't mean "missing" but just this EIP<some number> prefix.... didn't test any of these variations.

[mktcode]

I suggest gathering all relevant testable cases here now and adding them to the test suite as good as possible. The test I added is more a stub and only ensures that ?resolver=tokenlists still returns some image.

The unit test for URL pattern replacement should also be extended and the implementation then too of course.

[bonustrack]

@mktcode We should add a test case with a token and network prefix, we could have network prefix with both chain id and network short name like eip155:1:0x123 and arb1:1:0x123. Token URL with network prefix should already work with others resolvers.

[mktcode]

@bonustrack Ok, I can take care of that during this week.

Suggestion: Start issues with a continuously updated section, titled "To be tested:"

[mktcode]

Note: See updated description at the top.

[wa0x6e]

Why is there a yarn.lock update, without a dependencies update?

[mktcode]

Idk. Maybe because it hasn't been pushed before. Might be my mistake as well. Just delete it and run yarn again.

[wa0x6e]

Unit tests does not need such high timeout

[mktcode]

True. Please just take over this branch. I won't make any more changes to it.

[wa0x6e]

If we remove the await here:

• the first request triggering the token list refresh will always return immediately with nothing, with the list refreshing async
• a request waiting for list refresh will return immediately with current list, instead of waiting for new list, which is refreshing async

[mktcode]

That's the details I tried to clarify but without success. I still feel like my approach of completely decoupling the list updates and the resolver is the better one but I'm not making the decisions here. Please just take from this PR what makes sense to you.

[wa0x6e]

As mentioned here #282 (comment) this is not the expected image

[wa0x6e]

This is not working due to something unrelated to this PR: the shortname oeth is not supported:

stamp/src/utils.ts

Lines 50 to 58 in 3b50d97

	export function shortNameToChainId(shortName: string) {
	if (shortName === 'eth') return '1';
	if (shortName === 'bsc') return '56';
	if (shortName === 'ftm') return '250';
	if (shortName === 'matic') return '137';
	if (shortName === 'arb1') return '42161';
	return null;
	}

[mktcode]

this is not the expected image

yes, you'll have to decide what images make most sense to test.