[Resolved][Vote] Discuss the maintaince and performance issues of traditional time-series databases(InfluxDB and IoTDB)

[wu-sheng]

Hi, @apache/skywalking-committers

I have thought about this discussion and vote for months. Also, several people asked me about this dozens of times.
InfluxDB 1.x and IoTDB plugins exist for years and months, I noticed there are several issues.

1. InfluxDB faced serious performance impact when it is used in collecting/query traces and logs. Mostly due to trace ID making too many candidates for query index, which is not expected/designed in this kind of database.
2. IoTDB has no feedback since it is added. We have to consider maintenance payload in months.
3. Every time when we need to update the core, we have to take care of InfluxDB and IoTDB implementations, including e2e tests and inside logic.

In the SkyWalking community, we follow the principle, (mutable) core features should have stable committers to keep eyes on it, also, they should have stable feedback from the community. AFAIK, I don't think both of these features reach this standard.

So, I am initializing this discussion as well as voting, we should consider removing both of these options.

[kezhenxu94]

Hi. Committers and PMC members please vote in mailing list. We can’t see the details of poll results here.

[wu-sheng]

It is better on the mail list, I have sent the mail. If all agree with Yes, remove both. , then there is no concern.

[wu-sheng]

If you don't vote for Yes, remove both., please leave comments here.

[jixuan1989]

Hi, I come from IoTDB community, and thank @LIU-WEI-git for the contribution to integrating IoTDB with Skywalking.

IMO,

1. I suggest Skywalking keep the storage layer replaceable, which can give users more choices.
2. I also suggest Skywalking main code repo remove the two optional implementation if we think the maintenance cost is high...

In this case, I think on Skywalking user guide web page, we can have a introduction about how to implement a third part storage layer, and list the existing implementions.

For each implementation, we need to maintain a table to show the version compatibility,
e..g,

skywalking	IoTDB	IoTDB-skywalking plugin
[8.4, 8.9]	[v0.12.0, )	v0.1
[9.0, )	[v0.13.0, v0.14.0)	v0.2

[wu-sheng]

If there are people declaring maintain, and providing the repository path, we could add links to the doc.

[jixuan1989]

@LIU-WEI-git how do you think about that?

[jixuan1989]

well, I need to start a thread on iotdb mailing list to discuss whether we create a new repo or put the codes to iotdb root repo.

[wu-sheng]

I think once we have a consensus about removing IoTDB from the default release, the IoTDB community and original author @LIU-WEI-git have all rights to re-host this part of the codes, because

1. IoTDB and SkyWalking are sharing The Apache Software Foundation IP, the codes granted to us in APLv2, IoTDB community has been granted too.
2. @LIU-WEI-git as the original author, he has the full rights still to push the codes on his own repository if he like. Same as our committer @dmsolr for InfluxDB storage implementation codes(most of codes, considering several other contributors)

[LIU-WEI-git]

@LIU-WEI-git how do you think about that?

I support remove IoTDB storage option and move it to a new repo for those who interest it.

[jixuan1989]

yes, we want to discuss about whether to move the codes to iotdb's source code repo or create a new iotdb-prefixed apache repo :D 吴晟 Wu Sheng ***@***.***> 于2022年5月13日周五 17:56写道：

…

I think once we have a consensus about removing IoTDB from the default release, the IoTDB community and original author @LIU-WEI-git <https://github.com/LIU-WEI-git> have all rights to re-host this part of the codes, because 1. IoTDB and SkyWalking are sharing The Apache Software Foundation IP, the codes granted to us in APLv2, IoTDB community has been granted too. 2. @LIU-WEI-git <https://github.com/LIU-WEI-git> as the original author, he has the full rights still to push the codes on his own repository if he like. Same as our committer @dmsolr <https://github.com/dmsolr> for InfluxDB storage implementation codes(most of codes, considering several other contributors) — Reply to this email directly, view it on GitHub <#9059 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHZOVX4BHF4OISRUBWNBI3VJYRN7ANCNFSM5V2XPXEA> . You are receiving this because you commented.Message ID: ***@***.***>

[LIU-WEI-git]

Hi, I'm the original author of IoTDB storage implementation.

IoTDB is a timeseires database mainly facing IIoT and get a great performance improvement. SkyWalking IoTDB storage plugin is an attempt of IoTDB for OAP system. IoTDB and the plugin's schema design could not support well for those complex queries in SkyWalking and its implementation has flaws.

Thanks other contributors to maintenance of IoTDB storage plugin. I support remove IoTDB storage plugin from SkyWalking main repo.

[wu-sheng]

#9060 is for removing relative codes and tests.

[wu-sheng]

There is no objection to this. I am locking this discussion.