Cloud-Robotic AI Benchmarking for Edge-cloud Collaborative Lifelong Learning

[MooreZheng]

What would you like to be added/modified:
Based on real-world datasets provided by industry members of KubeEdge SIG AI, the issue aims to build a lifelong learning benchmarking on KubeEdge-Ianvs. Namely, it aims to help all Edge AI application developers to validate and select the best-matched algorithm of lifelong learning. It includes:

1. Work together to release a new dataset to the public!
2. Implement critical algorithm or system metrics, e.g., BWT, FWT and throughput;
3. (Optional) Develop a baseline algorithm for this benchmark;

Why is this needed:
It is estimated that by 2025, 75% of the world's data will be generated at the edge, and the computing power on the cloud will be more abundant. Edge-cloud collaborative artificial intelligence will become an inevitable trend, and its demand will be further released. Among them, the global service robot market is expected to reach 90-170 billion US dollars in 2030. The use of cloud-native edge computing and artificial intelligence technology to deal with the issues of the robot industry and complete industrial transformation has also become the focus of the industry.

In recent years, lifelong learning-related algorithms such as Lifelong SLAM and Lifelong Object Detection have become popular for the problem of edge-data heterogeneity and small samples, but the real-world practise requires further considerations on its edge-cloud collaborative nature. To further accelerate research and results transformation, the KubeEdge community released the first open source edge-cloud collaborative lifelong learning framework and its resource orchestration template on KubeEdge-Sedna in June 2021. Moreover, the collaborative AI benchmarking KubeEdge-Ianvs in July 2022 is also released with related benchmark datasets and compute metrics.

This project aims to develop the edge-cloud collaborative lifelong learning benchmarking that are suitable for robotic scenarios based on KubeEdge-Ianvs. This project will help all Edge AI application developers to validate and select the best-matched algorithm of lifelong learning. The benchmark can include dataset, metrics and algorithm. Specific applications include but are not limited to robot navigation, inspection, cleaning, delivery, etc. KubeEdge SIG AI has already prepared real-world datasets for everyone to explore!

Recommended Skills:
TensorFlow/Pytorch, Python

Useful links:
Introduction to Ianvs
Quick Start
How to test algorithms with Ianvs
Testing incremental learning in industrial defect detection
[Opensource Summit Japan] From Groud to Space: Cloud-Native Edge Machine-Learning Case Studies with Kubeedge-Sedna
[ACM e-Energy'22] Towards Lifelong Thermal Comfort Prediction with KubeEdge-Sedna
[ACM CIKM'22] Towards Edge-Cloud Collaborative Machine Learning: A Quality-aware Task Partition Framework
[KubeEdge云原生边缘计算公开课] 边缘智能进阶：适配多样场景和应对分布式系统
[KEAW'22] 边云协同终身学习在智慧园区及工业领域创新探索及落地

[MooreZheng]

If one has any question to this issue, you are very welcome to leave any message here. @luosiqi will also take a look at it.

[smilingprogrammer]

Hi @MooreZheng , my name is Abdulsobur. Just to clarify from the description. A Dataset is already available to solve this issue.

[smilingprogrammer]

Also @MooreZheng from description the project is about creating a model that helps AI engineers on validations and algorithm that are best fit for a certain task. Its then judged by the dataset, performance metrics and algorithm. Am i right?

[MooreZheng]

Greetings Abdulsobur.

1. Yes. A new dataset is available with labels and our AI model can train and inference on it. But the dataset is at present in a initial stage - some cleaning and documentation is needed.
2. Exactly. One might also want to take a look at the ianvs documentation for more examples.

[smilingprogrammer]

@MooreZheng
Nice, wanted to ask on the Data Processing too (cleaning). Thanks for making it very clear to me.

This project really looks very promising. Will take my time to go through the resources and documentation you shared too.

Will be applying for it and love to be mentored through the LFX mentorship.

[smilingprogrammer]

@MooreZheng
Please kindly share me the link to apply for it?
Thanks.

[MooreZheng]

@MooreZheng Please kindly share me the link to apply for it? Thanks.

Sure. Thank you so much for your interest.
@luosiqi is the project mentor. hey siqi, pls help to follow up.

Feel free to let us know if there is anything we could help. :D

[luosiqi]

@MooreZheng Please kindly share me the link to apply for it? Thanks.

Hi, you can apply for the project in the following link.
https://mentorship.lfx.linuxfoundation.org/project/50cdbd65-e0cd-4c0f-8c63-6bd5c603ba89

[smilingprogrammer]

Thanks for the help @MooreZheng @luosiqi

[abhiyant-10]

Hello @luosiqi , I am a final year CSE student based in India and upcoming research assistant at NTU Singapore. I posses the required skills and have also worked on various data analytics and ML/AI projects . This seems a interesting task and I would like to work on it . I am going through all the useful links for better understanding .Looking forward to work with you.

[luosiqi]

Hello @luosiqi , I am a final year CSE student based in India and upcoming research assistant at NTU Singapore. I posses the required skills and have also worked on various data analytics and ML/AI projects . This seems a interesting task and I would like to work on it . I am going through all the useful links for better understanding .Looking forward to work with you.

Hi, welcome to work with us. Please apply for the project with the following link. We look forward to your participation.
https://mentorship.lfx.linuxfoundation.org/project/50cdbd65-e0cd-4c0f-8c63-6bd5c603ba89

[smilingprogrammer]

Hi @luosiqi and @MooreZheng
From the ianvs documentation you shared and following the architecture and module mentioned in the documentation, will this project also include a Simulation Controller in the Test Case Controller?

[smilingprogrammer]

Also @MooreZheng @luosiqi will the test environment also involve support for CRUD actions?

[sachin7695]

I am very excited after reading about this project. I am pretty passionate about ai and robotics and this project will be a great breakthrough to hone my skills in ai and robotics both combined. I have applied for mentorship. looking forward to a positive response from the team

[Sai-Suraj-27]

Hi, I am suraj, I found this project very interesting. I have a small doubt @MooreZheng, in the example for system metrics did you mean to say throughput? (is it a spelling mistake?) or it is thoughput only? and BWT, FWT means Burrows-Wheeler-transform, and Fast Wavelet Transform algorithms respectively right? Please correct me if I am wrong.
Thank you.

[Hacker-Hacks-and-hacks]

@MooreZheng Please kindly share me the link to apply for it? Thanks.

Sure. Thank you so much for your interest. @luosiqi is the project mentor. hey siqi, pls help to follow up.

Feel free to let us know if there is anything we could help. :D

If one has any question to this issue, you are very welcome to leave any message here. @luosiqi will also take a look at it.

Will there be any stipend for the successful completion of this project?

[luosiqi]

Controller

Simulation controller has been realized

Hi @luosiqi and @MooreZheng From the ianvs documentation you shared and following the architecture and module mentioned in the documentation, will this project also include a Simulation Controller in the Test Case Controller?

In this project, simulation controller is important but not the key point. We more value new dataset publication and the corresponding algorithm or metrics. Moreover, if you are interested in simulation controller, you can try to implement it in this project.

[luosiqi]

I am very excited after reading about this project. I am pretty passionate about ai and robotics and this project will be a great breakthrough to hone my skills in ai and robotics both combined. I have applied for mentorship. looking forward to a positive response from the team

Hi, we also look forward to your participation.

[luosiqi]

@MooreZheng Please kindly share me the link to apply for it? Thanks.

Sure. Thank you so much for your interest. @luosiqi is the project mentor. hey siqi, pls help to follow up.
Feel free to let us know if there is anything we could help. :D

If one has any question to this issue, you are very welcome to leave any message here. @luosiqi will also take a look at it.

Will there be any stipend for the successful completion of this project?

Yes, of course. Stipend message can be found in https://mentorship.lfx.linuxfoundation.org.

[luosiqi]

Also @MooreZheng @luosiqi will the test environment also involve support for CRUD actions?

It only supports manual CRUD actions. In the following work, we consider to offer user interfaces for CRUD actions.

[Sai-Suraj-27]

Hi, I am suraj, I found this project very interesting. I have a small doubt @MooreZheng, in the example for system metrics did you mean to say throughput? (is it a spelling mistake?) or it is thoughput only? and BWT, FWT means Burrows-Wheeler-transform, and Fast Wavelet Transform algorithms respectively right? Please correct me if I am wrong. Thank you.

@luosiqi sir, can you clarify about this? I am also looking forward to working on this project, I am currently reading Ianvs documentation, and after getting a good understanding of the project I will apply to it, in the LFX platform.

[luosiqi]

Hi, I am suraj, I found this project very interesting. I have a small doubt @MooreZheng, in the example for system metrics did you mean to say throughput? (is it a spelling mistake?) or it is thoughput only? and BWT, FWT means Burrows-Wheeler-transform, and Fast Wavelet Transform algorithms respectively right? Please correct me if I am wrong. Thank you.

The word "throughput" is correct spelling which refers to the rate of message delivery over a communication channel. BWT and FWT are short for Backward Transfer and Forward Transfer. BWT is used to evaluate anti-forgetting ability for historical tasks while FWT is to measure generalization ability for future tasks.

[Sai-Suraj-27]

Oo...Ok ok got it, I got confused when I saw thoughput (@luosiqi maybe you can correct it in the main comment so that others will not confuse), Thanks for the clarification related to BWT, & FWT.

[luosiqi]

Oo...Ok ok got it, I got confused when I saw thoughput (@luosiqi maybe you can correct it in the main comment so that others will not confuse), Thanks for the clarification related to BWT, & FWT.

@MooreZheng please correct the word “thoughput”

[MooreZheng]

Oo...Ok ok got it, I got confused when I saw thoughput (@luosiqi maybe you can correct it in the main comment so that others will not confuse), Thanks for the clarification related to BWT, & FWT.

@MooreZheng please correct the word “thoughput”

Sure. Edited and thx for reminder.

[smilingprogrammer]

Also @MooreZheng @luosiqi will the test environment also involve support for CRUD actions?

It only supports manual CRUD actions. In the following work, we consider to offer user interfaces for CRUD actions.

Thanks for the clarification

[PC-11-00]

@MooreZheng @luosiqi Hey guys,
I hope you are doing well. I am new to this community, can you tell me how to get started ,any resources would be helpful.
Is there any community chat group , where I can connect with the people.

[monalisha31]

This project is quite interesting and i have applied through LFX mentorship program. Looking forward to your response.

[luosiqi]

This project is quite interesting and i have applied through LFX mentorship program. Looking forward to your response.

Welcome~

[hsj576]

Hi @luosiqi
I am a graduate student from Fudan University. My current research direction involves edge intelligence. I am very interested in this project and would like to participate in the research of algorithms related to lifelong learning in Ianvs. I have some questions about the attribution of academic achievements in this project. Is it possible for me to be the first author of the paper which covers some of my work during my internship?
I hope to hear from you soon. Thank you.

[ktwillcode]

Hi @luosiqi
Kartikeya this side, I applied for the mentorship and submitted a Cover letter and Resume. Is there anything more to complete the process?

[luosiqi]

@MooreZheng @luosiqi Hey guys, I hope you are doing well. I am new to this community, can you tell me how to get started ,any resources would be helpful. Is there any community chat group , where I can connect with the people.

You can connet us with kubeedge sig ai slack. https://app.slack.com/client/TDZ5TGXQW/C01EG84REVB/details

[luosiqi]

Hi @luosiqi I am a graduate student from Fudan University. My current research direction involves edge intelligence. I am very interested in this project and would like to participate in the research of algorithms related to lifelong learning in Ianvs. I have some questions about the attribution of academic achievements in this project. Is it possible for me to be the first author of the paper which covers some of my work during my internship? I hope to hear from you soon. Thank you.

Hello, as you may know, author order in an academic paper is sorted by contribution. While in LFX Mentorship, the participant plays the most important role in this project. It is very possible that you will be the the first author.

[luosiqi]

Hi @luosiqi Kartikeya this side, I applied for the mentorship and submitted a Cover letter and Resume. Is there anything more to complete the process?

As far as I'm concerned, after sumitting the application, all you have to do is to wait for the notification.

[hsj576]

Hi @luosiqi I am a graduate student from Fudan University. My current research direction involves edge intelligence. I am very interested in this project and would like to participate in the research of algorithms related to lifelong learning in Ianvs. I have some questions about the attribution of academic achievements in this project. Is it possible for me to be the first author of the paper which covers some of my work during my internship? I hope to hear from you soon. Thank you.

Hello, as you may know, author order in an academic paper is sorted by contribution. While in LFX Mentorship, the participant plays the most important role in this project. It is very possible that you will be the the first author.

Thanks for your reply!

[PC-11-00]

@MooreZheng @luosiqi Hey guys, I hope you are doing well. I am new to this community, can you tell me how to get started ,any resources would be helpful. Is there any community chat group , where I can connect with the people.

You can connet us with kubeedge sig ai slack. https://app.slack.com/client/TDZ5TGXQW/C01EG84REVB/details

Hey @luosiqi , thanks for the reply, but this link is not working

[Sai-Suraj-27]

@MooreZheng @luosiqi Hey guys, I hope you are doing well. I am new to this community, can you tell me how to get started ,any resources would be helpful. Is there any community chat group , where I can connect with the people.

You can connet us with kubeedge sig ai slack. https://app.slack.com/client/TDZ5TGXQW/C01EG84REVB/details

Hey @luosiqi , thanks for the reply, but this link is not working

https://join.slack.com/t/kubeedge/shared_invite/zt-1piyy1z4g-QUpjHfM_jFlICxhF6SOSew Can you try this once?

[PC-11-00]

@Sai-Suraj-27 Thanks ,it worked

[Sai-Suraj-27]

@luosiqi I read about different evaluation metrics in continuous learning I found BWT (-ve of Forgetting rate), FWT, and the average performance over all seen tasks after full training as the most common metrics used. So, are we going to implement the same metrics in this project or do you have any other specific metrics in mind?

[Sai-Suraj-27]

@Sai-Suraj-27 hey ! , i want to connect with you i want some guidence can you please connect with me -- twitter : lug__aman , Linkedin : https://www.linkedin.com/in/aman-kumar-5bb609228/ instagram : lug__aman

@Aman123lug Message me on Twitter bro, I guess you should not comment here unless it is related to this issue.

[Aman123lug]

@Sai-Suraj-27 okay

[Aman123lug]

@Sai-Suraj-27 in twitter i am not able to msg you bro don't know why

[Sai-Suraj-27]

@Aman123lug Email me, bro, please don't comment here. My email: [email protected]

[luosiqi]

@luosiqi I read about different evaluation metrics in continuous learning I found BWT (-ve of Forgetting rate), FWT, and the average performance over all seen tasks after full training as the most common metrics used. So, are we going to implement the same metrics in this project or do you have any other specific metrics in mind?

As mentioned in the description of this issue, we place priority on BWT and FWT in this project. In addition, we hope that mentees also come up with new metrics of contiuous learning or new definition of BWT and FWT.

[Sai-Suraj-27]

@luosiqi I read about different evaluation metrics in continuous learning I found BWT (-ve of Forgetting rate), FWT, and the average performance over all seen tasks after full training as the most common metrics used. So, are we going to implement the same metrics in this project or do you have any other specific metrics in mind?

As mentioned in the description of this issue, we place priority on BWT and FWT in this project. In addition, we hope that mentees also come up with new metrics of contiuous learning or new definition of BWT and FWT.

Okk...Got it👍

[sanggusti]

Hi @MooreZheng @luosiqi I've already submitted Cover Letter and Resume. Is there anything more to complete the process? I see some other mentorship applications needed to complete pretest or issue. Do we have one here? Thanks

[Gaurav7888]

Hi, My name is Gaurav Sarkar, final year undergrad student. I have submitted cover letter and resume. Looking forward to it.

[luosiqi]

Hi @MooreZheng @luosiqi I've already submitted Cover Letter and Resume. Is there anything more to complete the process? I see some other mentorship applications needed to complete pretest or issue. Do we have one here? Thanks

Actually, we do have pretest which will be published this week as an issue.

[luosiqi]

Hi, My name is Gaurav Sarkar, final year undergrad student. I have submitted cover letter and resume. Looking forward to it.

Hi, welcome. We look forward to your participation.

[luosiqi]

Hi all. For better selection, we raise a pretest for the application. Based on Ianvs, we designed challenges to evaluate the candidates at #54. Please take a look at this issue and try out to complete the tasks in it. And finally we will select the applicant who gains the highest score of the tasks as mentee of this project.

[mohitmishra786]

@luosiqi Hi, My name is Mohit Mishra. I have already submitted my resume & cover letter. Looking forward to it