Updates on 12.29 2023

1. modify the README.md in examples/cityscapes
2. add a proposal in docs/proposals
3. add .DS_Store to gitignore, which are useless file generated by MacOS

Signed-off-by: nailtu <[email protected]>

.gitignore

@@ -131,3 +131,7 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+# generated files by MacOS
+.DS_Store 
+*/.DS_Store 

docs/proposals/algorithms/lifelong-learning/Unknown_Task_Processing_Algorithm_based_on_Lifelong_Learning_of_Ianvs.md

@@ -0,0 +1,79 @@
+# Unknown Task Processing Algorithm based on Lifelong Learning of Ianvs
+
+
+## 1 Project Background
+In the robot anomaly detection and inspection scenarios, lifelong learning needs to deal with the identified unknown tasks. In the process of lifelong learning, the data of unknown tasks may be small samples and heterogeneous. Generative adversarial networks (GANs) are advanced generative models that can generate data based on the distribution of real data. KubeEdge SIG AI released a new feature on Sedna/Ianvs, trying to use GANs to solve the small sample problem, but the data generated by GANs is not labeled. Self-taught learning (STL) is an algorithm that uses sparse coding and unlabeled data to build high-level data representations to improve classification performance. Therefore, one solution is to use GANs and STL in the cloud to expand small samples and heterogeneous data respectively and extract data representations to help the model train better. With this solution, model training can converge faster and reduce the number of training rounds, achieving fast and efficient training in small samples and heterogeneous data scenarios.    
+
+
+# 2. Goals
+1. Unknown task processing with unlabeled data;
+2. Compared with the scenario using all data, the unknown task processing algorithm reduces the number of convergence iterations by 50% and the training loss is reduced by 50%;
+3. Integrate the algorithm into Sedna/Ianvs to complete the process.
+
+# 3. Proposal
+## Overview
+We propose an approach of combining [GAN](https://en.wikipedia.org/wiki/Generative_adversarial_network) and [Self-taught Learning](https://ai.stanford.edu/~hllee/icml07-selftaughtlearning.pdf) to solve small sample problem in Ianvs lifelong learning, as shown in the figure below, corresponding to the [pull request](https://github.com/kubeedge/ianvs/pull/90).
+
+![](./images/unknown_task_processing_algorithm_based_on_lifelong_learning_of_ianvs/ianvs-lifelonglearning.png)
+
+We describe the arichtecture and the process. More details can be seen in [Design Details](#Design%20Details).
+
+![](./images/unknown_task_processing_algorithm_based_on_lifelong_learning_of_ianvs/ianvs-lifelonglearning2.png)
+
+1. Train GAN with orginal small sample data;
+2. GAN generates more data according to the probability distribution;
+3. Train Autoencoder (which is consist of encoder and decoder) with the data generated by GAN;
+4. Use encoder to get data representation of original small sample data;
+5. Use data representation and orignal labels to train model that the unseen task model needs;
+6. Ouput a well trained model.
+
+# 4. Use Case
+The case uses the `Cityscapes Dataset` and trains the `DeepLabV3` model as an unknown task. Specifically, the leftImg8bit_trainvaltest in the `Cityscapes Dataset` is selected, which has 5000 images, and the size of each image is `2048*1024*3`. In this case, the `DeepLabV3` is integrated into Ianvs and the `DeepLabV3` is configured as an unseen task. The `Cityscapes Dataset` shown in figure below is one of the most authoritative and professional image semantic segmentation evaluation sets in the field of automatic driving. It focuses on the understanding of urban road environment in real scenes, and the task is more difficult and closer to the popular requirements of automatic driving. The `DeepLabV3` model is a semantic segmentation algorithm, and the use of empty convolution is discussed, this makes it possible to obtain a larger receptive field to obtain multi-scale information in the framework of cascade modules and spatial pyramid pooling.
+
+![](./images/unknown_task_processing_algorithm_based_on_lifelong_learning_of_ianvs/cityscape_dataset.png)
+
+The runtime environment is as follows: the operating system is Ubuntu 20.04, the CPU is Intel i7 processor, the memory is 32GB, the graphics card is NVIDIA RTX 3060(12GB), and the Ianvs version is v0.2.0.    
+
+Small sample environment settings. The case randomly selects 100 images in the data set as a small sample environment and compares it with the training of the entire data set.    
+
+Small sample unknown task training demonstration. In this case, small sample training is carried out on the Ianvs platform, and the training of all data sets is compared to prove the effectiveness of the scheme. The training process can be carried out in the Ianvs lifelong learning platform, or the trained model can be directly deployed to the Ianvs cloud.    
+
+We learn the data distribution of small sample data by GAN, and Figure shows the images generated by the generator. The generator generates data for self-learning module training.    
+
+![](./images/unknown_task_processing_algorithm_based_on_lifelong_learning_of_ianvs/generator_imgs.png)
+
+We use the images generated by the generator to train the STL module, Figure shows the data representations extracted by STL, the first line represents the original image, the second/third/fourth line represents the extracted data representation information.
+
+![](./images/unknown_task_processing_algorithm_based_on_lifelong_learning_of_ianvs/stl_imgs.png)  
+
+Figure shows the small sample unknown task training comparison. The following three training processes represent the effect of training with all data (yellow line), the effect of training with only 100 images (blue line), and the training effect after using only 100 images and using the trained encoder data representation (red line).
+
+![](./images/unknown_task_processing_algorithm_based_on_lifelong_learning_of_ianvs/exp_results.png)  
+
+It can be found that using only 100 images to train not only the loss decreases slowly, but also the phenomenon of oscillation instability, and the training loss variance of the last 500 rounds of iteration is 0.08. However, after the trained encoder data representation and then training, not only the loss drops quickly, the training is stable, and it achieves better results than using all the data for training. Specifically, our method converges after 20 rounds of iteration, and the number of iterations is 90% less than that of the training process using all the data sets. At the same time, our method can achieve a lower training loss (under the same loss function), which is 80% lower than the training process using the entire data set.   
+
+The case fully proves that the solution is effective in model training in the case of small samples and is suitable for Sedna/Ianvs lifelong learning.
+
+# 5. Design Details
+## Architecture
+
+- Overview  
+  Our algorithm includes GAN and STL, both of which are based on unlabeled data and only use labels for small sample data when training unknown tasks.
+  ![](./images/unknown_task_processing_algorithm_based_on_lifelong_learning_of_ianvs/overview.png)
+
+- GAN (we refer to [Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis](https://openreview.net/forum?id=1Fqg133qRaI)).   
+  The GAN module is designed for small sample data, while preventing training forgetting, the network layer is designed as deep as possible, and the deep information of the data is put forward.    
+
+  Discriminator
+
+  ![](./images/unknown_task_processing_algorithm_based_on_lifelong_learning_of_ianvs/discriminator.png)
+
+  Generator
+
+  ![](./images/unknown_task_processing_algorithm_based_on_lifelong_learning_of_ianvs/generator.png)
+
+
+- Convolutional AutoEncoder of Self-taught Learning    
+
+  In implementing the Self-taught Learning module, we use Convolutional AutoEncoder for fast extracting data representations.
+  ![](./images/unknown_task_processing_algorithm_based_on_lifelong_learning_of_ianvs/cae.png)

examples/cityscapes/unseen_task_processing/GANwithSelfTaughtLearning/README.md

@@ -1,45 +1,14 @@
 # Integrating GAN and Self-taught Learning into Ianvs Lifelong Learning
 
-## Overview
+Our algorithm is dedicated to solving the `small sample` and `heterogeneous data` issues in the Ianvs lifelong learning process via `Generative Adversarial Networks (GANs)` ans `Self-taught Learning (STL)`. Here we focus on how to deploy the algorithm in Ianvs. For the principle and algorithm details, please refer to [Unknown Task Processing Algorithm based on Lifelong Learning of Ianvs](../../../../docs/proposals/algorithms/lifelong-learning/Unknown_Task_Processing_Algorithm_based_on_Lifelong_Learning_of_Ianvs.md).  
 
-We proposal an approach of combining [GAN](https://en.wikipedia.org/wiki/Generative_adversarial_network) and [Self-taught Learning](https://ai.stanford.edu/~hllee/icml07-selftaughtlearning.pdf) to solve small sample problem in Ianvs lifelong learning, as shown in the figure below, corresponding to the [pull request](https://github.com/kubeedge/sedna/pull/378).
+## Device Requirements and Ianvs Preparation
 
-For quick start, just jump directly to [Developer Notes](##Developer%20Notes).
+For device requirements and Ianvs preparation, please refer to [lifelong_learning_bench](../../../cityscapes-synthia/lifelong_learning_bench/curb-detection/README.md).
+
+## Development Notes 
 
-![](./imgs/ianvs-lifelonglearning.png)
-
-We describe the arichtecture and the process. More details can be seen in [Architecture](##Architecture).
-
-![](./imgs/ianvs-lifelonglearning2.png)
-
-1. Train GAN with orginal small sample data
-2. GAN generates more data according to the probability distribution
-3. Train Autoencoder (which is consist of encoder and decoder) with the data generated by GAN
-4. Use encoder to get data representation of original small sample data
-5. Use data representation and orignal labels to train model that the unseen task model needs
-6. Ouput a well trained model
-
-## Architecture
-
-- Overview
-
-  ![](./imgs/overview.png)
-
-- GAN (we refer to [Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis](https://openreview.net/forum?id=1Fqg133qRaI))
-
-  Discriminator
-
-  ![](./imgs/discriminator.png)
-
-  Generator
-
-  ![](./imgs/generator.png)
-
-- Convolutional AutoEncoder of Self-taught Learning
-
-  ![](./imgs/cae.png)
-
-## Developer Notes
+Our algorithm for solving the `small sample` and `heterogeneous data` issues in the Ianvs lifelong learning process is in the folder `GANwithSelf-taughtLearning`. We show the details of `GANwithSelf-taughtLearning` below.
 
 ```bash
 GANwithSelf-taughtLearning # root path of the project
@@ -58,16 +27,32 @@ GANwithSelf-taughtLearning # root path of the project
 		- ./train_results # training results of the model to be trained
 	- util.py # util module
 ```
-For the trained weights and models, please refer to [Google drive](https://drive.google.com/drive/folders/1IOQCQ3sntxrbt7RtJIsSlBo0PFrR7Ets?usp=drive_link).    
+For the trained weights and models, please click [here](https://drive.google.com/drive/folders/1IOQCQ3sntxrbt7RtJIsSlBo0PFrR7Ets?usp=drive_link) to download.    
 
 There are two ways to use it. You can use the module on its own or as an integration into Ianvs.
 
+###  Use the Module on Its Own
 For using the module on its own, config the `config.yaml` to let `GANwithSelf-taughtLearning` know where the **dataset** is, what **model** you want to train and what the **hyperparameters** are.
 
-Then a common use process can be shown below:
+Then a common process can be shown below:
+
+1. run `python ./GAN/train.py`   
+The GAN module starts training over the dataset configured. Figure shows a training result with an NVIDIA RTX 3060 (12GB) graphics card.   
+![](imgs/GAN_training.png)
+
+2. run `python ./self-taught-learning/train.py`   
+The process first uses the trained GAN model to generate certain data (fake data) and then trains STL over fake data. The training process of Convolutional AutoEncoder of Self-taught Learning can be shown below.
+![](imgs/vae_training.png)
+
+3. run `./train.py`   
+The trained STL first extracts the labeled data to data representations and then the we use data representations and labels to train the unseen task which is configured in the `config.yaml`.
+
+
+### Use the Module as an Integration into Ianvs
+For using the module as an integration into Ianvs, please first compile this version of [Sedna](https://github.com/nailtu30/sedna/tree/ospp-final), where we integrate  `GANwithSelf-taughtLearning` into folder `lib/sedna/algorithms/unseen_task_processing`.    
+
+Then please personalize your own `config.yaml` as explained in [Use the Module on Its Own](###Use%20the%20Module%20on%20Its%20Own).
 
-1. run `python ./GAN/train.py`
-2. run `python ./self-taught-learning/train.py`
-3. run `./train.py`
+Last but not least, just start Ivans normally, where our algorithm has been integrated into the main process, and you can simulate unseen task via setting `model to be trained` in the `config.yaml`. We show a successful deployment below.
 
-For using the module as an integration into Ianvs, please first compile this version of [Sedna](https://github.com/nailtu30/sedna/tree/ospp-final), where we integrate  `GANwithSelf-taughtLearning` into folder `lib/sedna/algorithms/unseen_task_processing`. Then just start Ivans normally, where our algorithm has been integrated into the main process, and you can simulate unseen task via setting `model to be trained` in the `config.yaml`.
+![](imgs/success_launch.png)