Link to our repository: CS4245 Project Group 9
Point cloud is an important type of geometric data structure. The point cloud processing technique proposed in PointNet is very simple and lightweight. Its application is ranging from scene semantic parsing, part segmentation to object classification. All the while empirically showing strong performance on par or sometimes even better than state-of-the-art networks. In this case, we will be using its ability to classify objects in the Radar signatures of human activities dataset.
In this experiment, we will try to apply PointNet to data on radar signatures of human activities. The main question would thus be: "Is it feasible to apply PointNet to the 'Radar signatures of human activies' dataset?". We will do this by first preprocessing the data into point cloud and then applying PointNet to the processed data.
Over the past decades, radar-based human activity recognition systems have gained massive attention in applications such as personnel recognition, hand gesture recognition, and fall detection. In terms of classifying human motions, although many significant improvements have been made, it is a challenging task due to:
- Requirements for Feature Engineering
- Challenges for Classifying Real Human Activities
- Modeling of Spatial-temporal Characteristics
- Varying Aspect Angles
- Limited Datasets
To address these challenges, in this project, we tried to recognize six types of human movements using a radar sensor. The dataset considered in this project was used for an open challenge. Thus, classification results generated from many teams across the world have been recorded. Although various machine learning models have been tried to process this radar dataset as shown in Table 1, most of these models are using CNN-based neural networks. To the best of our knowledge, there are only a few works considering processing radar data as point cloud.
| [1] | [2] | [3] | [4] | |
|---|---|---|---|---|
| Method | RNN | SVM | Hierarchical classification | CNN |
| C.V | 10-fold | 10-fold | 10-fold | 10-fold |
| Avg. acc. | 94.3% | 92% | 95.4% | 95.43% |
Table 1: Accuracy of state of the art approaches
The goal of this project is to first formulate our problem of human activity recognition as a problem of estimating different features captured by the radar. Then to describe these features using radar point cloud. And lastly to implement the off-the-shelf point cloud processing techniques for prediction and explain the results. In the following sub-sections, the open radar dataset will be introduced first, along with a more in-detail explanation of how features of human movement (range, velocity, time, returned power) are encoded and how they can be extracted from the radar data. After that, some visualizations of the generated radar point cloud will be presented.
The radar dataset used in this project includes radar signatures of different indoor human activities performed by different people in different locations. It contains six types of human activities:
- Walking back and forth
- Sitting down on a chair
- Standing up
- Bending to pick up an object
- Drinking from a cup or glass
- Falling down
As for the hardware, the data was collected using a monostatic frequency-modulated continuous-wave (FMCW) radar (by Ancortek) operating at C-band (5.8 GHz) with a bandwidth of 400 MHz and a chirp duration of 1ms. The purpose of this dataset is to encourage researchers to develop various feature extraction and classification algorithms in the general context of assisted living, for example, to detect falls or anomalies in the normal pattern of activities of people. A more detailed introduction to the dataset can be found here.
In this project, an FMCW radar is used to obtain the information about the targets. The idea behind this radar is to transmit a frequency modulated signal whose frequency is changing over time from central
In order to obtain the range of the target, the following wave is sent through the transmitter:
Where
Where
The second part of the equation
$$ S_{b}=S_{tx}S^{*}{rx}=\alpha \cdot e^{j2\pi(f_ct +\frac{\beta t^2}{2} -f{c}(t-\tau (t))-\frac{\beta (t-\tau (t))^2}{2})}, $$
when we substitute
From this we can conclude that by analyzing the power spectrum of the beat signal and finding which frequency components are present in the beat signal, we can obtain the range and returned power from the radar to the targets. The range information can be calculated in the following manner:
There are two ways of measuring the velocity of the target, the first method is called the direct Doppler measurement. As its name implies, we calculate the Doppler shift directly by taking the Fourier Transformation along the time. However, this method requires a longer chirp time to have a good frequency resolution and an up-and-down chirp to decouple the frequency shift caused by the range and Doppler effect. The second method is called the indirect Doppler measurement. This method utilizes the fact that between each consecutive chirp, the small displacement of the target due to the constant velocity will lead to a constant phase change along the time. As we can easily calculate, this constant phase change is much more noticeable (magnified by the reciprocal of the wavelength of the carrier frequency) compared to the Doppler frequency shift.
In this project, the second method is used to determine the velocity of the target. As we would expect, the resolution of the velocity and time depends on the scan rate and the total integration time. Since the scan rate is a fixed parameter of the radar, we can also adjust the integration time (i.e. the number of scans, which in this project is 200), but we cannot increase the integration time as much as we want due to the resolution trade-off in time and frequency. Since the fixed scan rate is 1 KHz and 200 scans are coherently integrated, the time resolution will be 0.2 seconds.
For the 5.8GHz FMCW radar, the wavelength would be
For example, a velocity of
According to the above-discussed theories, the raw radar data is processed. In this section, we will show some visualization examples of the extracted features from the used radar dataset.
The range-time plot shows how the target is moving in the range. As shown in Figure 2, the target started his movement around 7m from the radar. During the 10s measurement time, he was continuously moving back and forth. Thus, to get the range information and extract the Doppler velocity of the target, we need to localize the range bins that contain the target across time.
Figure 2: The range-time plot of a human walking back and forth
As shown in Figure 3, the Velocity-time plot records the Doppler frequency shifts caused by the target's movement. In this visualization example, the sine-wave-like curve is caused by the torso movement of the target. As we can see the target has positive and negative velocity alternatively, this feature shows that the target is constantly moving forward and backward against the radar. Instead of only torso movement, some other small motions caused by activity such as moving hands and legs can also be captured by Doppler frequency shift.
Figure 3: The Doppler-time plot of a human walking back and forth
As shown in Figure 4, the generated point cloud extracts the important features present in each of the range, velocity, and time axis. In this visualization example, we can see that there is a specific shape of the point cloud which is obtained due to the walking activity performed by the person whereby the range and velocity of the object continuously oscillate because of the movement of the target user away/towards the radar during the whole time duration. Also, some higher velocity components are captured as points based on the movement of hands and legs which may be different than the movement of the whole body.
Figure 4: Generated point cloud with all features
### Translation To make use of PointNet, the aforementioned features need to be translated into a point cloud. To this end, the features were translated to a $n \times m$ matrix for each activity, where $n$ denotes the points and $m$ contains 4 values for the time (in seconds), the range (m), the velocity (m/s), and the signal power (dBm), respectively. Each matrix is written to a CSV file which can be read by PointNet.A vanilla version of PointNet, available online, was used for this project.
To reiterate, PointNet consumes raw cloud points as data. It uses a shared multi-layer perceptron to map each of the
The CSV files that were generated, needed to be preprocessed for PointNet to train. To this end, all activities containing the feature matrices were converted into a NumPy array. These arrays were then put into another NumPy array which is consumed by PointNet together with a NumPy array containing the labels for the activities.
For training, we first trained the vanilla PointNet. This was followed by modifying some configurations of the PointNet to observe the effect on PointNet performance. Three main configurations were tested; the network with tnet, nthe etwork with input feature normalization, and the network with data augmentation. Adam was used as the optimizer to train the network with a learning rate of
Initially, we train a PointNet without applying any tnet, input feature normalization, and data augmentation. Figure 5 shows the result of the experiment (this is the best result that we achieved). Here, the orange and blue curves show the train and test accuracy respectively. The result looks quite promising keeping in view that we have a used a PointNet which is a very basic point cloud classification network. Here, we must emphasize that our point cloud is not a spatial point cloud but a point cloud that represents the shape of the activity. The result clearly shows that point cloud representing the shape of the activity can be a good representation for classifying the different human activities.
Figure 5: Classification results for pointnet
Afterward, we conducted our first experiment to test whether a tnet helps in the classification results for our dataset. So, we trained PointNet by adding tnet. All other configurations were kept constant. Figure 6 shows the accuracy results for the train and test dataset. From, the figure, we can see that applying a tnet decreased the performance of the test dataset. This may be due to the reason that our dataset is too small and applying a tnet increases the network parameters causing the network to overfit to the training data. Also, applying tnet does not help in our case, since the purpose of tnet in the original paper was to normalize any transformation (e.g. rotation) of the different objects that may occur for different examples. But in our case, the shape of the point cloud cannot undergo such transformations i.e. rotation because of the nature of the way we extracted the point cloud.
Figure 6: Classification results for pointnet with tnet
In the second experiment, we investigated whether input feature normalization may help in classification performance. So, we trained a network where input features were normalized using mean and standard deviation. Figure 7 shows the results. From the figure, we can observe that even though the training accuracy improved, the testing accuracy decreased. One possible explanation can be that the feature normalization disrupts the structure of the point cloud condensing it into a very small range. This, in turn, may cause the data to be not very representative making a neural network overtrain on the training dataset.
Figure 7: Classification results for pointnet with input feature normalization
In the third experiment, we investigated the effect of data augmentation on the classification results. In this respect, we apply some data augmentation techniques from PointNet2. Here, three data augmentation techniques were applied: jitter, scaling and shift. Figure 8 shows the results. From the figure, we can see that data augmentation slightly decreased the performance. Possible reasons can be that we directly used the data augmentation paramaters coming from a network using a spatial point cloud. We think that further investigations related to what particular data augmentation techniques are essential for our point cloud dataset may help in improving the results. This can be investigated in future work.
Figure 8: Classification results for pointnet with data augmentation
Table 2 gives the comparison between the results achieved with different PointNet configurations. We can see that the vanilla PointNet gives the best performance for the testing set.
| Network | Training | Testing |
|---|---|---|
| PointNet | 93% | 88% |
| PointNet with tnet | 91% | 84% |
| PointNet with feature normalization | 95% | 83% |
| PointNet with data augmentation | 90% | 84% |
Table 2: Accuracy for different PointNet configurations
We have used a very small PointNet for the project. So, the network can be trained quickly. It took us around 20 minutes to train the network for 50 epochs on an Intel Xeon W-2245 CPU. The total number of parameters is
To reiterate, the main question of this experiment was: "Is it feasible to apply PointNet to the 'Radar signatures of human activities' dataset?".
As the results have shown, it is feasible to apply PointNet to this particular dataset, given that the dataset is preprocessed to comply with PointNet.
Since it is feasible, other follow-up questions were explored, such as 'How well does PointNet perform against state-of-the-art methods used for this particular dataset in terms of accuracy and time'.
Due to the lack of information provided by other state-of-the-art approaches and lack of time in this experiment, it is difficult to conclude how well PointNets relatively performs in terms of training time.
It is, however, possible to draw more of a concrete conclusion in terms of comparison of the accuracy of the PointNet and state-of-the-art approaches for this dataset.
As we can derive from table 2, the best performing PointNet approach leads to an 88% accuracy while the state-of-the-art approaches mentioned in table 1 have a much higher accuracy of 94.3%, 92%, 95.4%, and 95.43% respectively.
The PointNet approach could be expanded and fine-tuned more, for example, by using PointNet++ instead, to have a better accuracy performance. However, this is left as future work.
[1] Jiang, Haoyang, et al. "Human activity classification using radar signal and RNN networks." (2021): 1595-1599. [2] Li, Zhenghui, et al. "Multi-domains based human activity classification in radar." IET International Radar Conference (IET IRC 2020). Vol. 2020. IET, 2020. [3] Li, Xingzhuo, et al. "Radar-based hierarchical human activity classification." IET International Radar Conference (IET IRC 2020). Vol. 2020. IET, 2020. [4] Xiaolong, Zhou, Jin Tian, and Du Hao. "A lightweight network model for human activity classifiction based on pre-trained mobilenetv2." (2021): 1483-1487.