In 2018, I graduated from Xidian University with a bachelor's degree
+in communication engineering. In the same year, I was admitted to the
+University of Science and Technology of China as the first place in
+software engineering where I recievied a master's degree in 2021.
+
From July 2019 to June 2021, I was an intern at Intel Beijing engaged
+in the decision-making and planning of autonomous vehicles. From July
+2021 to August 2022, I worked on lane change planning for self-driving
+cars in Didi's autonomous driving department (Voyager).
+
I recieved the following awards:
+
+
First class award. Sep. 2014
+
Freshman Entrance Scholarship. Sep. 2018
+
Bosch scholarship. Sep. 2019
+
The second prize of the National Postgraduate Mathematical Contest
+in Modeling. Nov. 2019
During my internship at HPCC Systems in 2023, I had the incredible
+opportunity to work on a dynamic project that pushed the boundaries of
+my technical skills and knowledge. In this post, I will summarize the
+key aspects and achievements of my internship project, highlighting the
+challenges I faced, the solutions I implemented, and the valuable
+lessons I learned along the way.
In HPCC Systems, pictures need to be uploaded to HPCC Systems before
+they can be accessed. This blog post documents how to upload an image to
+HPCC Systems
ECL is not a procedural language, that is, the order in which
+operations are performed is not necessarily the order given by the code,
+which can cause trouble in calculating the execution time of operations.
+This blog describes how to calculate the execution time of certain
+operations for users to evaluate performance.
According to the configuration instructions provided by TensorFlow
+and CUDA, I installed and configured the environment, but HPCC Systems
+couldn't detect the GPU. This blog post documents the process of how I
+resolved this issue.
When upgrading the HPCC GNN bundle from TensorFlow1.x to
+TensorFlow2.x, I encountered an error of
+numpy() is only available when eager execution is enabled.
+This blog records the cause and solution of the problem.
In the ECL language, operations such as OUTPUT cannot be performed in
+the definition, complicating our view of intermediate variables. This
+blog introduces debugging tips for several ECL programming language.
The Node.js, Hexo and Next themes used in previous blogs are
+outdated, causing a lot of compatibility issues. So I reconfigured my
+blog and this post records my configuration and development process.
According to the configuration instructions provided by TensorFlow
+and CUDA, I installed and configured the environment, but HPCC Systems
+couldn't detect the GPU. This blog post documents the process of how I
+resolved this issue.
+
+
I run the test code:
+
1 2 3 4 5 6 7 8 9 10 11 12
IMPORT PYTHON3 AS PYTHON;
STRING GPUtest() := EMBED(Python) import tensorflow as tf if tf.test.is_gpu_available(): return 'available' else: return 'unavailable' ENDEMBED;
res := GPUtest(); OUTPUT(res, NAMED('res'))
+
Reason : In my previous environment setup, I installed TensorFlow and
+CUDA in root mode, but I only configured the environment information in
+the .bashrc file of the current user.
+However, HPCC Systems creates a new user named "hpcc" and uses the
+environment variables from that user. As a result, in the "hpcc" user,
+the LD_LIBRARY_PATH and other environment variables were
+not present, causing CUDA and GPU recognition to fail.
+
I first modify the password of hpcc:
+
1
sudo passwd hpcc
+
In Ubuntu, there are two methods to switch to another user:
+
+
su user: The su command requires you to enter the password of the
+target user. You must know the password of the target user and have root
+user privileges. When switching to the target user using the su command,
+the target user's complete environment variables are not loaded. It only
+switches to the target user's identity and inherits the current user's
+environment variables.
+
sudo -i -u user: When using the sudo -i -u user command to switch to
+the target user, the target user's complete environment variables are
+loaded. It switches you to the target user's identity and loads the
+target user's environment settings.
+
+
According to the environment variable setting rules in Linux, I have
+added the previously set environment variables into
+/etc/profile:
Using sudo -i -u hpcc to enter the hpcc user and typing
+env, I found that the previous settings have taken effect.
+Please note that at this point, you should not use su hpcc
+to enter the user, as it would load incorrect environment variables.
+
However, even after making these settings, it appears that HPCC still
+cannot properly recognize the GPU.
+
So I tried running the code in HPCC, retrieving the environment
+variables, and restarting HPCC Systems. After that, I ran:
+
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
IMPORT PYTHON3 AS PYTHON;
STRING GPUtest() := EMBED(Python) import tensorflow as tf import subprocess
I found that the environment variable LD_LIBRARY_PATH
+was not loaded correctly. Could it be because the
+CUDNN_PATH is using a Python statement that was not
+executed correctly? To test this, I changed CUDNN_PATH
+to
In the ECL language, operations such as OUTPUT cannot be performed in
+the definition, complicating our view of intermediate variables. This
+blog introduces debugging tips for several ECL programming language.
+
+
Use WHEN to output
+intermediate variables
+
Unlike programming languages such as C++ and Python, in ECL, the
+OUTPUT operation cannot be performed in the function
+definition. It will prompt
+
+
WHEN must be used to associate an action with a definition
+
+
We can use WHEN to output intermediate variables. For
+example:
+
1 2 3 4 5 6 7 8 9 10 11 12 13 14
//a FUNCTION with side-effect Action namesTable := FUNCTION namesRecord := RECORD STRING20 surname; STRING10 forename; INTEGER2 age := 25; END; o := OUTPUT('namesTable used by user <x>'); ds := DATASET([{'x','y',22}],namesRecord); RETURN WHEN(ds,O); END;
z := namesTable(); OUTPUT(z);
+
For multiple outputs, SEQUENTIAL can be used. For
+example:
+
1 2 3 4 5 6 7 8 9 10 11 12 13
//a FUNCTION with side-effect Action namesTable := FUNCTION namesRecord := RECORD STRING20 surname; STRING10 forename; INTEGER2 age := 25; END; ds := DATASET([{'x','y',22}],namesRecord); RETURN WHEN(ds,SEQUENTIAL(OUTPUT('namesTable used by user <x>'), OUTPUT('test2'))); END;
z := namesTable(); OUTPUT(z);
+
Logging with Std.System.Log
+
By using Std.System.Log, some logs can be recorded. For
+example:
+
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
IMPORT Std.System.Log AS Syslog;
//a FUNCTION with side-effect Action namesTable := FUNCTION namesRecord := RECORD STRING20 surname; STRING10 forename; INTEGER2 age := 25; END; ds := DATASET([{'x','y',22}],namesRecord); RETURN WHEN(ds, Syslog.addWorkunitInformation('test1234')); END;
z := namesTable(); OUTPUT(z);
+
In Helpers of ECL Watch, in EclAgentLog, you can find
+the corresponding log.
+
+
+image-20230619044834916
+
+
In
+embedded languages, use assert to check intermediate variables
+
When we need to debug some embedded Python codes, we cannot use the
+print function to print intermediate variables and can only
+output them externally by means of an
+assert False, 'message'. For example, if I want to view the
+summary of the model I defined, I can use the following:
When upgrading the HPCC GNN bundle from TensorFlow1.x to
+TensorFlow2.x, I encountered an error of
+numpy() is only available when eager execution is enabled.
+This blog records the cause and solution of the problem.
+
+
Problem Description
+
When writing the test code, I encountered a strange bug.
+
I defined a TensorFlow model:
+
1 2 3 4 5 6 7 8 9 10 11
import tensorflow as tf from tensorflow import keras from tensorflow.keras import models,layers
numpy() is only available when eager execution is enabled.
+
+
Cause and solution
+
After many investigations, it is found that the result of
+tf.executing_eagerly() is False. This is strange. I am
+using TensorFlow2.12, and the eager mode is enabled by
+default. Why does this error still occur?
+
The reason is that I upgraded from the compatibility mode of
+TensorFlow2. That is, the original TensorFlow mode is:
+
1 2
import tensorflow.compat.v1 as tf # V2.x tf.disable_v2_behavior()
+
Before the HPCC System Platform is restarted,
+tf.disable_v2_behavior() will always take effect, so the
+value of tf.executing_eagerly() is False.
Learn the basics of ECL, the powerful programming language built for
+big data analytics.
+
+
File type
+
There are two kinds of operations in ECL, definition (Definitions)
+and execution (Actions). After using EXPORT to execute the definition
+operation, it will no longer be able to perform operations in this
+file.
+
Similarly, ECL has two kinds of files. Their suffixes are
+.ecl. Definitions file and build execution (BWR, Builder
+Window Runnable) file. The difference is
+
+
The definition file always contains EXPORT and SHARED definitions
+and contains no execution operations. Therefore, the file cannot be
+executed through Submit.
+
The BWR file contains at least one execution operation and no EXPORT
+and SHARED definition operations.
+
+
Define variables
+
Variable names cannot have spaces and end with a semicolon. Format
+for defining variables:
Among them, n indicates that this integer occupies number of
+bytes, which can be 1~8. The default is 8.
+
IntType describes whether the high bit of the number is at
+the low address or the low bit is at the low address. Can take either
+BIG_ENDIAN or LITTLE_ENDIAN. Default is
+LITTLE_ENDIAN.
+
UNSIGNED, used to describe whether it is signed or not, the default
+is signed.
+
REAL[** n **] represents a floating point
+number, n can be 4 (7 significant figures) or 8 (15 significant
+figures)
+
Set
+
All elements must be of the same type.
+
Example:
+
1 2 3
SetInts := [1,2,3,4,5]; // an INTEGER set with 5 elements SetExp := [1,2+3,45,SomeIntegerDefinition,7*3]; SetSomeField := SET(SomeFile, SomeField);
+
SET can be accessed by subscript, subscript starts from
+1
+
1 2 3
MySet := [5,4,3,2,1]; ReverseNum := MySet[2]; //indexing to MySet's element number 2, //so ReverseNum contains the value 4
+
Strings are treated as SET with multiple 1-character elements, so
+they can also be accessed by subscript
+
1 2
MyString := 'ABCDE'; MySubString := MyString[2]; // MySubString is 'B'
+
Strings support range access:
+
1 2 3 4
MyString := 'ABCDE'; MySubString1 := MyString[2..4]; // MySubString1 is 'BCD' MySubString2 := MyString[ ..4]; // MySubString2 is 'ABCD' MySubString3 := MyString[2.. ]; // MySubString3 is 'BCDE'
+
The data type in the Set can be specified:
+
1 2 3 4 5 6
SET OF INTEGER1 SetValues := [5,10,15,20];
IsInSetFunction(SET OF INTEGER1 x=SetValues,y) := y IN x;
How to use: EXPORT[ VIRTUAL ]
+definition, only one EXPORT Module is allowed
+in each file, and the name of this Module must be the
+same as the file name.
+
VIRTUAL is optional. If specified, the definition is only valid
+within the Module. Allows usage as Module.Definition from other
+files.
+
EXPORT allows nesting. If you want to access a value in Module from
+another file, this value must also be modified by EXPORT.
Enums can be useful when you want to represent a limited set of
+possible values for a variable or a parameter. For example:
+
1 2 3 4
Color := ENUM(RED=1, GREEN=2, BLUE=3); myColor := Color.RED;
OUTPUT(myColor); # 1
+
RECORD
+
A RECORD in ECL represents the structure or format of a
+dataset. It is similar to the concept of a "table" in a SQL database,
+where each field in the record is similar to a column in the table. It
+defines the data types and names of fields.
+
For example:
+
1 2 3 4 5 6 7 8 9 10 11
ChildRec := RECORD UNSIGNED4 person_id; STRING20 per_surname; STRING20 per_forename; END;
rec := RECORD STRING20 name; INTEGER4 age; BOOLEAN isEmployed; END;
+
usually used with DATASET.
+
DATASET
+
It represents a set of data. A dataset is a group of records with the
+same record layout. A record layout is defined using the
+RECORD structure, which contains a set of fields, each with
+a name and a type.
For example, to output 111 to the test panel, you can write:
+
1
OUTPUT(1111, NAMED('test'));
+
TABLE
+
Used to create a new dataset (Dataset). The TABLE
+function takes a set of records (each defined by a RECORD
+structure) and an optional filter condition, and returns a new
+dataset.
In PROJECT, the TRANSFORM operation is performed on each piece of
+data, and then a new data set is obtained, which is one-to-one in
+quantity. And NORMALIZE is to expand a single piece of data into
+multiple pieces of data. In some cases, you may have a field that
+contains repeated data, and you may wish to split each repetition into a
+separate record. In this case, you can use the NORMALIZE
+function. The NORMALIZE function is used by receiving a
+dataset and a TRANSFORM function and returning a new dataset. In the
+conversion function, you need to define how to split the original
+records into new records.
+
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Layout := RECORD STRING10 name; INTEGER4 times; END;
ds.times indicates the number of times to repeat, and
+the TRANSFORM function defines how to convert the original
+record into a new record. In the output dataset, John and
+Jane will appear 3 and 2 times, respectively. Note that in
+the NORMALIZE function, for the current data, use LEFT
+reference.
INTEGER4 MySquare(INTEGER4 val) := EMBED(Python) return val * val ENDEMBED;
OUTPUT(MySquare(5)); // 25
+
LOCAL
+
The LOCAL keyword is mainly used to limit the scope of
+data or functions, or to control how data is distributed in the
+cluster.
+
If a definition (for example, a dataset or function) is declared
+LOCAL, then this definition is only visible in the ECL
+statement in which it is declared. This is similar to local variables in
+other programming languages. For example:
+
1 2 3 4
ECLCopy codemyFunction := FUNCTION LOCAL myLocalValue := 5; // Only available in this function RETURN myLocalValue * 2; END;
+
Additionally, when the LOCAL keyword is used on a
+dataset, it indicates that the dataset should be computed locally on
+each node, rather than across the entire cluster. This can be useful for
+reducing network communication and speeding up calculations. For
+example, the following ECL statement creates a dataset that is computed
+locally on each node:
In this example, myRecordDef is a record definition that
+describes the structure of each record in the dataset. Each node will
+process a portion of this dataset, not the entire dataset.
+
ASSERT
+
It is often used to judge whether a function has obtained the
+expected result.
In HPCC Systems, pictures need to be uploaded to HPCC Systems before
+they can be accessed. This blog post documents how to upload an image to
+HPCC Systems
+
+
First upload the image to Landing Zones in ECL Watch:
+
+
+image-20230719123923331
+
+
After uploading, select this picture, and in Import, select BLOB:
+
+
+image-20230719123949833
+
+
Set up as follows:
+
+
+image-20230719124004497
+
+
Then use the following code to get the Hex content of this image:
+
1 2 3 4 5 6 7 8 9
imageRecord := RECORD STRING filename; DATA image; //first 4 bytes contain the length of the image data UNSIGNED8 RecPos{virtual(fileposition)}; END;
If you want to convert this Hex content into np array for subsequent
+processing, you can use the following code:
+
1 2 3 4 5 6 7 8 9 10
STRING hexToNparry(DATA byte_array):= EMBED(Python) from PIL import Image import numpy as np import io bytes_data = bytes(byte_array) image = Image.open(io.BytesIO(bytes_data)) I_array = np.array(image) # this is all you need return ''.join(str(i) for i in I_array.shape) # check the shape of this image, height * width* channel ENDEMBED; OUTPUT(hexToNparry(imageData[1].image), NAMED('npShape'));
The Node.js, Hexo and Next themes used in previous blogs are
+outdated, causing a lot of compatibility issues. So I reconfigured my
+blog and this post records my configuration and development process.
+
+
Install Hexo
+
+
Install Node.js, use the command node --version to
+view the installed version, the version I installed is 16.17.0.
+
Install Hexo using npm install -g hexo-cli.
+
Use the following commands to initialize and complete the most
+basic website building.
+
1 2 3
$ hexo init <BlogDir> $ cd <folder> $ npm install
+
+
My Hexo version is 6.3.0. For more detailed commands, please refer to
+This Doc.
+
Using Next theme
+
There are currently three versions of the hexo next theme: the earliest
+version, the second
+version, these two versions have been stopped maintenance, and the
+latest
+version is still being maintained. Please refer to this
+doc for the relationship between the three versions.
+
All subsequent configurations are based on the latest
+version.
+
+
Download and install the Next theme:
+
1 2
$ cd hexo-site $ git clone https://github.com/next-theme/hexo-theme-next themes/next
+
Set theme: next in
+Hexo site __config.yml to enable the Next theme.
+
Set scheme: Pisces in
+Next Theme __config.yml(You can use other scheme).
+
+
Use abbrlink
+
Hexo uses the Markdown file name as the link by default. If the file
+name has Chinese, Hexo need to escape the it. In addition, it may lead
+to a very long link . Modifying the article title causes the article
+title to not match the file name. Modifying the file name will cause the
+article link to be modified. None of these above is helpful for search
+engines to search your blog.
+
So I hash the filename and use the hash as the link to the article.
+In this way, the article link will never change, which is convenient for
+search engines to search.
+
Install hexo-abbrlink:
+
1
$ npm install hexo-abbrlink --save
+
In Hexo site __config.yml set:
+
1 2 3 4 5 6
permalink:/post/:abbrlink/
# abbrlink config abbrlink: alg:crc32# crc16(default) and crc32 rep:hex# dec(default) and hex
When using hexo-abbrlink, the address in
+the default Markdown syntax for adding images
+ is not converted,
+resulting in the image not being displayed. Referring to the previous
+hexo-asset-image plugin, I wrote a new plugin to solve this
+problem.
In this way, your blog's post structure will look like this:
+
1 2 3
YourBlogDoc ├── image.jpg YourBlogDoc.md
+
Insert image image.jpg by
+ or
+.
+Then use hexo clean, hexo g,
+hexo server to preview.
+
Blog appearance modification
+
Modify hyperlink color
+
In
+<BlogDir>/themes/next/source/css/_common/components/post/post-body.styl,
+add:
There are two plugins that support Algolia search:
+hexo-algolia and hexo-algoliasearch, the
+former can only search article titles, while the latter can search
+article content. I use the latter for the English site.
Record Application ID,
+Search-Only API Key, Admin Key
+
Install
+npm install hexo-algoliasearch --save
+
In Hexo site __config.yml, set:
+
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
algolia: appId:"Your App ID" apiKey:"Your Search Only API Key" adminApiKey:"40321c7c207e7f73b63a19aa24c4761b" chunkSize:5000 indexName:"Your Index Name" fields: -content:strip:truncate,0,500 -excerpt:strip -gallery -permalink -photos -slug -tags -title
+
In Next Theme __config.yml, set:
+
1 2 3 4
algolia_search: enable:true hits: per_page:10
+
+
I think this way will be better: If it is found, it will display the
+found results, instead of showing that XXX is found within XXX
+milliseconds, like Google, Baidu does. If it is not found, it will
+display "not found".
+
In
+<BlogDir>/themes/next/source/js/third-party/search/algolia-search.js,
+delete:
The comment system that comes with the Next theme is valine, which
+is a comment system without a backend, and it is said that there are
+some information security issues. So I use waline. First we need to follow
+waline's configuration document to set the required components, and also
+install the plugin Hexo NexT
+Waline:
# Word limit, no limit when setting to 0 wordLimit: 0
# Whether enable login, can choose from 'enable', 'disable' and 'force' login: enable
# comment per page pageSize: 10
+
If you don't want to enable comments on a page, please set the
+follows in the header of the article:
+
1
comments: false
+
The default comment description is Waline, which is very
+strange. In <BlodDir>\themes\next\languages\en.yml,
+add it under post:
+
1 2 3
post: comments: waline:Comments
+
then you modify Waline to Comments.
+
Set up FTP synchronization
+
Currently I use a virtual host to host my blog and need to use FTP to
+upload files. According to ChaosTong/hexo-deployer-ftpsync,
+I simply modified it to accommodate the latest node.js.
Use FTP to upload a file named .htaccess to the root
+directory of the site, the content is
+
1 2 3
RewriteEngine On RewriteCond %{HTTP:KERSSL} !on RewriteRule .* https://www.YourDomin.com/$1 [R=301,L]
+
Since the contents of the public folder will change
+after hexo d, it has to be copied, pasted and uploaded
+again each time. I use the alias command (function in Windows
+PowerShell) to execute multiple commands at a time, as follows:
+
1
function blogd {cd D:\Blog ; hexo clean ; hexo g ; hexo algolia; Copy-Item -Path .\.htaccess -Destination .\public\; hexo d}
+
Please note that the setting method may be different under different
+virtual hosts
+
Conclusion
+
The results of the above tests in my local are 0 Error, 0 Warning
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/post/96495af0/image-20230526213027939-1685786902862-1.png b/post/96495af0/image-20230526213027939-1685786902862-1.png
new file mode 100644
index 0000000..f0a2ab5
Binary files /dev/null and b/post/96495af0/image-20230526213027939-1685786902862-1.png differ
diff --git a/post/96495af0/image-20230526213027939.png b/post/96495af0/image-20230526213027939.png
new file mode 100644
index 0000000..f0a2ab5
Binary files /dev/null and b/post/96495af0/image-20230526213027939.png differ
diff --git a/post/96495af0/image-20230526222727457-1685786902862-3.png b/post/96495af0/image-20230526222727457-1685786902862-3.png
new file mode 100644
index 0000000..81b0346
Binary files /dev/null and b/post/96495af0/image-20230526222727457-1685786902862-3.png differ
diff --git a/post/96495af0/image-20230526222727457.png b/post/96495af0/image-20230526222727457.png
new file mode 100644
index 0000000..81b0346
Binary files /dev/null and b/post/96495af0/image-20230526222727457.png differ
diff --git a/post/96495af0/image-20230526222802693-1685786902862-2.png b/post/96495af0/image-20230526222802693-1685786902862-2.png
new file mode 100644
index 0000000..0a1731c
Binary files /dev/null and b/post/96495af0/image-20230526222802693-1685786902862-2.png differ
diff --git a/post/96495af0/image-20230526222802693.png b/post/96495af0/image-20230526222802693.png
new file mode 100644
index 0000000..0a1731c
Binary files /dev/null and b/post/96495af0/image-20230526222802693.png differ
diff --git a/post/96495af0/index.html b/post/96495af0/index.html
new file mode 100644
index 0000000..49f09d7
--- /dev/null
+++ b/post/96495af0/index.html
@@ -0,0 +1,455 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Install HPCC Platforms and Setup ECL IDE | Boqiang's Blog
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
HPCC Systems is a platform
+purpose-built for high-speed data engineering. This article shows how to
+build HPCC Platform and set up ECL IDE.
+
+
Install HPCC Platforms
+
+
Download the installation package of HPCC Platforms corresponding
+to the operating system here.
+For example, if my system is Ubuntu-22.04 LTS, then download this
+file:
+
+
+image-20230526213027939
+
+
Run
+dpkg -i hpccsystems-platform-community_9.0.8-1jammy_amd64_withsymbols.deb.
+There may be some errors at this time. Ignore it, please.
+
Run sudo apt-get update
+
Run sudo apt-get install -f
+
Verify that the program installed successfully: Run
+cd /opt/HPCCSystems. The installation is successful if no
+errors have occurred and the directory is entered successfully.
+
Check plugins: Run cd /opt/HPCCSystems/plugins,
+ls -lart
+
Check current service status: Run
+sudo /etc/init.d/hpcc-init status
+
Start HPCC service: Run
+sudo /etc/init.d/hpcc-init start
+
Check the IP address: Run ip addr show. You can
+access IP addr: 8010 to login ECL Watch
+
+
Install ECL IDE
+
ECL IDE is an IDE specially designed for ECL programming language,
+related documents can be found here.
+
First, download ECL IDE according to the platform, and find ECL IDE
+and Client Tools here.
+
When you log in for the first time, the preference interface will pop
+up, enter the IP obtained in Install HPCC Platforms in the IP
+address, click Advanced, and leave Attribute Server blank. Other options
+can be set according to your preferences.
+
In the ECL Folder of the Compiler tab, add your own code
+directory.
+
HelloWorld
+
In the lower right Repository window, in My Files, right click and
+Insert Folder, then in the newly created folder, right click and Insert
+File.
+
+
+image-20230526222727457
+
+
In the newly created file, enter:
+
1
OUTPUT('Hello World!');
+
Click Submit in the upper right corner, and you can see the result in
+the result tab.
+
+
+image-20230526222802693
+
+
Install the HPCC Systems
+GNN bundle
+
The HPCC Systems machine learning library can be found here.
+Among them, ML_Core is the
+dependency of all other HPCC Systems machine learning packages.
+
Before installing the GNN bundle, you need to install TensorFlow
+first.
+
My HPCC Platform is on WSL2-Ubuntu22.04, and my ECL IDE is on
+Windows, so I need to install GNN Bundle in Windows.
During my internship at HPCC Systems in 2023, I had the incredible
+opportunity to work on a dynamic project that pushed the boundaries of
+my technical skills and knowledge. In this post, I will summarize the
+key aspects and achievements of my internship project, highlighting the
+challenges I faced, the solutions I implemented, and the valuable
+lessons I learned along the way.
+
+
Background and Motivation
+
+
HPCC Systems: The HPCC
+Systems platform is a remarkably powerful system for solving big data
+challenges.
The GNN bundle was originally developed against the TensorFlow1.5
+interface. We currently support TensorFlow2.0 using a TensorFlow
+compatibility mode. We want to achieve the following goals in this
+project:
+
+
Adapt GNN to directly use the TensorFlow 2.0
+interface: Our primary objective is to migrate the GNN bundle
+to directly integrate with the TensorFlow 2.0 interface, leveraging its
+improved features and capabilities for enhanced performance and
+efficiency.
+
Provide effective support for GPUs: We intend to
+ensure effective support for GPUs, allowing users to harness the power
+of graphical processing units for accelerated neural network
+computations and faster training times.
+
Full support for pre-trained models: The upgraded
+GNN bundle will be designed to fully support and accommodate the latest
+and most advanced neural network models available. This will enable
+users to leverage state-of-the-art architectures and achieve better
+results for their specific tasks.
+
Save/Load trained model for later reuse: 1.We want
+users to be able to save/load their trained models for later use.
+
+
+
Technical details
+
Upgrade functions to
+TensorFlow2.x
+
We upgraded some functions to native TensorFlow2.x without
+changing the API:
+
+
DefineModel: Define a Keras / Tensorflow model
+using Keras sytax.
+
DefineFuncModel: Construct a Keras Functional
+Model.
+
+
These functions play a crucial role in defining the neural network
+architecture. In the upgraded GNN bundle, both functions have been
+enhanced to utilize the TensorFlow 2.x API. DefineModel
+follows a sequential approach, where the layers are stacked in a strict
+order. On the other hand, DefineFuncModel offers a more
+flexible definition, allowing users to specify not only the shape of
+each layer but also the connections between different layers. These
+upgraded functions now support loading state-of-the-art models, which
+provide users with more advanced and powerful neural network
+architectures for their tasks.
+
+
Fit: Train the model using training data.
+
+
The Fit function is used for training the neural
+network. It allows users to set parameters such as batch size and epoch
+number, providing greater control over the training process. With the
+upgrade to TensorFlow 2.x, the training process becomes more efficient
+and streamlined.
+
+
EvaluateMod: Evaluate the trained model using
+some test metrics.
+
Predict: Predict the results using the trained
+model.
+
+
After training the model using the Fit function, the
+EvaluateMod function helps users assess the model's
+performance by providing metrics on how well it has been trained. The
+Predict function allows users to apply the trained neural
+network for tasks like recognition and classification.
+
+
GetWeights: Get the weights currently associated
+with the model.
+
SetWeights: Set the weights of the model from
+the results of GetWeights.
+
+
These functions are essential for saving and loading models. Users
+can use GetWeights to obtain the model's weights, and
+SetWeights to set the model's weights, enabling easy model
+persistence. This way, trained models can be saved and loaded,
+facilitating model continuation for further training or deployment to
+customers.
+
Create new functions
+
To enhance user experience and provide greater convenience, we have
+introduced several new functions in the GNN bundle.
+
+
GetModel: Get the structure and weights of the
+model. This function allows users to access comprehensive details of the
+model, including its structure and weights. By using
+GetModel, users can gain a thorough understanding of the
+neural network architecture.
+
SetModel: Create a Keras model from previously
+saved DATASET.
+
+
Additionally, we have incorporated some developer-oriented functions
+tailored for debugging purposes:
+
+
GetSummary: Get the summary of the Keras model. The
+GetSummary function offers developers access to the
+underlying TensorFlow information. This allows developers to obtain
+essential insights into the inner workings of the GNN bundle, aiding
+them in identifying and resolving any issues that may arise.
+
IsGPUAvailable: Tests whether the GPU is available.
+The IsGPUAvailable function serves as a practical tool for
+developers to determine whether a GPU (Graphics Processing Unit) is
+available for use. This information is crucial when optimizing the GNN
+bundle's performance and ensuring efficient hardware utilization.
+
+
By providing these user-friendly and developer-oriented functions, we
+strive to make the GNN bundle more accessible, efficient, and robust,
+catering to the needs of both users and developers alike.
+
Support Pre-trained Model
+
The recently upgraded GNN bundle introduces support for numerous
+state-of-the-art models, which have been defined and trained to serve
+specific tasks. With over 70 pre-trained
+models available, users can readily utilize them for their projects.
+Additionally, these pre-trained models can act as the foundation for
+user-specific models, allowing customization by modifying the input and
+output and expanding the neural network, a technique commonly known as
+transfer learning.
+
One notable example among the supported pre-trained models is
+ResNet50. ResNet50 is a deep convolutional neural network comprising 50
+layers. By loading the pre-trained version of this network, which was
+trained on an extensive dataset of more than a million images from the
+ImageNet database, users gain access to a powerful image classifier. The
+pre-trained neural network is capable of categorizing images into 1000
+object categories, including items such as keyboards, mice, pencils, as
+well as various animals. As a result of its extensive training, this
+neural network has acquired a vast knowledge of feature representations
+for a diverse range of images.
+
+
+ResNet50 structure
+
+
Transfer Learning
+
The enhanced GNN bundle offers the ability to load numerous
+pre-trained models, opening up possibilities for various impressive
+applications. For instance, consider ResNet-50, a model commonly used to
+classify 1000 different objects. However, if you wish to employ it
+specifically for classifying vegetables or woolen cloth, you can achieve
+this by leveraging the pre-trained ResNet-50 as a starting point for
+training. This involves modifying the input and output components of the
+model while preserving the existing ResNet-50 weights, rather than
+initializing the weights randomly. Such an approach allows you to tailor
+the model to your specific classification tasks effectively.
+
+
Also, you can use the pre-trained model as a part of your neural
+network architecture, you can add layers to it.
+
Performance Testing
+
TL; DR:
+
+
When using the same GPU, the upgraded GNN outperforms the previous
+version with TensorFlow 1.x, achieving a remarkable 1.25 times
+performance improvement.
+
The upgraded GNN takes advantage of GPU acceleration, resulting in a
+substantial performance boost. When compared to using only CPU, the
+performance improvement is an impressive 3.71 times.
+
+
FAQ
+
Q: Does it support video processing?
+
A: Yes!
+
Q: Did it change the existing API.
+
A: No. This means that the previous program can still be used without
+modification.
+
Conclusion
+
A high point during my internship was successfully contributing to a
+complex coding task, receiving recognition from the team for my
+innovative approach. This achievement bolstered my confidence and
+affirmed my passion for software development.
+
While on this journey, I encountered a low point when I faced an
+unforeseen roadblock that temporarily hindered project progress. For
+instance, when I come across unfamiliar grammar challenges within the
+realm of the ECL programming language, I often find myself facing
+significant obstacles. I am truly grateful for the assistance provided
+by Lili and Roger, as their guidance proved instrumental in successfully
+navigating through these complexities. This setback taught me resilience
+and the importance of seeking guidance when confronted with unfamiliar
+challenges.
+
The main takeaways from this internship include a heightened
+appreciation for collaborative teamwork. These lessons will undoubtedly
+shape my future endeavors in the field of technology.
+
References
+
The following article records the problems and solutions I
+encountered in this project.
ECL is not a procedural language, that is, the order in which
+operations are performed is not necessarily the order given by the code,
+which can cause trouble in calculating the execution time of operations.
+This blog describes how to calculate the execution time of certain
+operations for users to evaluate performance.
+
+
In ECL, you can use the following code to get the current system
+time:
+
1 2
IMPORT STD; STD.Date.CurrentTime(TRUE);
+
Use ORDERED or SEQUENCIAL to force certain
+operations to be executed in order:
+
1 2 3 4
ORDERED([OUTPUT(STD.Date.CurrentTime(TRUE), NAMED('startTime')), operations....., OUTPUT(STD.Date.CurrentTime(TRUE), NAMED('endTime')), Some operations]);
Install directly in the system without using Anaconda.
+
+
Install CUDA
+
First, find the corresponding CUDA version according to the
+TensorFlow you want to install, and install the corresponding CUDA. For
+example, I want to install TensorFlow version 2.12, which uses CUDA
+11.8, so I download CUDA from here.
+I use Linux-X86-64-WSL-Ubuntu-2.0, and I choose the installation package
+of runfile(local). This is the most convenient and recommended!
+
In the prompt box that pops up, select Install:
+
+
+image-20230602232528589
+
+
After installation, you need to manually add environment variables in
+~/.bashrc, add:
Tensorflow should be installed using su so that all users can see it,
+and must be installed using the same version of Python3 as is embedded
+in the HPCC Systems platform.
Install TensorFlow using pip:
+pip install tensorflow==2.12.*
+
Check whether the installation is successful: exit root mode and
+run
+python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
+
Or you can use this to show the name of GPU(s):
+
1 2 3 4
import tensorflow as tf gpu_device_name = tf.test.gpu_device_name() print(gpu_device_name)
Adjust the number of Nodes in HPCC Systems under WSL2.
+
+
I am using WSL2 - Ubuntu 22.04, and WSL2 uses all the CPUs in the
+current Windows by default, so in this case, to modify the number of
+nodes in HPCC Systems, you only need to set
+slavesPerNode="The number of nodes you want to set" in
+/etc/HPCCSystems/environment.xml.
+
+
+
+
+
+
+
+
+
+