Long time to compile complicated processor

[jaypiper]

Hi. We encounter scalability issue when we try to use essent to compile the MinimalConfig of XiangShan (a complicated open-source high performance RISC-V processor). We use a low firrtl file as the input of essent. But the size of the low firrtl file is 140MB, and it contains about 1.5 million lines. This low firrtl file is attached in the zip file at the end of this post. It takes about 13 hours for essent to generate a header file on a server with i9-9900k and 64GB memory.

To profile essent, we use option -ell Debug -fll Debug and save the log into essent.log, which is also attached. We find that the small step took about 11 hours.

$ grep "took" essent.log 
[mffc] took: 29577
[single] took: 3210
[siblings] took: 491903
[small] took: 40090142
[down] took: 5801425
[small2] took: 1273610

Besides, the size of the generated header file is 688MB, and it contains about 5.5 million lines. This huge header file is also attached. It takes days for g++ to compile this huge header file.

Note that the MinimalConfig of XiangShan is dual-issue. But the DefaultConfig of XiangShan is six-issue and the design is much more complicated. Before we try DefaultConfig with essent, we think that we must address the scalability issue of essent.

Below are some suggestions.

1. From firrtl to .h: We are not familiar with the internal of essent. We hope that the log file attached can help you to analyze the performance.

2. From .h to elf:

   • To enable parallel compilation of g++, we write a python script to split the huge header file into multiple source files. We only keep the class definition in the header file, and move all constructors and functions into source files. Currently there are no more than 100 functions in a source file. The python script splits the huge header file into about 300 source files. We think that essent can support such file splitting feature, with the file granularity configurable by user.
   • But it still takes about 2.5 hours for g++ to compile a single split source file with size 2MB. After profiling g++, we find that g++ spends 98% of the time to parse struct body, among of which 80% of the time is cost for name lookup. This is because there are 0.4 million member variables in the class definition of the top module. However, users do not care about the name of these variables. One suggestion is to merge all variables into a big vector. Then essent allocates and manipulates each individual signal in this big vector. To keep readability, essent can generate comments to record the original semantics of signal allocation and manipulation. For example,

// defined in the struct
UInt<1000000> all_signals;
/* 2: cache_real_hit
  4: valid
  5: hit
  ...
*/


// in the function body
all_signals(2) = all_signals(4) & all_signals(5); // cache_real_hit = valid & hit;

essent-XiangShan.zip

[jaypiper]

It takes less than 10 minutes for clang++ to compile splitted source files, and the cost is acceptable. But long time for essent to compile remains a problem.

[sbeamer]

Thank you for sharing this! Your profiling of the compile time is especially helpful!

We haven't had a chance to run essent on anything this big. Improving scalability is definitely interesting for our future work.