Memory usage when generating large site

[leesei]

#1735
#1529

I'll try to recreate the environment first.

[leesei]

Generate stub files with leesei/hexo-genstubs.

With genstubs.js -N 5000 -P 100:

229M    source/_posts
128M    stub-posts.zip

UPDATE: genstubs.js -N 500 -P 100 is enough to trigger OOM.

This is enough to generate OOM on my Arch Linux with 8G RAM.
The Hexo site is generated by hexo init.

[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]

$ hexo generate

<--- Last few GCs --->

    7958 ms: Scavenge 1408.7 (1457.0) -> 1408.7 (1457.0) MB, 17.1 / 0 ms (+ 33.0 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
    8984 ms: Mark-sweep 1408.7 (1457.0) -> 1405.9 (1454.0) MB, 1026.3 / 0 ms (+ 967.1 ms in 2405 steps since start of marking, biggest step 33.0 ms) [last resort gc].
   10009 ms: Mark-sweep 1405.9 (1454.0) -> 1405.8 (1457.0) MB, 1024.3 / 0 ms [last resort gc].

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

[tommy351]

Thanks for the test. 💯 👍

[xieguigang]

My site have over 8000 pages, and encounter this problem too, where launch hexo with server command, then hexo out of memory, but if launched with generate command, it give me this problem:

$ hexo generate
INFO  Start processing
FATAL Something's wrong. Maybe you can find the solution here: http://hexo.io/docs/troubleshooting.html
Template render error: (unknown path) [Line 4, Column 929]
  expected variable end
    at Object.exports.prettifyError (D:\gcmodeller-hexo\docs\node_modules\nunjucks\src\lib.js:34:15)
    at Obj.extend.render (D:\gcmodeller-hexo\docs\node_modules\nunjucks\src\environment.js:468:27)
    at Obj.extend.renderString (D:\gcmodeller-hexo\docs\node_modules\nunjucks\src\environment.js:326:21)
    at D:\gcmodeller-hexo\docs\node_modules\hexo\lib\extend\tag.js:66:9
    at Promise._execute (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\debuggability.js:272:9)
    at Promise._resolveFromExecutor (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\promise.js:473:18)
    at new Promise (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\promise.js:77:14)
    at Tag.render (D:\gcmodeller-hexo\docs\node_modules\hexo\lib\extend\tag.js:64:10)
    at Object.tagFilter [as onRenderEnd] (D:\gcmodeller-hexo\docs\node_modules\hexo\lib\hexo\post.js:253:16)
    at D:\gcmodeller-hexo\docs\node_modules\hexo\lib\hexo\render.js:63:19
    at tryCatcher (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\util.js:16:23)
    at Promise._settlePromiseFromHandler (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\promise.js:502:31)
    at Promise._settlePromise (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\promise.js:559:18)
    at Promise._settlePromise0 (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\promise.js:604:10)
    at Promise._settlePromises (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\promise.js:683:18)
    at Async._drainQueue (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\async.js:138:16)
    at Async._drainQueues (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\async.js:148:10)
    at Immediate.Async.drainQueues [as _onImmediate] (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\async.js:17:14)
    at tryOnImmediate (timers.js:543:15)
    at processImmediate [as _immediateCallback] (timers.js:523:5)
FATAL (unknown path) [Line 4, Column 929]
  expected variable end
Template render error: (unknown path) [Line 4, Column 929]
  expected variable end
    at Object.exports.prettifyError (D:\gcmodeller-hexo\docs\node_modules\nunjucks\src\lib.js:34:15)
    at Obj.extend.render (D:\gcmodeller-hexo\docs\node_modules\nunjucks\src\environment.js:468:27)
    at Obj.extend.renderString (D:\gcmodeller-hexo\docs\node_modules\nunjucks\src\environment.js:326:21)
    at D:\gcmodeller-hexo\docs\node_modules\hexo\lib\extend\tag.js:66:9
    at Promise._execute (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\debuggability.js:272:9)
    at Promise._resolveFromExecutor (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\promise.js:473:18)
    at new Promise (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\promise.js:77:14)
    at Tag.render (D:\gcmodeller-hexo\docs\node_modules\hexo\lib\extend\tag.js:64:10)
    at Object.tagFilter [as onRenderEnd] (D:\gcmodeller-hexo\docs\node_modules\hexo\lib\hexo\post.js:253:16)
    at D:\gcmodeller-hexo\docs\node_modules\hexo\lib\hexo\render.js:63:19
    at tryCatcher (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\util.js:16:23)
    at Promise._settlePromiseFromHandler (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\promise.js:502:31)
    at Promise._settlePromise (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\promise.js:559:18)
    at Promise._settlePromise0 (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\promise.js:604:10)
    at Promise._settlePromises (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\promise.js:683:18)
    at Async._drainQueue (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\async.js:138:16)
    at Async._drainQueues (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\async.js:148:10)
    at Immediate.Async.drainQueues [as _onImmediate] (D:\gcmodeller-hexo\docs\node_modules\bluebird\js\release\async.js:17:14)
    at tryOnImmediate (timers.js:543:15)
    at processImmediate [as _immediateCallback] (timers.js:523:5)

[leesei]

@xieguigang Refer to #1771 (comment) to help you debug "Template render error".
You probably have an invalid tag. You are welcome to post your test case there.

[stevenjoezhang]

Closing due to #3756