javascript - Are promises and closures consuming all of my memory? -

javascript - Are promises and closures consuming all of my memory? -

i working on project using node.js controlling scheme relatively big scale machine learning images. running out of memory pretty while trying though trying optimize usage much possible , info should not take excessive amount of space. code relies heavily on promises , anonymous functions manage pipeline , wondering if that's why i'm seeing crazy high usage on test case.

just context using mnist dataset testing can found here. training set consists of 60,000 20x20 images. these extracting overfeat features, description of can found here. boils downwards 4,096 element array each image, 60,000 of them. caching image , feature info in redis.

a quick computation tells me total feature set here should 4096 * 8 * 60000 = 1966080000 bytes or appx 1.82gb of memory assuming each element of array 64bit javascript number. images should take little amount of space , not storing them in memory. when run code seeing more 8-10gb of memory used after extraction/loading. when trying more operations on info (like print out json file can create sure extraction worked right) consume 16gb of available memory on computer, crashing node script.

so general question is: why seeing such high memory usage? because of heavy utilize of promises/closures? can refactor code utilize less memory , allow more variables garbage collected?

the code available here review. aware little rough far organization goes.

your code uses "promise" library fair memory hoggy , not built raw performance. if switch bluebird promises can considerably more items in ram drastically cut down memory usage.

here benchmark results doxbee-sequential:

results 10000 parallel executions, 1 ms per i/o op file time(ms) memory(mb) promises-bluebird.js 280 26.64 promises-then-promise.js 1775 134.73

and under bench parallel (--p 25):

file time(ms) memory(mb) promises-bluebird.js 483 63.32 promises-then-promise.js 2553 338.36

you can see the total benchmark here.

javascript node.js memory-management closures promise