High memory use decoding JPEG2000 with cornerstone

Some users have reported higher than expected memory use in cornerstone when decoding JPEG2000.  This is a known issue and unfortunately there is no quick fix for it.  The underlying cause is that cornerstone utilizes OpenJPEG (an open source C++ library) to do the decoding.  Since the browser cannot run C++ directly, we use EMSCRIPTEM to cross compile OpenJPEG into Javascript.  For those that know C++ and Javascript, you will immediately wonder how memory management is handled (since Javascript is garbage collected and C++ is not).  The way EMSCRIPTEM handles memory management is by allocating a contiguous byte array in javascript that represents the "heap" or address space exposed to the C++ code.

EMSCRIPTEM has two modes for managing the heap - dynamic and fixed.  With dynamic, the heap size grows dynamically based on the needs of the application.  With static, the heap is preallocated.  Dynamic is good when you have no idea how much memory you will need, but this comes with a cost of lower performance.  Static is good when you know how much memory you will need and want to maximize performance, but the cost is that it will fail to decode if you need more memory than you allocated.

It turns out that OpenJPEG requires a lot of memory to decode JPEG2000 images.  The bigger the image, the more memory is required.  It also turns out that JPEG2000 decoding is slower than desired with fixed heap mode and unusably slow with using dynamic heap mode.  The default codec for cornerstone is to use the fixed memory mode with a 60MB heap.  This size was selected by testing out the highest resolution image I could find and making sure it decode properly.  The actual in browser memory allocated for the OpenJPEG codec is actually much larger than 60 MB - the code takes some memory and EMSCRIPTEM has some overhead itself.

So lets calculate how much memory is required for a single 2048x2560 x 16 bit XRay encoded with JPEG2000.  Lets assume the image is lossless compressed and gets a 3:1 compression ratio.  The uncompressed image would be 10MB, the compressed image would be 3.33 MB.  Here is what happens in memory:

cornerstoneWADOImageLoader:
  DICOM P10 instance - 3.33 MB

OpenJPEG Codec:
  fixed 60 MB heap + code- ~70 MB? (probably higher, hard to tell) *for each web worker*

cornerstone:
  uncompressed version of image: 10 MB (stored in cornerstone's image cache)
  canvas RGBA back buffer: 20 MB (could be stored in GPU, not sure)

Each time you load a DICOM P10 image, it gets cached as well as the uncompressed version of the image.  The OpenJPEG codec should have a fixed size overhead - but I haven't measured exactly what that is yet.  This fixed size is multiplied by the number of web workers you launch.  The number depends on the browser, but for chrome, it is the number of cores in the system so this can add up quickly.  The more images you decode, the more memory is required.  Loading a multi-frame instance will result in a lot of memory use, especially if they are high resolution images like digital breast tomosynthesis.

A few other things to keep in mind:
1) Browsers are using more and more memory with each release.  You may be surprised to find that your javascript application is using up 1GB of RAM before it even loads its first image!
2) Javascript is garbage collected.  This means it can hang on to memory for a period of time even though its not needed any more.  This can cause your application to need more RAM that it should, especially when dealing with big memory allocations like we have in medical imaging
3) You have to be very careful to not hold on to references to image data that are not needed.  Not doing so will result in the garbage collector being unable to free the memory which will result in a browser crash.