Monday, October 16, 2017

High memory use decoding JPEG2000 with cornerstone

Some users have reported higher than expected memory use in cornerstone when decoding JPEG2000.  This is a known issue and unfortunately there is no quick fix for it.  The underlying cause is that cornerstone utilizes OpenJPEG (an open source C++ library) to do the decoding.  Since the browser cannot run C++ directly, we use EMSCRIPTEM to cross compile OpenJPEG into Javascript.  For those that know C++ and Javascript, you will immediately wonder how memory management is handled (since Javascript is garbage collected and C++ is not).  The way EMSCRIPTEM handles memory management is by allocating a contiguous byte array in javascript that represents the "heap" or address space exposed to the C++ code.

EMSCRIPTEM has two modes for managing the heap - dynamic and fixed.  With dynamic, the heap size grows dynamically based on the needs of the application.  With static, the heap is preallocated.  Dynamic is good when you have no idea how much memory you will need, but this comes with a cost of lower performance.  Static is good when you know how much memory you will need and want to maximize performance, but the cost is that it will fail to decode if you need more memory than you allocated.

It turns out that OpenJPEG requires a lot of memory to decode JPEG2000 images.  The bigger the image, the more memory is required.  It also turns out that JPEG2000 decoding is slower than desired with fixed heap mode and unusably slow with using dynamic heap mode.  The default codec for cornerstone is to use the fixed memory mode with a 60MB heap.  This size was selected by testing out the highest resolution image I could find and making sure it decode properly.  The actual in browser memory allocated for the OpenJPEG codec is actually much larger than 60 MB - the code takes some memory and EMSCRIPTEM has some overhead itself.

So lets calculate how much memory is required for a single 2048x2560 x 16 bit XRay encoded with JPEG2000.  Lets assume the image is lossless compressed and gets a 3:1 compression ratio.  The uncompressed image would be 10MB, the compressed image would be 3.33 MB.  Here is what happens in memory:

cornerstoneWADOImageLoader:
  DICOM P10 instance - 3.33 MB

OpenJPEG Codec:
  fixed 60 MB heap + code- ~70 MB? (probably higher, hard to tell) *for each web worker*

cornerstone:
  uncompressed version of image: 10 MB (stored in cornerstone's image cache)
  canvas RGBA back buffer: 20 MB (could be stored in GPU, not sure)

Each time you load a DICOM P10 image, it gets cached as well as the uncompressed version of the image.  The OpenJPEG codec should have a fixed size overhead - but I haven't measured exactly what that is yet.  This fixed size is multiplied by the number of web workers you launch.  The number depends on the browser, but for chrome, it is the number of cores in the system so this can add up quickly.  The more images you decode, the more memory is required.  Loading a multi-frame instance will result in a lot of memory use, especially if they are high resolution images like digital breast tomosynthesis.

A few other things to keep in mind:
1) Browsers are using more and more memory with each release.  You may be surprised to find that your javascript application is using up 1GB of RAM before it even loads its first image!
2) Javascript is garbage collected.  This means it can hang on to memory for a period of time even though its not needed any more.  This can cause your application to need more RAM that it should, especially when dealing with big memory allocations like we have in medical imaging
3) You have to be very careful to not hold on to references to image data that are not needed.  Not doing so will result in the garbage collector being unable to free the memory which will result in a browser crash.





Thursday, March 16, 2017

Team NucleusHealth's GitHub stats for our ONC Blockchain code-a-thon project

The NucleusHealth team created an immense amount of highly complex security code in a very short time period for the ONC Blockchain code-a-thon.  Will post more about the experience once I recover (I was up 36 hours straight).  Here are the raw stats:


Contribution stats (by author):

Matt Brinkman <emjaybee83@gmail.com>:
 insertions:    807 (19%)
 deletions:     453 (41%)
 files:         80 (6%)
 commits:       15 (15%)
 first commit:  Tue Mar 14 07:38:51 2017 -0500
 last commit:   Tue Mar 14 23:17:21 2017 -0500

Jason Klotzer <jklotzer@nucleushealth.io>:
 insertions:    491 (12%)
 deletions:     61 (5%)
 files:         819 (68%)
 commits:       12 (13%)
 first commit:  Tue Mar 14 10:17:11 2017 -0500
 last commit:   Wed Mar 15 02:01:19 2017 -0500

Durand R.:
 insertions:    1167 (28%)
 deletions:     233 (21%)
 files:         105 (9%)
 commits:       15 (17%)
 first commit:  Tue Mar 14 11:46:18 2017 -0400
 last commit:   Tue Mar 14 22:42:22 2017 -0400

Jason Nye:
 insertions:    520 (12%)
 deletions:     134 (12%)
 files:         78 (6%)
 commits:       22 (26%)
 first commit:  Tue Mar 14 10:20:10 2017 -0500
 last commit:   Wed Mar 15 00:11:46 2017 -0500

Chris Hafey:
 insertions:    1218 (29%)
 deletions:     212 (19%)
 files:         127 (11%)
 commits:       22 (26%)
 first commit:  Tue Mar 14 09:43:25 2017 -0400
 last commit:   Wed Mar 15 02:46:13 2017 -0400

total:
 insertions:    4203 (100%)
 deletions:     1093 (100%)
 files:         1209 (100%)
 commits:       86 (100%)

 851 files changed, 4203 insertions(+)

Tuesday, March 7, 2017

Applying blockchain to healthcare - part 10 (wallets)

In prior blog posts, we explored how cryptographic key pairs are used by ethereum.  You may recall that a private key is a random number, the public key is derived from the private key and the ethereum address is derived from the public key.  The derivations are one way - you cannot take an address and compute the public key and you cannot take the public key and compute the private key.

An ethereum account is generally referred to by its address, but also includes the corresponding private key and public key as those are needed to digitally sign and encrypt.  Here is an example of a private key and the corresponding ethereum address that is used in my private ethereum network:

private key: 0xf059416a2f6bb05d0770bbacb24a6430757aa7db5c15959838ed142b486df5b8

ethereum address: 
0x5DFE021F45f00Ae83B0aA963bE44A1310a782fCC

The private key, public key and address are all arrays of bytes with a hexadecimal string form.  As described in prior blog posts, ethereum's use of cryptographic key pairs changes the way identity and trust are handled.  While this approach brings many benefits, there are some drawbacks:

The private key is too long to memorize  

This is different from username/password authentication where users are able to remember the username and password for an account.  To address this issue, the private key needs to be stored digitally rather than remembered.

The private key cannot be recovered if lost

If you lose your private key, it is lost forever.  You cannot get the private key from the public key or the address as those are not reversible.  This is different than username/password authentication which has the ability to reset your password if you forget it or contact an administrator to reset your password.  To address this issue, users must take appropriate precautions to backup the private key.

The private key cannot be changed  

If someone obtains your private key, they have full control over it and it cannot be revoked.  This is different that username/password authentication where you can change your password if someone obtains it.  A private key must therefore be kept secret.  

Protecting private keys with wallets

To address the issues above a "wallet" is used to manage private keys.  A wallet is responsible for storing your private key in a safe manner while still providing services to digital sign and decrypt with it.  There are two main kinds of wallets - software wallets and hardware wallets.  

A software wallet is a software application you run on your computer which will store your private key encrypted with a user selected password in a file.  To use the private key, the user must unlock the private key by entering user selected password.  Once the account is unlocked, it can be used to digitally sign a message (e.g. an ethereum transaction).  This approach keeps the private key secret within the wallet.  Encrypting the private key with a user selected password adds an additional layer of security to the private key.  If an attacker is able to obtain the file with the encrypted private key, they will need to also know the password to obtain the private key.  The two software wallets I recommend are metamask and myetherwallet.

A hardware wallet is a physical device stores your private key and provides a way to create digital signatures and decryption with the private keys.  This is typically done with what is called a "secure element" - a chip which is considered unhackable - even by the NSA.  A hardware wallet provides an interface to the user to confirm use of the private key - for example a physical button the user has to push to confirm its use.  A hardware wallet is also protected by a password or PIN requiring the user to first "unlock" it before it can be used.  I personally have a ledger nano s and ledger blue.  There are also apps you can get for your mobile device such as the Ethers Wallet iOS App or uPort.  One nice thing about using mobile devices to manage private keys is that almost everyone has one and some have biometric authentication (e.g. fingerprint).


Ledger Nano S hardware wallet


What do wallets mean for healthcare?

Adopting blockchain in healthcare requires educating end users (patients, physicians, staff) about how to use and manage their private keys.  This is perhaps the biggest barrier to blockchain adoption as end user's paradigm for authentication is currently username/password based.  End users will naturally assume a private key can be handled in the same way a password can. They might think that they can share it with someone they trust and change it later if that trust changes.  They might think they can get someone else to change the private key for them if they happen to lose it or forget it.  I have already seen many users lose their ether because they didn't understand the importance of their private key and handle it in a safe way.

Even with proper education, private key management is too unforgiving to be used as the only way to protect medical records.  Fortunately ethereum smart contracts can be used as an identity in place of the private key and include workarounds for these unforgiving scenarios.  This will be explored in the next blog post.

Monday, March 6, 2017

Applying blockchain to healthcare - part 9 (access control)

In my last blog post, we explored how ethereum accounts and digital signatures are used to implement a new form of identity and trust.  In a prior blog post, we used the following Patient smart contract:

pragma solidity ^0.4.2;

contract Patient {

  string public name;
  string public dateOfBirth;
  string public gender;

  // Event that is fired when patient is changed
  event PatientChanged(string what);

  // FAMILY^GIVEN^MIDDLE
  function SetName(string _name) {
    name = _name;
    PatientChanged("name"); // fire the event
  }
  // YYYYMMDD 
  function SetDateOfBirth(string _dateOfBirth) {
    dateOfBirth = _dateOfBirth;
    PatientChanged("dateOfBirth"); // fire the event
  }
  // M,F,U,O
  function SetGender(string _gender) {
    gender = _gender;
    PatientChanged("gender"); // fire the event
  }
}

While this smart contract allows anyone to access the data, it also allows anyone to modify it.  Once this smart contract is published, anyone can see the patient's name, date of birth and gender.  Anyone can also change these properties.  Cleary this does not meet the need for privacy or control.  While access is a major privacy concern - it isn't something we will explore in this post (but will in the future).  In this blog post, we will be exploring how to implement access control on modifications to smart contract instances.  We will do this by using a new smart contract named PatientAllergies which manages a list of allergies for a patient.  Access to allergies is an important part of patient safety as giving a patient a medication they have an allergy to can kill them.  Here is the PatientAllergies smart contract:

pragma solidity ^0.4.2;

contract PatientAllergies {
    // the address of the owner (the patient)
    address public owner;
    // address of physician that can add allergies
    address public physician;
    // name of the patient LAST^FIRST
    string public name;
    // array of allergies this patient has  
    string[] public allergies;

    // constructor that sets the owner to the address creating
    // this smart contract
    function PatientAllergies() {
        owner = msg.sender;
    }

    // allows owner to change the patient name
    function SetName(string _name) {
        // only allow the owner to change the patient name
        if(msg.sender != owner) {
            throw;
        }
        name = _name;
    }

    // allows physician to add an allergy
    function AddAllergy(string _allergie) {
        if(msg.sender != physician) {
            throw;
        }
        allergies.push(_allergie);
    }
  
    // allows owner to set the physician that can add allergies
    function SetPhysician(address _physician) {
        if(msg.sender != owner) {
            throw;
        }
        physician = _physician;
    }
}

You may recall that an ethereum transaction includes a digital signature that ethereum uses to make sure that the account that created the transaction is who they say they are.  The transaction therefore includes the account information and is exposed to smart contracts via the msg.sender property.  msg represents the message (or transaction) being send to the smart contract.  msg.sender is the address of the account that sent it.  We can implement access control by checking msg.sender in functions to see if the caller (user) is able to perform specific operations.  In this case, we assume the account that created the smart contract is the patient and store it in the owner property.  Once this is done, we can add an access control check in the SetName() function so only the owner (patient) can change his name.  We also add a function SetPhysician() that the owner (patient) can invoke to set the address of a physician who is trusted to update the list of allergies.  The AddAllergy() function checks that the account trying to modify the list of allergies is in fact the address the owner(patient) has allowed.

What does this example mean to healthcare?

For patients, it means they can manage their medical record online and control who can make changes to them.  In this case, the patient is the only person who can change the patient name.  The physician that the patient chooses is the only person who can add to the allergy list.

For physicians, it means they can obtain access to the patients list of allergies by interacting directly with the patient.  The patient just needs to give the physician the address of the smart contract and the physician can see the allergies.  The physician does not have to wait until his EMR is integrated with some other EMR, the hospital is connected to an HIE or IT has the right VPN in place.  Trust and access is managed directly between the patient and the physician.

Please keep in mind that this example is overly simplified to illustrate how access control can be implemented with blockchain.  A real application would be much more complex allowing multiple physicians to manipulate the allergy list, the ability to revoke physicians from the access control list and provisions of changing the owner in case the owners private key is compromised.

Next up is how to protect private keys with wallets

Sunday, March 5, 2017

Applying blockchain to healthcare - part 8 (accounts)

Ethereum utilizes public key cryptography to secure the blockchain.  Public key cryptography uses a pair of keys: a private key which is kept secret (like a password) and a public key which can be shared with anyone.  A private key is created using a random number generator and the public key is generated from an algorithm that takes the private key as input.  The algorithm only works one way - the public key can always be derived from the private key, but the private key cannot be derived from the public key.  Ethereum takes this cryptographic key pair one step further by generating an Ethereum address from the public key.  This algorithm is also one way - an address can be derived from the public key, but not vice versa.

The ethereum network only knows about addresses - it does not store private keys or public keys.  When a user wants to make a change to the ethereum network, it creates a transaction.  Each transaction must be paid for using ether - the currency of the ethereum network.  Each ethereum account has an associated balance of ether that is used to pay for transactions.  To ensure that the private key associated with the account paying for the transaction has authorized the transaction, each transaction includes a digital signature produced by the  elliptic curve digital signatures algorithm (ECDSA).  Ethereum checks the digital signature and will not process transactions unless they are valid.

In the prior blog post, you may recall metamask prompting you to confirm a transaction:



Metamask is a browser extension that provides secure and user friendly management of ethereum security.  Metamask lets you as a user create ethereum accounts and presents confirmation dialogs for you to approve spending any of your ether.  In the example above, the name I gave this ethereum account "Coinbase (local)" which corresponds to an account in my ethereum private network.  You will also see some of the characters of the ethereum account - "5DEFE02...2fcc".  Next you will notice the ether balance of the associated account - in this case "216675.000" ETH and the value in USD "4191279.82 USD".  You will also notice it telling you that it is creating a new contract "New Contract" and the maximum amount of ether it will cost to execute the transaction "0.094000 ETH".  Given that the EVM which runs smart contracts it Turing complete, it is not possible to determine the cost of a transaction ahead of time so an upper bound is specified when creating the transaction.  After the transaction is executed, the actual cost is dedicated from the account.

Ethereum's approach to security is important to understand as it is the foundation of how identity and trust is managed which is what everyone is so excited about.  Ethereum's private keys provide a universal way for end users to be identified.  Digital signatures provide a way for ethereum to verify identities.  This is actually quite profound as it allows end users to interact directly with the ethereum blockchain without having to trust any intermediaries.  This is completely different than almost every application we work with today.  Today, we have to prove who we are to the applications usually by logging in using a password.  This works great, but requires that we fully trust the application we are logging into.  In the case of ethereum, there is no application layer to trust.  We as end users can directly assert who we are and what we want changed in the blockchain and no intermediate application layer is required. An end user therefore has full control over any data they put on the blockchain and no human, government or corporation can change it.

What does this mean for healthcare?

It means a patient can put their medical records on the blockchain and control it directly.  They can choose who can see it and who can't.  They can choose who can add to it, and who cannot.  It eliminates the need for patients to trust applications (such as patient portals), the administrators of those applications (system admins), or the corporations (hospitals) that control their data.

It means a physician can obtain access to a patient's medical records by interacting directly with that patient.  The layers of security currently in place are not needed.  Hospitals would not have to have connections with an HIE or other hospitals.  The EMRs from different hospitals do not need to interoperate.  IT does not need to be involve.d  The physician would be able to directly access the information they need, all it requires is the patient to approve it.

Suppose a patient is on vacation and suddenly faints.  The patient is admitted to the local hospital emergency department and since the hospital has no record of this patient, has no knowledge of this patients medical history.  The patient might be able to provide some of his medical history verbally - but how accurate is it?  The hospital can try to obtain the medical records from the patient's hometown hospital - but can it be obtained fast enough and can it be imported into the local hospitals information systems?  Storing medical records on the blockchain can solve these issues and it all begins with identity.  End users simply need ethereum accounts and access to the public blockchain.

Next up is using ethereum accounts to implement access control on the patient record.

Saturday, March 4, 2017

Applying blockchain to healthcare - part 7 (filters)

In my last post, I showed how events could be added to the Patient smart contract so external systems can be notified when it has changed.  In this post I will show how these events can be detected from a web browser application using filters and then use them to look at how the state of the patient smart contract changed over time.  As previously mentioned, blockchain is immutable and update operations work differently than they do in normal databases.  Rather than overwriting the old value with the new one, ethereum records the change and keeps the old value around.  When interacting with a smart contract instance, the default behavior is to use the most recent state.  It is possible though to access historical state - much the same way you can access older versions of a file with a revision control system.  You can find the source code for this application here,  including instructions on how to run it on your local machine (if you have a Mac that is).

After starting the application, you are given the choice to create a new patient smart contract instance or attach to an existing one (assuming you know its address):



In this case, we don't have an existing instance so press "Create" to create a new one.  Metamask will prompt you to confirm the transaction - press "Accept" to do so:



After several seconds, the newly created patient smart contract instance will be created and stored in your private ethereum network.  The address for this newly created instance is displayed in the UI and controls appear to modify the properties of the patient smart contract:


At this point, you can set the values for the patient's name, dob and gender by entering the strings and pressing the corresponding "save" button.  Metamask will prompt you to confirm the transaction every time you press save.  After waiting a few seconds, you can hit the "Refresh" button to see the history of changes to this patient:



The change log shows a few things - the block number that contains the change, the timestamp of the block, the type of change and the values for name, dob and gender at that time point.  Recall that a blockchain is a series of blocks each of which has a pointer to the prior block thus creating a chain.  The blocks store transactions will cause changes to the state of the blockchain.

As stated before, the blockchain is immutable so all prior states are always accessible.  Changing the name from "DOE^JOHN" to "SMITH^WILLIAM" results in a new block being added with this change, but does not go back and modify prior blocks as the blockchain is immutable:



Immutability is an important concept of blockchain that has some interesting implications.  It means that records cannot ever be tampered with.  There is no human, corporation or government that can go remove or alter information on the blockchain.  Once data is added to the blockchain, it will be there forever!  In healthcare, there are few (if any) use cases for destructive changes to data.  This makes blockchain an excellent fit for storing medical records.  Most healthcare applications today have poor (if any) support for accessing historical changes to medical records.  With blockchain, keeping track of changes is automatic and doesn't require any special coding by developers.

You should now have a better understanding of immutability, how it applies to healthcare.  You should also have a better understanding of the ethereum event and filter mechanisms.  Next up is exploring the concept of identity management and trust with blockchain.







Wednesday, March 1, 2017

Applying blockchain to healthcare - part 6 (events)

In my last post, I added validation logic to the Patient smart contract.  In this post, I want to add an event that is fired every time the Patient smart contract instance is changed.  Events are useful for two main things - notifying systems external to ethereum about changes in ethereum and for simple querying.  In this blog post we will focus on using events for communicating with systems external to ethereum about changes to the Patient smart contract instance.  This could be used to update the UI automatically or to keep an external database in sync with the data stored in ethereum.  Keeping a database in sync is a common use case since ethereum has limited query support.

Here is the Patient smart contract with an event "PatientChanged" that is fired every time the patient smart contract instance is changed:

pragma solidity ^0.4.2;

contract Patient {

  string public name;
  string public dateOfBirth;
  string public gender;

  // Event that is fired when patient is changed
  event PatientChanged(string what);

  // FAMILY^GIVEN^MIDDLE
  function SetName(string _name) {
    name = _name;
    PatientChanged("name"); // fire the event
  }
  // YYYYMMDD 
  function SetDateOfBirth(string _dateOfBirth) {
    dateOfBirth = _dateOfBirth;
    PatientChanged("dateOfBirth"); // fire the event
  }
  // M,F,U,O
  function SetGender(string _gender) {
    gender = _gender;
    PatientChanged("gender"); // fire the event
  }
}

Note that I removed the validation logic added from the last post as it is not necessary to discuss events.  In this case, the property that caused the change is included with the event.  You can see this in action by pasting the above smart contract into Remix and changing a property:


In the screen shot you will notice that the event PatientChanged is logged with the value "name" listed.  You can try changing the other properties and you will see the event fired with what being either dateOfBirth or gender.  External systems can create a filter to listen to various types of events.  Filters take as input "topics" which can include any of the following:
1) Address of a smart contract - will return all events related to the smart contract at the specified address
2) Event type - will return all events of a specific type (e.g. all PatientChanged events for all Patient smart contracts)
3) Event data - will return all events that have a specific data item. NOTE: This is not done in the example above.

Next up is setting up a filter to detect modifications to the patient smart contract instance and inspecting the past state of the patient object (remember that blockchain is immutable and all prior state is accessible forever)