Simplifier on STU3


by Martijn Harthoorn

As of today Simplifier supports FHIR STU3. Our team has been working hard to give you a FHIR registry that can read, write, listen and speak FHIR STU3 resources. Now you can start uploading STU3 resources.

The change in the FHIR specification from DSTU2 to STU3 itself is not all that big. And had Simplifier been an offline tool, it would have been an overseeable load to upgrade to the new version.

But we have been on a bigger journey. Because we have thousands of profiles and other resources from the whole FHIR community and from over the whole world that are all DSTU2.

And so we didn’t just upgrade Simplifier to STU3, but rather made Simplifier multilingual. Simplifier now supports both DSTU2 and STU3.

Upgrading
From the beginning of January, right after the San Antonio connectathon,  until the end of March, our team has been busy upgrading Simplifier to a new framework (ASP.NET core) to make it more stable and much faster (pages resolve up to 20 times faster!). This allowed us to take the next step: to be able to host multiple FHIR engines inside.

All features are now able to switch the FHIR engine from under their seat depending on the version of the resource they are dealing with. This includes uploading, downloading, the GitHub webhook, the history compare, the FHIR endpoints.

Whats the difference for you?
Each project and each resource now has a FHIR version. In the past we kept numbers like 0.50 (which is actually DSTU1) and 1.0.2 (which is DSTU2). And we only used these numbers for resources. These numbers were set by the version of the internal FHIR engine we used. All these number have now been replaced by “DSTU1”, “DSTU2”, or “STU3”.

Plus, the versions are no longer set by Simplifier, but by you. You can set your project to a FHIR version. And all resources you upload will be read as belonging to that version. You can always switch the FHIR version of your project and nothing will happen to your resources. They will keep the version they already had.

In the near future we will make it possible for you to manually change the FHIR version for the resource too. This goes for all users, whether you have a free account or a paid account.

Simplifier as a FHIR server
As of today the Simplifier FHIR endpoints speak STU3. Both the public endpoint:

https://stu3.simplifier.net/open/StructureDefinition

and the project endpoint:

https://stu3.simplifier.net/ [project]

This also means that you can only connect with client tools that speak STU3.

Validation

A feature that a lot of our users have been asking for is validation. All example STU3 resources will now have a Validation button. We will be adding new features to validation soon, like bulk validation, validation through the FHIR endpoint and validation in Gist.

Are we done?

Probably not. Making a platform ready for multiple versions of a standard is a tricky business. So we expect to mend a few bugs and make some improvements over the weeks to come. Use the feedback button if you think we can do better.

As FHIR registry builders, we are in the business of canonical URLS. Yours to be specific. Since all the base resources in the FHIR spec have the exact same URL in STU3 as they had in DSTU2, you can imagine that we had to practically have to build a wall between the DSTU2 and STU3 resources, to make sure they don’t start referencing to each other. Imagine validating or creating a snapshot using a base profile from the wrong version of the standard. You’re right. It can’t be done. So when you deal with STU3 we will filter out everything DSTU2.

Staying at DSTU2.
Of course you can stay in DSTU2 as long as you like. Simplifier will support DSTU2 for years to come. But remember that the FHIR spec moves forward and so will all of our new features. So if you can migrate to STU3, you probably should.

Moving your resources to STU3
The community has several tools to help you migrate your resources to STU3. Migrating your profiles is slightly more difficult, but there are options, besides doing it manually. And remember, before you upload STU3 resources, make sure you set your Simplifier project to STU3 first!

Happy profiling!

Introducing Forge for STU3

DutchKingWedding

Royal tears of joy at the official STU3 release ceremony

Today marks the community release of Forge for FHIR STU3. A joyful moment, right after our national Dutch King’s day and just in time for the upcoming HL7 WGM in Madrid.

Hoera! Hoera! Hoera!

You can download Forge, the FHIR profile editor, for free from the FHIR profile registry at simplifier.net. In this article we’ll introduce you to the new release and try to answer some common questions about migration to STU3.

Side by side installation

The new Forge for STU3 ClickOnce installer package has a new application identity that allows side by side installation. This means that you can install and use Forge for STU3 next to an existing installation of Forge for DSTU2 on the same machine. Each release will use it’s own version of the FHIR core resource definitions and manage it’s own application configuration settings separately.

dutch-king-willem-alexander-with-wife-maxima-and-animal-pelt

Side by side installation

As of now, future application updates will target the Forge STU3 release. You can still use Forge for DSTU2 and remain to keep using it for as long as you need. But from now on, the Forge DSTU2 branch will no longer be actively developed as our development efforts are now focused on the STU3 branch. We will remain to provide support for the final Forge DSTU2 release to our commercial customers until they have migrated to STU3.

Profiling changes in STU3

FHIR STU3 introduces a lot of updated and new resources and datatypes. Changes to the conformance resources have the largest impact on profiling. Fortunately, the main FHIR conformance resources are relatively mature and haven’t changed drastically, with one notable exception in the terminology space: CodeSystem has been separated from ValueSet. However this change has no impact on Forge.

You will notice some relatively minor changes to the StructureDefinition resource and the ElementDefinition datatype. Some properties have been renamed and there are a couple of new properties, e.g. to define FHIRPath expressions.

One of the more notable changes involves ElementDefinition.type.profile. In DSTU2, an element type can define multiple profiles. However there are some complications with this approach:

  1. Multiple type profiles pose a problem for dynamic code generation, UI input form generation etc.
    Note that although Forge for DSTU2 (secretly…) supports specifying multiple profiles per type, all known FHIR DSTU2 implementations actually only consider the primary type profile. Practical use of this technique has been actively discouraged by the FHIR infrastructure group. We are not aware of any customers to rely on profiles having multiple type profiles.
  2. When an element has type ResourceReference, we would like to be able to express constraints on the reference target. However ElementDefinition.type.profile expresses constraints on the ResourceReference itself.

FHIR STU3 solves these problems by:

  1. Introducing ElementDefinition.type.targetprofile wich applies when the type code equals ResourceReference and expresses a constraint on the reference target. As before, you can use ElementDefinition.type.profile to constrain the ResourceReference datatype itself.
  2. Limiting ElementDefinition.type.profile max cardinality to 1. This takes away any ambiguity for e.g. code/UI generation.

FHIR STU3 also introduces a lot of changes to the other (clinical) resources and data types. Forge for STU3 includes a new STU3-specific version of specification.zip that contains updated definitions for all the existing and newly introduced resources and data types in STU3.

Migration to STU3

As FHIR STU3 introduces breaking changes to StructureDefinition and ElementDefinition, Forge for STU3 is not compatible with existing DSTU2 profiles.

It is possible to automatically convert existing DSTU2 StructureDefinition resources to the new STU3 format. However an automatic conversion tool can only update the meta information in the profile. It cannot safely convert the actual profiling constraints. Because STU3 introduces changes to many of the resources, some of the constraints in a DSTU2 profile will no longer be valid when converted to STU3. For example, when a DSTU2 profile constrains an element that has been renamed/removed in the corresponding STU3 resource. Therefore migrating profiles from DSTU2 to STU3 will usually require manual review and corrections.

We have a team of experienced consultants with a track record in FHIR profiling who can assist and support you with smooth profile conversion to STU3. Feel free to contact us for a quote or if you want to learn more about our professional FHIR consultancy services.

King-Willem-Alexander-Queen-Maxima-Princesses-Catharina-Amalia

Dutch royal family is said to be “super excited” about FHIR STU3

Looking back

We’ve come a long way since the first release of Forge. The initial Forge DSTU1 release attempted to provide a reference implementation of the FHIR standard. Our team tried to demonstrate that the basic FHIR (profiling) principles could indeed be implemented and put to use. Forge was originally designed to hide the user from the complexity of the underlying XML/JSON wire representation. The novel UI design quickly drew a small but growing user base and succeeded to stimulate both the adoption as well as the development of the FHIR standard itself.

After updating Forge to DSTU2, we focused our efforts on improving the quality of the generated differential and snapshot components. Implementing a proper and fully capable snapshot generator in the .NET FHIR API library proved to be a (royal pain in the ***) challenging and time consuming undertaking that claimed a great deal of development time during 2016. But those efforts eventually drove the implementation of some very powerful profiling features in Forge, such as support for derived profiles, deep inline constraints on complex extensions and dynamic expansion of choice type elements and properties. It took a fair bit of time and effort to stabilize some of the more complex features. But thanks to feedback from our users and from the FHIR profiling community, we managed to solve most of the reported issues.

Moving from DSTU2 to STU3, we have noticed a clear trend in Forge issue reports where they slowly but steadily involve more and more complex and rare profiling use cases. This indicates that basic FHIR profiling features have indeed matured and stabilized and that our users are starting to apply more advanced profiling techniques. And while these newly reported issues may not always be easy to solve, generally they do seem to confirm that FHIR profiling is proving to be effective and productive!

Looking forward

Now that we have updated our full tooling stack to STU3 (The official FHIR .NET API library; Forge, the FHIR profile editor; simplifier.net, the FHIR profile registry; and last but not least Vonk, our new enterprise-grade FHIR server), we are looking forward to unleash them onto the FHIR community, starting at the upcoming HL7 WGM in Madrid.

Our simplifier.net platform now supports STU3, but will also remain to serve existing DSTU2 profiles. We will cooperate with our Simplifier users in order to provide migration strategies to update existing DSTU2 projects to STU3. Under the hood, our team has updated the simplifier.net environment to the new Microsoft .NET Core platform, which brings a significant performance increase, improved response times and higher scalability. We are experimenting with the new Microsoft Orleans platform, an exciting new cloud-based technology, in order to implement scalable validation support and initial results are quite promising. This will allow our simplifier.net platform to provide computationally intensive features to an ever increasing and demanding user base.

As Forge has been updated to STU3 and most of the underlying complex profiling logic is finished and maturing, we can now focus our development efforts towards improving the user experience, file management, online simplifier integration and profiling workflow support. We have received a lot of great suggestions from the community about possible improvements and new features that will allow us to further improve the application. Your feedback remains very valuable to us, so please don’t hesitate to submit issue reports and/or feature requests! Should you have a need for commercial support, then you may benefit from one of our simplifier.net account plans.

Last but not least, our brand new Vonk server is now ready for STU3. Vonk is a modular and highly scalable FHIR server based on the new Microsoft .NET core platform. Aside from performance improvements, the new platform also enables some surprising new applications, as recently one of our bright engineers successfully managed to spin up a Docker image of Vonk server running .Net core on Linux. The new flexible options for host operating systems ensure that our Vonk server will smoothly integrate into any existing customer infrastructure.

Final thoughts

Kings day

Millions join the Forge STU3 release party, Amsterdam 2017

We hope that you enjoy the new Forge STU3 release, as well as all our other FHIR tools. The Furore FHIR team celebrated this important milestone in Amsterdam on our King’s birthday, which traditionally involves orange outfits, DJ-powered canal boat rides and abundant substance abuse. And now that the orange smoke has cleared up, we’re back making FHIR simple.

 

Download Forge for STU3 for free from simplifier.net

Happy profiling and long live the King!

 

Zo waerlijck helpe my Forge almechtig

 

Vonk FHIR Server in public Beta

Take a ride

Furore has built a new FHIR Server named Vonk. In keywords: enterprise grade, support available, SQL Server and MongoDB, FHIR Facade, cross-platform, Dockerized, .Net Core, STU3. As of now it is in public Beta. Try it at http://vonk.furore.com or download a trial at Simplifier.

Inspiration

FietsCK-01

This is my bike. After being a simple city bike for the first 25 years, I decided to keep the good parts and improve the rest. Now it suits all my many bike needs. It is clean. No gears, no cables, no chain cover, just the items that I really need. It is strong but not heavy. Steel frame, steel carrier, able to carry my wife on a romantic trip and it doesn’t break on the many potholes and loose tiles in the city. It is versatile. It can ride both dirt- and tarmac roads. Run lightly on my way to work, carry all the groceries for a week on Friday. It has the right size. Large enough for my 1.95m length, but it still easily fits in a bicycle rack. It never gets in my way in the shed. Maybe the best part is the carbon belt drive that I had mounted recently. Unbreakable and yet free of maintenance. And now that the bike is 27 years old and still my daily commuting bike I can faithfully say that it’s proven technology. And my inspiration for developing Vonk.

The old bike

Back in 2012 when FHIR had just started, Furore started implementing the server part of the specification in .Net, and we called it Spark. A reference implementation to test the spec and provide the world with a real FHIR server to develop and test against. It has gained a lot of attraction, well beyond these purposes. Many requests for e.g. production use, SQL Server, STU3 and company support reached us.

The new bike

So we decided to respond with a new incarnation of Spark that would be able to serve as a very versatile yet production ready FHIR server. And it deserved it’s own name: Vonk. Vonk packs together our extensive experience with FHIR, the HL7 FHIR .Net library and Spark in a fully new solution based on Microsoft .Net Core.

fietsck-03.jpg

New, robust technology

Several of you may have seen my talk on building a FHIR Server (> 19:30) at FHIR DevDays last November. By now we have realized many of the ideas that I presented in that talk. As of today we present the official Beta release of Vonk.

Ready to ride

My bike is always ready to ride. For a quick schoolrun or an extended birdwatching ride – it will just run. Vonk FHIR Server is the same. Install and run. It is a stand alone FHIR server with MongoDB storage. Or SQL Server and then truly support transactions. Or run a few quick tests on the in-memory store. The storage API that we created to accommodate these very different storage engines enables us to add other other back ends easily, either with or without support of Entity Framework.

Change the wheels

A few weeks ago my birdwatching buddy lured me into a dirt road. Dirt became mud and then deep mud. Even my bike has limits. It meant I had to continue walking. But with Vonk you just change the wheels! With Vonk FHIR Facade you can leverage the power of the storage API to plug it on top of your existing database. Thus Vonk acts as a FHIR REST facade on top of your (legacy) system. And although Vonk is enterprise grade, it is not heavy. So you can unleash your systems small and large all in the same way. All you need to do is create an implementation of the storage abstraction for your use case. We already did so ourselves: the Simplifier.net platform runs on Vonk. The same abstraction even allows you to address other (web) services for retrieving or storing the actual data, thereby using Vonk as a hub, possibly backed by other Vonk instances that are facades to your internal systems. When creating a facade you program the new wheels in .Net, with all the power that comes with that. On the roadmap is creating facades by configuration.

Assemble your own

No gears, no handbrakes, clean as can be. But a bit too clean if you need to climb a hill. In the Netherlands we don’t do hills, but Vonk can be used around the globe. Vonk FHIR Components gives you all the good parts of Vonk, for you to assemble your own perfect FHIR Server. Or integrate it into a larger web server. Or turn it into a hometrainer. Just add and leave out parts as you like. Vonk FHIR Components will become available as NuGet packages.

Run all the tasks you need

From groceries to commuting: even my bike is not as versatile as Vonk. In this beta release it will do STU3 and DSTU2, JSON and XML, and most of the regular FHIR interactions plus a few bonus ones:

  • capability statement (metadata)
  • create, read, update, delete
  • search on all specified parameters, also chained, choices, multiple resource types, include related resources
  • full resource version management, enabling history on resource, resource type and system level
  • process batches of interactions, in a transaction if you wish (requires transactional storage)
  • validate a resource against the spec or a custom profile
  • extensible to custom operation (‘$your-operation’)

The road ahead

More features are on their way. Priority is volatile and can be influenced by customer requests.

  • support for several more options, e.g. _summary, _elements and conditional operations
  • subscriptions
  • documents ($document)
  • proper support for ‘Binary’ resources

Lock it

fietsck-06.jpg

Security in Amsterdam is not about riding with a helmet, but putting at least two solid locks on your bike. We know a thing or two about not letting your bike get stolen. Or your data. So Vonk runs on https, can be fronted by a reverse proxy (e.g. Apache or NGINX) and exposes just the FHIR RESTful API endpoints. But a fully locked bike brings you nowhere. You need access, but limited access if needed. Therefore Vonk can participate in an OAuth2 single sign-on environment. Access to resources can be based on the claims in the OAuth2 access token. You can use the preconfigured scopes and launch contexts as defined by SMART-on-FHIR, or configure your own.

Ride anywhere

Although my bike won’t ride on water (in Amsterdam we have canalbikes for that) or deep mud, just about any other surface will do. And Vonk is no different: it runs on Windows, Linux, OSX, as an executable or in a Docker container, on premise or in the cloud, as an Azure App Service – you choose.

Ride fast

When my colleague Marco showed me Vonk running on Docker I was so thrilled I even overtook e-bikes at 30 km/h on my way home. With Vonk we also made performance a number one priority. Using the newest Microsoft ASP.Net technology and proven storage solutions we managed to implement a very performant FHIR Server. Being lightweight and fully asynchronous you can scale out or up as you see fit.

Be in control

In the Netherlands there is wind. Usually in your face. You need to lie low. So my first upgrade was the handlebar that allows me to lie comfortably on it and still keep control over the bike. Extensive semantic logging to just about any target and a connection to Microsoft Application Insights allows you to control Vonk just as well. Integrated validation can be used as a gatekeeper to accept only compliant resources. On the roadmap is also an extensible management API to:

FietsCK-05

  • add profiles, or restrict to specific profiles
  • add or restrict the search parameters
  • enable or disable operations

Where is the blueprint?

I’d be happy to tell you how I built this bike. But Vonk is not open source. Vonk is a commercial product based on license fees and paid support. Nevertheless we want to continue to support the FHIR community as we already do with the FHIR .Net API and Forge. In short it is possible to use Vonk for free if you support the FHIR community in any way with that usage. Just contact us to see if you’re proposed usage fits this goal.

What happens to the old parts?

I kept them in my shed as spares. As for Spark: This is up to the community. Furore has put a lot of effort in Spark and provides all of this effort to the community. It’s open source, available on GitHub. But Furore will neither port Spark to STU3 nor provide support for DSTU2.

Name

I never gave my bike a name. But ‘Vonk’ means two things. First, it is the Dutch word for Spark. After struggling with our company name Furore we thought the FHIR community was up for the next Dutch challenge. Second it reminds us of our dear collegue Hans Vonkeman who has passed away far too young and thus it is a kind of a tribute.

And now I’ve been at the keyboard long enough. Off for a ride. Do you ride along? Visit the getting started to see how you can get a trial Vonk FHIR Server as binaries or in a Docker container.

Prefer to ride along face to face? I’ll be at the HL7 WGM in Madrid next week, albeit not by bike. Or visit the FHIR DevDays here in Amsterdam in November. There I will be by bike.

Cheers, Christiaan
FietsCK-07

Simplifier and Forge Migration to STU3

by Martijn Harthoorn

TL;DR
Simplifier, Forge, and Vonk (the successor of Spark) will be migrated to STU3 before the Madrid WGM in May.

The week after March 20th, the next version of the FHIR specification STU3 will go live. This will be the first STU. Notice the D is gone? You can find out why here.

As the tools that we provide to the FHIR community are quite popular and heavily used, we need a very good plan to migrate to STU3. Of course we want to bring you STU3 as soon as possible. But we are not going to drop DSTU2 support just like that. Looking at several of our customers, we expect some companies will remain dependent on DSTU2 for a while. Here’s what we’ll do with our tooling.

Forge

Forge has gathered a significant user base. And since Forge is distributed via a ClickOnce installer, we will create a separate side-by-side release for STU3. This means that you will be able to install and run both the current DSTU2 and STU3 Forge releases on the same machine. We will not distribute the STU3 release as an auto-update of the current DSTU2 release; otherwise existing Forge users would no longer be able to manage any DSTU2 profiles. We will announce the STU3 release via the Furore news feed on Simplifier and also on the FHIR chat groups on Zulip.

Forge depends heavily on the FHIR .NET API, which we are currently upgrading to STU3. STU3 brings changes in some of the core resources and datatypes. STU3 also introduces some important changes in the meta data. Therefore DSTU2 and STU3 profiles are not directly interchangeable.

Simplifier

The migration of Simplifier is more complex. In addition to introducing support for STU3, we are also migrating Simplifier to a newer technology stack (ASP.NET core for anyone who’s interested). An additional complication is that Simplifier needs to be able to host both DSTU2 profiles and STU3 profiles at the same time. We will make a strong push towards STU3 after march 20th, but that does not imply that we will abandon DSTU2.

We will make projects on Simplifier FHIR version specific to help you, and our tooling,  figure out how and what we should render. Of course, it makes no sense to try and render a DSTU2 profile with a STU3 render engine. It would just fail.

The strategy that we will follow is what we call a graceful fallback strategy. This means that we will move all our internal Simplifier tooling to STU3. And we will move all the rendering logic to STU3. But we will make it possible to still host and even upload DSTU2 profiles. This is not as simple as it may seem. After careful consideration, we decided to implement FHIR validation as a separate step, instead of being tightly integrated as a mandatory step to the import. Currently you are not able to upload DSTU1 or STU3 resources, because the FHIR engine in Simplifier  considers it to be invalid input (does not conform to FHIR DSTU2). We are going to change the upload procedure, so that the server will accept invalid FHIR input. And instead of rejecting invalid input, Simplifier will inform you that it is not valid STU3 FHIR.

But this is where our graceful fallback strategy comes into play: we can’t keep everything live for all versions of FHIR. We think it’s not beneficial for the standard either. Whenever you can migrate to STU3, you should. But we will make an effort to do what we can to keep DSTU2 working for you. We keep the existing tool set for DSTU2 alive. And the more we manage to do, the better your DSTU2 experience you will be.

Schedule

This is our schedule

February 13th – API
We started work on our libraries and packages on NuGet to be as ready as we can for STU3. This includes the model (base resources) parsing and serialization, validation, snapshot generation.

March 20th – FHIR
FHIR STU3 is live.

April 1st – Simplifier
This is not a precise date yet, but around this time we expect our first internal release for BETA STU3 . This is also when we start working on making Simplifier still support DSTU2.

April 17th – Simplifier & Forge
Around this time Simplifier and Forge for will go live with STU3 BETA for the daredevils.

May 6th – Simplifier
Before this date, a week before the Madrid WGM starts, Simplifier and Forge will be battle tested for STU3 and you should be ready to use them.

When something goes live, we’ll tweet about it.

Migration of your profiles

Whether you manage your profiles on Simplifier or on your local disk, migrating structure definitions automatically is no task that can be done without human supervision. But there are several tools in the community to help you. And of course our consultancy team can offer to help you with that too. They have helped a wide range of customers around the globe with migrating their profiles to STU3.

 

 

 

Profile versioning

by Martijn Harthoorn

In the Simplifier team we think a lot about Profiles. We try to think of the places where they will be used and how they will be used. I will share with you one recent discussion we had about versioning of profiles.

The discussion started after a comment in the FHIR community from Lloyd McKenzie of the FHIR core team that a lot of profiles have references to other profiles (profiled references), that when updated, cause cascading updates. Let’s explore this. Assume we are working with a profile on Patient for our Hospital X. Let’s give the first published version of this profile the following canonical URI:
http://hospitalx/Patient-V1 (and lets abbreviate it to P1)

Now let’s assume we also have DiagnosticReports in our hospital, so we create a profile for those, and we give it this canonical URI:
http://hospitalx/DiagnosticReport-V1. (abbreviating it to D1)

As you might know a DiagnosticReport references a Patient. It’s not unlikely to assume here that we want our DiagnosticReports to only refer to Patient resources from our own hospital. For that FHIR allows you to profile the reference to Patient: a constrained reference. In your profile you tell that whatever patient your diagnostic report is about, that patient has be conformant to a Patient profile with the given canonical URI. In our case, the profile on DiagnosticReport (D1) will have a reference to Patient constrained to only allow P1.

This might seem a logical thing to do, but it turns out that it can be quite a burden on the authors of these profiles. If you publish a new version of your Patient profile (let’s call it P2), your existing diagnostic report profiles (D1) won’t allow referencing it because they have the reference definition constrained to only accept references to patients profiled on P1.

profileonpatientchanged

If you would update the DiagnosticReport profile, references to Patients of the P1 profile are not allowed either.

2017-01-05-13_43_40-drawing1-visio-professional

In some situations, this can cascade to an update of 50 profiles with a breaking change.

To be more precise: the problem that profile authors face is that they have to update the canonical URI in all profiles with a constrained reference. It also means that they want to be able to republish their profiles at once as a batch. You don’t want to have your other profiles reference an older version.

For these two steps, changing the references and republishing, it is possible to build tooling that can do most of the work automatically. We’re working hard on getting such capability within Simplifier for you.

But our discussion brought two other interesting versioning challenges to light.

The first challenge concerns authoring and deployment collaboration. Let’s say that the central administration of our hospital X upgrades their patient profile. But the lab creates their own DiagnosticReport profiles, and these DiagnosticReport profiles (of course) constrain the reference to Patient to only allow patients from within the hospital. Again D1 references P1. Now after the upgrade of the central administration for Patient from P1 to P2, the lab won’t be able to accept new P2 patients that are sent from within their own hospital, unless they upgraded their servers at the exact same time. This might be a good thing, because a breaking change is a breaking change for a reason. Note here that a breaking change is the only reason to change your canonical URI. But our presumption is that in a lot of cases, the profiled reference is not a matter of exact conformity. The lab doesn’t care about the all the exact fields of the patient. The only reason why the Patient reference was profiled, was to be sure they would not send or receive Patient data from outside the hospital. For example: they possibly only cared that the Patients they receive all have an identity given out by the central administration. For this likely scenario, a  canonical URL change creates a huge unnecessary burden that might trigger a very expensive cascade of downstream updates in profiles.

There are solutions here. For example: the hospital could start of with a profile “Patient Zero” (P0) that only has the hospital identity field. Then, have the ‘real’ profiles (P1 and P2) be derived profiles from “Patient zero”. This is not very obvious and of course requires a forward thinking profile author that had the smarts to take this step as his first action of business. It won’t be easy fix afterwards.

The second challenge that surfaced in our discussion is about our own tool Simplifier.net and possibly any FHIR registry. Once you have published one of your profiles in Simplifier, you have assigned it a unique canonical URI, you marked it as [Active] and by that you made a promise to your profile consumers that you won’t introduce any breaking changes for that profile. But of course at some point you are going to have to change something. Whether or not you are introducing a small update or a breaking change, you would like to continue working with that same profile. In the case of your Simplifier project, you don’t want to lose your version history, issue tracking and activity log. However you also don’t want to change the status of your published resource from “active” back to “draft”. Currently, you will have to create a new resource and work from there. That’s something we realized we have to change as creators of the tool.

For now our plan for the future is to allow creating draft versions of a resource even after it’s been published, instead of making you create a new resource. This will also require registering a canonical URI for each published version of your resource instead of how it is now: just once, which is automatically the latest version.

Your feedback here is appreciated as always. There is a feedback button on the top of the page in Simplifier.net. Our support team always reads it and responds to it.

Happy new year and happy profiling to everyone.

Forge

Forge 14.4 technical overview

Introduction

Last Friday (November 25th) we published a new Forge release, Forge 14.4 for DSTU2 – FHIR DevDays 2016 edition. You can download the new Forge release for free from www.simplifier.net. The previous Forge release (13.2) dates from June 10th, so clearly it’s been a while since we’ve published an update. In this blog post I’ll elaborate on the new features.

Profiles on profiles

Profiles created in Forge 13.2 and earlier always express a set of constraints on a FHIR core datatype or resource. However the FHIR specification also allows you to author so-called profiles-on-profiles or derived profiles. A derived profile represents a set of constraints on another, existing (non-core) base profile. This allows you to define a hierarchy of profiles that are based on each other.

In a typical situation, a national standards organization publishes a set of official national FHIR profiles defining some common country-wide constraints. Organizations within this country create and publish their own profiles that are derived from the official national profiles. For example, the Dutch HL7 affiliate publishes a national Dutch Patient profile with a specific constraint on Patient.identifier for the Dutch social security number (BSN). Dutch organizations can now author and publish their own, more specialized profiles with additional organizational constraints based on (derived from) the national profiles. Because different organizational profiles all conform to a common set of national profiles, they provide a certain well-defined level of interoperability within the country.

profile-hierarchy

Hierarchy of Profiles

The rules for derived profiles are similar to the rules for profiles on core datatypes or resources. A profile is valid if all resources that conform to the profile also conform to the associated base profile. This implies that a profile is only allowed constrain the base profile. A profile is not allowed to relax/remove constraints defined by the base profile.

Note: In a sense, we can interpret a profile as a filter or predicate on the resources space, cf. a WHERE clause in a SQL expression. The number of constraints that a profile expresses is inversely proportional to the total amount of conforming resources. As derived profiles become more and more specific, the total set of conforming resources becomes smaller and smaller. The extreme case being a profile that matches no resources at all; such a profile may very well be valid according to the FHIR spec, albeit meaningless. Similar to the SQL predicate WHERE 1=0 which is perfectly valid but will always evaluate to an empty set.

new-derived-profile

In Forge release 14.4, you can now create and edit profiles based on other (non-core) profiles. The command New Derived Profile creates a new derived profile based on an existing base profile on disk. The initial implementation is still somewhat limited and provides support for fairly simple base profiles (e.g. without slicing). Future updates will add support for more advanced scenario’s such as reslicing.

Resolving resources

Profiles created in previous Forge releases until version 13.2 were always based on a core datatype or resource definition.The Forge application directory contains an archive “specification.zip” that contains the official definitions of all FHIR core datatypes and resources. Upon application startup, Forge scans the ZIP archive and loads all the available definitions. When creating a new profile, Forge extracts the selected core base definition from the ZIP file and initializes a new profile from the selected base.

In the new 14.4 release we’ve added support to create and edit profiles based on other existing profiles. Forge now needs to resolve the custom base profile from a custom location, as it can no longer be resolved from the default ZIP archive that contains only the core definitions.

A profile can also express constraints on element types via external profiles. For example, a Patient profile may introduce a custom Identifier profile on the Patient.identifier element. In this case, Forge also needs to resolve the specified external profile in order to dynamically generate user interface components for profile child elements and verify the specified constraints.

The same holds for extension elements in a profile. Each extension element is mapped to an external extension definition. Forge needs to resolve the specified extension definitions in order to generate UI and verify constraints.

Forge should be able to resolve external profiles from the containing (working) folder on disk, from a separate external directory, from a FHIR server or from the FHIR registry Simplifier.net. To implement this, Forge requires a generic and flexible system to resolve external (user defined) profiles on the fly, separate from the standard core datatype and resource definitions.

Fortunately (and not coincidentally…) the FHIR .NET API library provides a flexible and efficient resolving mechanism based on the generic IResourceResolver interface, as explained by Ewout in his recent article Validation – and other new features in the .NET API stack. The library exposes default implementations that support resolving resources from disk and online. The library also exposes convenient decorators for caching and resolving from multiple sources based on priority. Forge 14.4 now leverages the FHIR .NET API to implement a flexible multi-step mechanism for resolving external (conformance) resources.

resourceresolver

Resource Resolvers

Upon opening an existing profile in Forge 14.4, the application creates an associated resolver that is bound to the containing folder on disk (including subfolders). The resolver provides cached access to all external profiles in the same folder structure. Future updates will introduce support for the configuration of additional sources, e.g. on local disk or online.

Note: for now, please keep related profiles together in a common folder and Forge should find them. Be aware that the resource resolvers cannot resolve profiles with duplicates, i.e. multiple profiles in the same folder that share a common canonical URI. When Forge detects duplicate profiles, it will display an error message and list the conflicting files. In that case, manually delete/move all the obsolete duplicates from the folder and retry.

The current Session Explorer in Forge provides a conceptual/logical view of resources and resource containers. However the new resource resolving approach introduces a new folder-based workflow that is fundamentally different from the earlier approach and does not fit well with the current Session Explorer. Therefore the Session Explorer will soon be phased out in favor of a new folder-based approach (cf. Visual Studio Solution Explorer).

Snapshot generation

One of the largest changes in the Forge 14.4 release involves a complete new implementation of the logic involving the generation of snapshot and differential components. Let’s take a closer look at these concepts.

snapchat-glasses

Snapshot Spectacles

A FHIR profile defines a structured set of elements. Each element definition has a set of properties. The combination of all the element definitions and associated properties uniquely defines a specific datatype or resource structure. This holds for both a core datatype or resource definition, as well as for a (derived) profile. FHIR defines this complete representation as the so-called “snapshot” component. The official name might be a bit misleading in that it suggests a captured state at a specific moment in time; while this is not incorrect, the key aspect of the snapshot component is that it intends to provide a complete and full representation of a certain data structure. An application that receives a profile snapshot has all the available information and does not need to resolve any external profiles.

A snapshot component can be quite large (~ thousands of lines of XML/JSON). And for a derived profile, the snapshot component contains a lot of redundant information: complete definitions of all the unconstrained elements and properties that are inherited from the base profile. Provided an application has access to the base resource, then it is possible to (re-)generate all the inherited redundant information. So we can also represent a profile by only describing the actual profile constraints and omitting all the information that is implicitly inherited from the base profile. Such a sparse representation of profile constraints is defined by FHIR as the “differential” component. The differential component only expresses the actual constraints with respect to the associated base profile. Generally, the differential component is (much) smaller than the snapshot component. Also the differential provides a convenient serialized representation for profile authors to browse and inspect, contrary to the snapshot component that contains a lot of redundant noise which makes it difficult to navigate.

The FHIR specification requires that a profile should contain either a snapshot component or a differential component or both (invariant sdf-6). A serialized profile with only a differential component has a smaller size and is therefore cheaper to persist and transmit. However this puts the burden on the receiving side to (re-)generate the snapshot component from the associated base profile. A serialized profile that contains a snapshot component is much larger but also self-sufficient; it contains all the available information and the receiver does not need access to any external profiles.

Forge releases up until version 13.2 contained custom application logic to generate the snapshot component from the differential. The old logic was fairly simple but useful as long as Forge was limited to profiles on core datatypes and resources. However in order to implement support for profiles on profiles (a.k.a. derived profiles), the existing snapshot generation logic was no longer sufficiently capable and needed to become a lot smarter. As we now had to completely re-implement snapshot generation anyway, we decided to move the logic out of Forge and into the open source FHIR .NET API library.

The implementation of full-fledged snapshot generation proved to be quite an undertaking. During the process we discovered that some low-level aspects were not yet clearly and/or fully defined by the FHIR specification. So we cooperated with the FHIR core team and profiling community to fill in the blanks and complete the missing parts of the specification. Especially Chris Grenz‘s contributions proved to be invaluable – Kudos! Working closely together with Chris, who works at Mayo Clinic, we managed to define how to unambiguously express some remaining advanced scenario’s and subsequently to align both our implementations.

Unfortunately the snapshot component is not completely deterministic, e.g. element id’s may vary, list order may vary etc. So FHIR cannot and does not define a “canonical” snapshot format that would allow us to perform a byte-wise or node-by-node comparison of two snapshot versions. Therefore a snapshot comparison algorithm needs some custom logic to handle special FHIR rules.

Although snapshot generation certainly isn’t the sexiest topic within the FHIR spec, it is a crucial aspect for any system dealing with profiles (including validation). And feature-complete snapshot generation proved to be quite challenging to implement. For example, in order to generate snapshots for core type definitions, the algorithm must be able to detect and handle recursive type relations, e.g. Element.id and Extension.extension. Also the correct implementation of different scenario’s involving (re-)slicing proved to be non-trivial. So we’re quite pleased that this fundamental part of the FHIR specification is now clearly defined and completely implemented (barring some unknown unknowns…). If you’re interested in the implementation (you are definitely a FHIR nerd like us and) you can inspect the Hl7.Fhir.Specification component source on Github. Fortunately usage is very simple:

IResourceResolver resolver = ...
StructureDefinition structureDef = ...
var generator = new SnapshotGenerator(resolver, settings);
generator.Update(structureDef);

The snapshot generator uses the specified IResourceResolver instance to resolve references to external profiles and/or extension definitions. If this argument equals null, then the generator automatically falls back to the set of FHIR core datatype and resource definitions. The optional settings parameter allows the caller to specify some custom configuration options for the generator.

simplifierSimplifier.net is not yet capable of automatically (re-)generating the snapshot component. Currently, when you publish a profile, you have to explicitly submit the snapshot component in order for Simplifier.net to render the complete structure. Note that if you publish a profile to Simplifier from within Forge, the application automatically handles this. Once the new release of the FHIR .NET API is ready and stable, we will update Simplifier.net and integrate the new snapshot generator. From then on, Simplifier will be able to accept profiles with only a differential component and automatically (re-)generate the snapshot. Our team is also working on implementing a dynamic differential rendering in Simplifier, so the user can activate a filtered view of only the differential profile constraints.

Differential generation

We’ve seen how snapshot generation transforms a sparse tree of constraints into a complete representation of the target structure. This functionality is now provided by the FHIR .NET API library, in the form of an atomic operation (cf. pipeline). Of course, there also exists an inverse transformation that generates a sparse differential component from a complete snapshot representation of a structure. The differential representation only contains nodes that are constrained with respect to the base resource. Unconstrained nodes are excluded from the differential.

differential

Differential Structure

Forge generates the differential representation on-the-fly and keeps it in sync with the profile while it is being edited. This approach is completely different from the snapshot generation. Each element definition and property in a user profile is associated with the corresponding node in the underlying base profile. This allows Forge to determine which profile nodes are actually constrained (not empty and not equal to base). To no surprise, the process to find the corresponding nodes in the base profile turns out to be quite involved, esp. for slicing.

Originally, up until version 13.2, Forge contained custom logic to resolve corresponding nodes in the base profile. The original logic was limited to handling only fairly simple profiles, esp. core datatype and resource definitions and was not very efficient on memory usage. As the application developed further, we were slowly stretching the limits of the original algorithm. Users started reporting unpredictable behavior (i.e. differential was not always properly synchronized) that was hard/impossible to fix. Eventually this was no longer maintainable. So we decided to also completely rewrite the differential generation logic.

Fortunately, the new snapshot generator logic in the FHIR .NET API library already needs to resolve corresponding nodes in the base profile anyway (because each differential constraint is merged onto the corresponding base node). So Forge does no longer have to perform that same complex exercise itself. Instead, Forge now receives all the base profile associations as a useful by-product of the snapshot generator. This is quite efficient and increases the loading speed of a profile. Also we no longer need to maintain and harmonize separate but similar implementations in Forge and the .NET API library, as all common and reusable logic has now been moved into the API.

Since each node in the loaded profile is now associated with the corresponding node in the base profile, it is now trivial to determine which nodes are actually constrained with respect to the base resource. Only constrained nodes are actually included in the differential. Unconstrained element properties (either empty or equal to base) are set to null in the underlying PoCo. Also complex nodes and element definitions having all child properties equal to null are cleared in the PoCo and their state is automatically re-evaluated whenever a child node has changed. This way, all unconstrained nodes are effectively excluded from the differential component. The whole process is quite efficient as after a change, only the affected nodes are re-evaluated.

storagehandlers

Generate Differential

The serialized xml representation is (re-)generated in the background and rendered whenever the user activates the Xml tab. This ensures that the displayed xml is always up to date without impeding the application performance. The Forge UI only displays the differential, by design, in order to save memory. By default, profiles are also saved to disk with only the differential component. The Options menu provides a configuration setting to toggle the serialization of snapshots components when saving profiles to disk. Needles to say that profiles with snapshots are significantly larger, therefore in Forge the snapshot option is disabled by default.

Conclusion

This article is way too long… hats off if you managed to read and process all this information. And I still feel like I just scratched the surface. Time permitting, maybe I’ll author another article about the specifics of the snapshot generator implementation, for those few FHIR die hards that care (the usual suspects).

sad-tree2

Sparse Tree

Looking back to 2016, the Furore FHIR team has invested a lot of time and effort in (re-)designing and (re-)implementing the fundamental infrastructural components of a FHIR ecosystem, thereby laying the groundwork for a plethora of useful productivity-enhancing features that we can now start building and enhancing in 2017. I must confess that I really enjoyed working  on some of the hard core abstract logic. But the nerdy stuff can also be quite demanding, time consuming and eventually goes at the expense of basic social behavior and interpersonal relations… So I’ve learned a long time ago that, even though I take joy in higher abstract thinking, I shouldn’t bury myself in it for all too long, as there is a personal price to pay. Now the Forge user interface desperately needs some love – as well as my estranged girlfriend, who’s totally fed up with me rambling on and on about snapshots and such. So I am eagerly looking forward to next year, when I can focus again on improving usability, workflow and my personal love life. But first let’s celebrate the holidays, decorate a tree, spend some quality time with family and friends and hopefully regain some of those long lost social skills.

Happy profiling!

Michel Rutten

Forge

Forging Profiles on Profiles

Introduction

Recently (Friday November 25th) we published a new Forge release, Forge 14.4 for DSTU2 – FHIR DevDays 2016 edition. You can download the new Forge release for free from Simplifier.net. The previous Forge release (13.2) dates from June 10th, so clearly it’s been a while since we’ve published an update. In this blog post I’ll elaborate on the most important new feature: support for profiles on profiles, also known as derived profiles.

What is a derived profile?

Out of the box FHIR resources are designed to cover 80% of the common use cases. In order to cover the remaining 20%, FHIR introduces profiles. A FHIR profile defines a set of constraints on a standard FHIR resource or datatype. This allows you to define a specialized variant of a standard resource for a specific use case. And because profiles are based on common FHIR resources, they provide a certain basic level of global interoperability with other (external) FHIR systems.

Obviously, as we limit the context to a specific country, region or organization, we can probably harmonize more aspects and provide increased interoperability within that limited context. FHIR supports this common use case by allowing you to define so-called profiles on profiles a.k.a. derived profiles. A derived profile defines a set of constraints on another existing profile, the so-called base profile. The derived profile implicitly inherits all the existing constraints from the associated base profile. A derived profile can only further constrain the base profile, similar to profiles on core resources.

Example:

  • The HL7 affiliate for the Netherlands publishes a national profile for a Dutch patient. The profile is based on the standard FHIR Patient resource and defines a number of additional constraints, e.g. in order to identify Dutch patients by their unique national social security number (BSN). The profile also constrains the generic Patient.careProvider reference to a specific Dutch practitioner profile.
  • A Dutch hospital authors a derived Patient profile for use within the organization. The organizational Patient profile is based on the national Dutch patient profile and defines some additional constraints that only apply within the organization. The hospital’s custom Patient profile defines additional constraints for the unique Patient identifier that is assigned by the hospital EHR system. The profile also prohibits the Patient.animal element, as it does not apply.
Hierarchy of Profiles

Hierarchy of profiles

As the example demonstrates, this allows you to create a hierarchy of profiles, e.g. on a national, regional, organizational and/or departmental level. As we navigate deeper down the profile hierarchy, we encounter more specific profiles for more limited use contexts.

Authoring a derived profile in Forge

New Derived ProfileUsing the new Forge 14.4 release, you can now easily author a derived profile. Forge 14.4 introduces a new command “New Derived Profile“. You can find the new command directly on the start screen, in the File menu and also in the Session Explorer toolbar. The new command resembles the existing “New Profile” command, but with one important difference: you can select an existing profile as the underlying base profile. Forge will display a File Open dialog window to select a suitable base profile from disk. After selecting the base profile, Forge will create and display the new derived profile.

For example, let’s assume that Mayo Clinic wants to create their own derived Patient profile based on the US DAF Patient profile which defines a set of national constraints for the US Patient. Below, we can see the original DAF Patient base profile loaded in Forge. All the constraints defined by the base profile are clearly indicated by the yellow pen icon:

DAF Patient Profile

Notes:

  • The blue exclamation icon indicates a “must support” element
  • The blue paperclip icon indicates extension elements
  • The yellow pen icon indicates a constrained element

Now Mayo Clinics creates a new derived Patient profile based on the DAF Patient profile. The new derived Patient profile inherits all constraints that are defined by the base DAF Patient profile. Initially, the new derived profile does not introduce any new constraints, as is clearly visible in the Forge Element Tree:

Mayo Clinic Patient Profile

The screenshot demonstrates that the derived Patient profile indeed inherits all the constraints defined by the base DAF Patient profile (e.g. you can see the inherited extensions such as ethnicity). Because the implicitly inherited constraints are not defined by the derived profile itself, they are not considered to be part of it. This is clearly visible in the Forge element tree, as none of the elements in the new derived profile show a yellow pen icon (except on the root element, because the derived profile has a new name & url). In other words, the new derived profile shown above is (almost) empty.

Now Mayo Clinic can define additional specific constraints in the derived profile. Any newly defined constraints will actually be part of the derived profile and consequently they will be indicated by a yellow pen in Forge. For example, the screenshot below shows how the derived profile defines an additional constraint on the Patient.photo element:

Mayo Clinic Patient Profile with constraint on photo

Below you can see the serialized XML of the derived Patient profile. We can clearly see the newly introduced constraint on the Patient.photo element. Note that the root element definition is mandatory in DSTU2, even if the element is not constrained. This requirement will be deprecated in STU3. Also note the (mandatory) reference to the underlying DAF Patient base profile.

<?xml version="1.0" encoding="utf-8"?>
<StructureDefinition xmlns="http://hl7.org/fhir">
  <meta>
    <lastUpdated value="2016-12-01T16:47:22.41+01:00" />
  </meta>
  <url value="StructureDefinition/MayClinicPatientProfile" />
  <name value="Mayo Clinic Patient Profile" />
  <status value="draft" />
  <date value="2016-12-01T16:28:17+01:00" />
  <fhirVersion value="1.0.2" />
  <kind value="resource" />
  <constrainedType value="Patient" />
  <abstract value="false" />
  <base value="http://hl7.org/fhir/StructureDefinition/daf-patient" />
  <differential>
    <element>
      <path value="Patient" />
    </element>
    <element>
      <path value="Patient.photo" />
      <max value="0" />
    </element>
  </differential>
</StructureDefinition>

Conclusion

This concludes our introduction of the new support for derived profiles in Forge. The example demonstrates that the authoring process is actually quite easy for end users and quite similar to creating profiles on core resources. After all, in FHIR we try to move complexity away to the developers of tools and servers.

As simple as this all seems to the end user, the actual implementation entailed a long and winding road straight into the bowels of the FHIR specification. In a separate blog post, I will provide some more in-depth background information about the technical challenges we faced when trying to implement support for derived profiles, and how we addressed these.

If you have questions concerning (derived) profiles or need a bit of assistance, then you are strongly encouraged to join the online FHIR discussion groups. The FHIR community is very active, friendly and helpful and usually you quickly receive helpful responses from FHIR users all over the world. The conformance stream is specifically intended to discuss FHIR profiling and related tools including Forge. Also if you’ve implemented FHIR in a production environment, then please inform the FHIR community and share your story in the implementers stream!

Two final notes:

  1. Today we have published a minor hotfix release (14.4.1) with a couple of bugfixes. Should you encounter any issues while trying to repeat the exercise above, then please make sure that you have upgraded to the most recent version. Users of release 14.4 will receive an auto-update notification. Users of earlier releases should manually download and install the new version.
  2. The new Forge release does not yet support base profiles containing slices. If you select a base profile containing slices, snapshot generation will fail and generate a runtime exception. We will introduce support for sliced base profiles in a future application update.

Happy profiling!

Michel Rutten