Queue backlog issue - 3.1

lundi 23 mars 2015

Mirth Version - 3.1.0.7420

OS - Windows Server 2008

Java 7 64bit

MSQL

Memory allocated to mirth - 10gb (uses it during peak processing)

Memory allocated to Sql - 6gb

CPU - 24 core - low cpu utilization. Spikes into 20% - 30% range, but nothing higher.



Currently having issues with queue backlogs. This occurs if a particular system (channel) goes down or the application sending the messages goes down. If down long enough, and during peak times, mirth will never catch up till later that night. If everything is running there are no issues, and messages send, no queue backlog.



Data being sent to Mirth is xml data that we then convert to hl7. The conversion happens at the individual channel level as its quite typical that the hl7 message will differ, sometimes significantly, per channel. The messages are received via an http listener and average size of xml doc is 200kb. Reports are typically larger, I'd need to capture a few to see typical xml size.



Channel setup



Forwarding channel (http listener) - Simply receives the message and sends it to another channel that handles the filtering. Utilizes source queuing and destination queuing. This is a leftover channel from 2.1.1. With source queuing, we can probably get rid of this. This was utilized in 2.1.1 because we didn't want system sending the messages to backup. We just wanted the messages in mirth and the queuing to happen there.



Direct channel - This handles all the filtering and does so at the source transformer level. We filter via 'destinationSet.remove(['channelNames'])'. This was to optimize the channel as we were seeing other performance issues and queue backlogs if the filter was handled at the destination level. There are a total of 15 destinations a single message could go to. That one message is guaranteed to go to at least 1 destination. 2 of those destinations are high activity (MWL and dictation system). Destinations it sends to are other channels that have source queuing turned on. Data sent is xml. Channels that receive from this channel convert xml to hl7.



When the backlog occurs we see it at the Forwarding channel and Direct channel. Forwarding channel, for instance, will have 10k in destination queue (it's only sending to Direct and Direct has source queuing, so not sure why it hangs here). Direct channel will backlog at one of the two highest activity channels (mwl or dictation system). Again both destinations have source queuing turned on, so not sure why they hang here.



Is it possible we are just hitting the limits of what can be processed? It's not like Mirth stops processing data.





Queue backlog issue - 3.1

0 commentaires:

Enregistrer un commentaire