Tag Archives: Task Notifications

K2 – no task notifications processing due to “Unable to load EventBus Server” error

K2 versions older than Five rely on MSMQ for task notification emails queuing and processing. Sometimes you may see the following scenario:

You start process which supposed to send task notification. Notification not delivered and there is no error messages logged nor in K2 host server log, MSMQ diagnostic log nor in Eventbus.ClientRecorderError table logged at the time of task assignment. Obviously process does not go into error state as notification delivery performed on “best possible effort” principle and not preventing user from completing assigned task using his worklist which can be exposed through different UIs.
At the same time it is possible to see that queue exist and messages gets queued into it (Computer Management snap-in > Services and Applications > Message Queuing > Eventbus > Queue Messages:

If it is not highly loaded production environment you will be able to see that message gets queued per each client event, and you can also see that Message ID gets assigned with increment of 2, with new messages being added to the end of the queue. At this stage you may think that Message Queuing may not run – but this is unlikely as when this service stopped message cannot be placed into queue (that definitely will be logged on K2 side) and you unable to view Queue contents in computer management with the following error:

Other possibility is somehow malformed error message and some glitch in MSMQ service itself – again a bit unlikely as new messages gets queued just fine, but you can try Message Service restart along with removing topmost message from the queue. If that does not help (which most likely will be the case), check the very beginning of your latest K2 host server log (defalut location – “C:\Program Files (x86)\K2 blackpearl\Host Server\Bin”), to verify if it contains the following errors:

In case very beginning of your log contains these error messages:

  • “7030 Unable to create ‘dlx\EventBus’ queue : ‘Message Queue service is not available.'”
  • “7030 Unable to create ‘dlx\EventBus Error’ queue : ‘Message Queue service is not available.'”
  • “EventQueueProcessing.Main”,”7026 Unable to load EventBus Server”

This means that Message Queuing service on K2 server was not running at the time of K2 server start.

To resolve this make sure that Message Queuing service is up and running and restart K2 service – unfortunately there is no other way to reinitialize MSMQ processing thread which gets initialized on K2 service startup.

You may run into this situation after server reboot if K2 service started before Message Queuing service. To prevent this situation from happening you may want to configure K2 service startup to Automatic (Delayed Start) while keeping Message Queuing configured for Automatic start – that will add some delay to K2 service startup as it only will be attempted after all other services configured for Automatic startup will be started, but honestly I don’t see any problem with that – server reboot translates into downtime in any case and slight delay does not make any big difference here. Another, and possible more “fine grained” option to address this is to configure dependency on MSMQ service for K2. To do that run CMD windows in elevated mode and execute the following command:

sc config “K2 blackpearl Server” depend= MSMQ

After successful execution of this command the following dependency for K2 service gets created:

As per MSFT documentation this setting ensures than on machine startup services listed as dependencies are started before attempting to start service which depends on them.

Please follow and like us:
error0

K2 MSMQ thread & MSMQ abort exception

There is one interesting fix included into K2 4.7 May CU which, as far as I know, is not mentioned in related KBs for some reason (KB001888 and  FP KBs). I just wanted to share some information about it as it may be useful for those who run older versions of K2 preceding to 4.7 May CU and additionally mention how to take K2 process memory dump conveniently.
First things first – the issue and related background information. Within K2 process there is a single thread which checks message queue, it is single thread and it process messages one by one (this is by design, as doing in a multi thread fashion is not necessarily best idea). Exception may occur in the process of message bus message processing and this dedicated K2 thread tries to abort this transaction to perform a retry which normally works just fine, but in rare scenarios MSMQ can throw another exception which causes K2 MSMQ processing thread to stop completely.
On the surface symptoms of this issue: suddenly your K2 users no longer receive any K2 task notifications at all, despite the fact that all relevant settings had not been changed and absolutely correct. Brand new test process confirms that task notifications do not work indeed for all usres while Email Events keep working fine. Quick fix/corrective action: restart K2 service – and all delayed task notifications should be gradually dispatched (depending on how long time ago MSMQ thread stopped, your delay for processing the piled queue may be quite big). Service restart resolves this because K2 starts MSMQ thread on service start up.
Good news is that 4.7 May CU has fix for this built in and when transaction abort is necessary MSMQ thread first checks if transaction in question is not already rollback/committed or completed status and only if this is not the case attempts to abort it. If even then K2 receives an exception then message moved into error queue.
I can imagine that when you run K2 version without this fix in production you may be reluctant to restart K2 service not being 99% sure that you deal with this specific issue, and there is a way to do that. To verify that your MSMQ thread within K2 process is still running you may follow these steps:
1) Take full memory dump of K2 process – for quick check on K2 threads either dump taken from Task Manager or procdump.exe will suffice.

In task manager you just have to locate K2HostSerever.exe from the list of processes, right click on it and select “Create dump file”, like that:

It will show you pop up indicated where dump file has been created once done, something like that:

Things gets more tricky with procdump.exe as you now need to obtain process PID, but it is way more configurable and allows you do more things when it comes to taking process memory dumps. As I work in support I really don’t like to repeat explanations on how to use it and where do I find PID and why PID is not displayed in my Task Manager and so on. So I created this tiny script for that (get it on GitHub):

$procid=get-process "K2HostServer" |select -expand id
# -ma - Write a full dump file which includes All the Image, Mapped and Private memory
# -ma - Just takes a full dump immidiately from running process
# -e -ma - tales full dump file on unhandled exception aka crash dump
C:\procdump\procdump.exe -accepteula -ma $procid C:\procdump\k2hostserver.dmp

2) Once dump is taken open it using DebugDiag and search for MSMQ, if MSMQ thread is running you should be able to find something like this:

If you unable to find this in memory dump of your running K2 process it most likely means that MSMQ thread had been stopped due to exception, and if you look under the Previous .NET Exceptions Report (Exceptions in all .NET Heaps) section, you most likely may see the aborted exception with MSMQ stack there.

On a side note DebugDiag also allows you to see if call which failed was made from K2 WorfkflowServer namespace or from something custom/external – that is also very useful to check in the very beginning of troubleshooting process.

I hope you may find this information useful and interesting for one reason or another 🙂 Stay tuned for new posts.

Please follow and like us:
error0