There is one interesting fix included into K2 4.7 May CU which, as far as I know, is not mentioned in related KBs for some reason (KB001888 and FP KBs). I just wanted to share some information about it as it may be useful for those who run older versions of K2 preceding to 4.7 May CU and additionally mention how to take K2 process memory dump conveniently.
First things first – the issue and related background information. Within K2 process there is a single thread which checks message queue, it is single thread and it process messages one by one (this is by design, as doing in a multi thread fashion is not necessarily best idea). Exception may occur in the process of message bus message processing and this dedicated K2 thread tries to abort this transaction to perform a retry which normally works just fine, but in rare scenarios MSMQ can throw another exception which causes K2 MSMQ processing thread to stop completely.
On the surface symptoms of this issue: suddenly your K2 users no longer receive any K2 task notifications at all, despite the fact that all relevant settings had not been changed and absolutely correct. Brand new test process confirms that task notifications do not work indeed for all usres while Email Events keep working fine. Quick fix/corrective action: restart K2 service – and all delayed task notifications should be gradually dispatched (depending on how long time ago MSMQ thread stopped, your delay for processing the piled queue may be quite big). Service restart resolves this because K2 starts MSMQ thread on service start up.
Good news is that 4.7 May CU has fix for this built in and when transaction abort is necessary MSMQ thread first checks if transaction in question is not already rollback/committed or completed status and only if this is not the case attempts to abort it. If even then K2 receives an exception then message moved into error queue.
I can imagine that when you run K2 version without this fix in production you may be reluctant to restart K2 service not being 99% sure that you deal with this specific issue, and there is a way to do that. To verify that your MSMQ thread within K2 process is still running you may follow these steps:
1) Take full memory dump of K2 process – for quick check on K2 threads either dump taken from Task Manager or procdump.exe will suffice.
In task manager you just have to locate K2HostSerever.exe from the list of processes, right click on it and select “Create dump file”, like that:
It will show you pop up indicated where dump file has been created once done, something like that:
Things gets more tricky with procdump.exe as you now need to obtain process PID, but it is way more configurable and allows you do more things when it comes to taking process memory dumps. As I work in support I really don’t like to repeat explanations on how to use it and where do I find PID and why PID is not displayed in my Task Manager and so on. So I created this tiny script for that (get it on GitHub):
$procid=get-process "K2HostServer" |select -expand id # -ma - Write a full dump file which includes All the Image, Mapped and Private memory # -ma - Just takes a full dump immidiately from running process # -e -ma - tales full dump file on unhandled exception aka crash dump C:\procdump\procdump.exe -accepteula -ma $procid C:\procdump\k2hostserver.dmp
2) Once dump is taken open it using DebugDiag and search for MSMQ, if MSMQ thread is running you should be able to find something like this:
If you unable to find this in memory dump of your running K2 process it most likely means that MSMQ thread had been stopped due to exception, and if you look under the Previous .NET Exceptions Report (Exceptions in all .NET Heaps) section, you most likely may see the aborted exception with MSMQ stack there.
On a side note DebugDiag also allows you to see if call which failed was made from K2 WorfkflowServer namespace or from something custom/external – that is also very useful to check in the very beginning of troubleshooting process.
I hope you may find this information useful and interesting for one reason or another 🙂 Stay tuned for new posts.