Curious case of stubborn Workgroup flag

I’ve just recently got a support case from the client where no matter that we tried MSMQ won’t work in Directory Service Integration mode, resulting in the following warning from K2 blackpearl Setup Manager:

MSMQ component is a prerequisite for K2 and it seems that all you need to do covered in K2 documentation: Installation and Configuration Guide > Prepare > Supporting Components Configuration > Message Queuing (MSMQ). As an example of functionality which will be impacted by this issue I can mention task notification emails as those have to be queued in MSMQ before being picked up by the Eventbus.

Yet information there was not sufficient, and required me to do some prolonged troubleshooting and googling. But the thing is that MSFT documentation does not highlight root cause which I found either and only some old obscure blog post lead me in the right direction.

So according to K2 documentation we have at the moment we supposed to set permissions and instead of outlining what permissions to set and where, KB strangely starts from where to set them, omitting WHAT & WHY parts 🙂 Moreover it suggests to set permissions on what its called “root object” and according to KB documentation means MSMQ container itself:

But this is not possible at all – this Root Object does not exist before you installed MSMQ Directory Integration Service, and if was installed correctly there is no need for you to go and set any permissions. But if it was installed but not in Directory Integration mode it won’t have its properties exposed in AD:

For WHAT & WHY parts K2 documentation elegantly refers you to MSFT TechNet. There you have to navigate and do a bit of careful reading to find out that setting permissions in AD DS required before installing Directory Service Integration Features of Message Queuing (why not start with this in K2 documentation?).

MSFT KB says that special permissions have to be set IF you installing Directory Service Integration feature of Message Queuing on a domain controller (we can safely ignore this as we need to install feature on K2 server and we won’t be installing K2 on domain controller, except for “all-in-one” test box scenario). Next MSFT KB says that you have to grant the Network Service account the Create MSMQ Configuration Objects permission to the computer object in AD DS before installing the Directory Services Integration feature on a computer that is a domain controller. So all in all according to MSFT KB you only need to set permissions in case you installing MSMQ Directory integration on DC. So no need for extra permissions, right? There are some required permissions but normally they are in place by default.

To save you pain making sense of all this documentation you have to mess with permissions only when installing MSMQ on domain controller, period. But recently I had a support case where this just does not want to work and after unsuccessful troubleshooting attempts with client I revert to my test lab and run into the same issue there. The fact that I run into it only on one server tells me that it only happens if your computer ACL was somehow customized or locked down either intentionally or not. If that pesky Workgroup flag keeps reverting to 1 despite you installed Directory Service Integration Feature, please make sure that SELF identity has the following rights on your K2/MSMQ server in AD:

So in short K2 KB should be structured like that:

1. K2 requires MSMQ Server and MSMQ Directory Service Integration components to be installed. DS integration improves security as it enables publishing queue properties to AD, authentication and encryption of messages using certificates registered in the directory.

2. Before installing MSMQ on domain controller machine you may need to grant additional permissions as described in Microsoft documentation: https://technet.microsoft.com/en-us/library/cc730960(v=ws.11).aspx

NOTE: this is only applicable for “all-in-one” test servers where you may want K2 to coexist on the same machine with DC. This is not applicable for production deployments where K2 runs on the dedicated AD DS member server.

3. In case you doing normal installation on AD DS member server you usually do not have to grant any special permissions except for if you security ACL for K2 server is customized. If you installed MSMQ Directory Service integration on domain member server and Workgroup flag reverting to 1 all the time then check that SELF identity has Create MSMQ Configuration objects and Delete MSMQ Configuration objects rights granted over your K2 server computer object:

NOTE: (1) My tests show me that it is sufficient to apply these permissions to “This objects only”, but you should understand that enabling disabling Directory Service Integration feature requires restart of the machine, so it seems it is all about rights at the point of installation and first reboot after it to create MSMQ objects correctly – I noticed that if I delete MSMQ objects, revoke rights and even reboot the machine Workgroup flag keep staying set to 0, but reinstalling MSMQ feature reveals this problem again. Granting rights sometimes work on the fly without reinstalling MSMQ – just restart the service.

(2) These advanced permissions may become messy way to quickly as you inherit them from your domain and then each time you click on Add button separate ACL entry is being created for your computer object, so watch out for explicit Deny settings overriding your Allow grants.

(3) As usual monitor Administrative Events view in Event Viewer and watch out for MSMQ related errors – it should be able to tell you what’s wrong.

4. When installing K2 on standalone servers (not joined to domain) or in cases when you unable to make Directory Service Integration work you can consider using workgroup mode and private queues. Though if latter is the case it is much better investigate and resolve your Directory Service Integration issue.

Apart from possibility of better structured related information in K2 documentation I’m wondering why MSFT does not have full set of permissions required for Directory Service Integration mode published somewhere? It seems this issue with lacking rights for SELF identity does not happen as by default in Windows Server 2012 R2 domain on newly provisioned server its ACL entry looks as follows:

Hope this article will save someone some time which may be wasted on troubleshooting this.

UPDATE: And as for original support case, there issue persisted even after we granted all required rights to SELF object. At this point we identified that MSMQ logs the following error with code 0xc00e050f, which indicated that MsmqServices object was missing in client’s AD DS domain Configuration partition (though to find this out it was necessary to read some forum t thread created 13 years ago and relevant for Windows Server 2003 🙂 ). Here is the steps how to verify its presence and correct this if necessary:

1.Run ADSIEdit

2. Expand the the configuration container, then expand Services. Check whether MsmqServices object and the regular MSMQ object are there

3. If the MsmqServices object is missing, right-click on Services and select “New Object”. For the object type, select mSMQEnterpriseSettings, for the object name use MsmqServices

4. Save the changes and close ADSIEdit, and force AD DS replication to avoid situation when machine where MSMQ is being installed queries some other DC than the one where you made this modification

5. Retry the MSMQ installation, it should now succeed.