Tag Archives: Troubleshooting

K2 – no task notifications processing due to “Unable to load EventBus Server” error

K2 versions older than Five rely on MSMQ for task notification emails queuing and processing. Sometimes you may see the following scenario:

You start process which supposed to send task notification. Notification not delivered and there is no error messages logged nor in K2 host server log, MSMQ diagnostic log nor in Eventbus.ClientRecorderError table logged at the time of task assignment. Obviously process does not go into error state as notification delivery performed on “best possible effort” principle and not preventing user from completing assigned task using his worklist which can be exposed through different UIs.
At the same time it is possible to see that queue exist and messages gets queued into it (Computer Management snap-in > Services and Applications > Message Queuing > Eventbus > Queue Messages:

If it is not highly loaded production environment you will be able to see that message gets queued per each client event, and you can also see that Message ID gets assigned with increment of 2, with new messages being added to the end of the queue. At this stage you may think that Message Queuing may not run – but this is unlikely as when this service stopped message cannot be placed into queue (that definitely will be logged on K2 side) and you unable to view Queue contents in computer management with the following error:

Other possibility is somehow malformed error message and some glitch in MSMQ service itself – again a bit unlikely as new messages gets queued just fine, but you can try Message Service restart along with removing topmost message from the queue. If that does not help (which most likely will be the case), check the very beginning of your latest K2 host server log (defalut location – “C:\Program Files (x86)\K2 blackpearl\Host Server\Bin”), to verify if it contains the following errors:

In case very beginning of your log contains these error messages:

  • “7030 Unable to create ‘dlx\EventBus’ queue : ‘Message Queue service is not available.'”
  • “7030 Unable to create ‘dlx\EventBus Error’ queue : ‘Message Queue service is not available.'”
  • “EventQueueProcessing.Main”,”7026 Unable to load EventBus Server”

This means that Message Queuing service on K2 server was not running at the time of K2 server start.

To resolve this make sure that Message Queuing service is up and running and restart K2 service – unfortunately there is no other way to reinitialize MSMQ processing thread which gets initialized on K2 service startup.

You may run into this situation after server reboot if K2 service started before Message Queuing service. To prevent this situation from happening you may want to configure K2 service startup to Automatic (Delayed Start) while keeping Message Queuing configured for Automatic start – that will add some delay to K2 service startup as it only will be attempted after all other services configured for Automatic startup will be started, but honestly I don’t see any problem with that – server reboot translates into downtime in any case and slight delay does not make any big difference here. Another, and possible more “fine grained” option to address this is to configure dependency on MSMQ service for K2. To do that run CMD windows in elevated mode and execute the following command:

sc config “K2 blackpearl Server” depend= MSMQ

After successful execution of this command the following dependency for K2 service gets created:

As per MSFT documentation this setting ensures than on machine startup services listed as dependencies are started before attempting to start service which depends on them.

Please follow and like us:
error0

Unable to connect to named SQL instance on remote machine

Recently I was doing installation of K2 5.2 on Azure VMs with SQL server named instance hosted on separate Azure VM. I’ve created SQL Server alias on K2 VM but then run into issue – neither K2 Setup Manager nor SSMS were able to connect to SQL through alias. I next tried direct connection via server\instance name which also failed. SSMS showed me the following error:

Could not open a connection to SQL Server. Microsoft SQL Server, Error: 53

I first focused on network connectivity between VM:

  • Confirmed that I can ping SQL Server VM from K2 Server VM
  • Confirmed that no firewall enabled on VM and Azure VMs on the same network with nothing blocking connectivity between them
  • I tried to use telnet to test port 1433 – it failed

This is what kept me focused on network connectivity layer a bit longer then necessary. But after confirming that SQL Server not listening on port 1433 using netstat -na | find “1433” it became quite clear that focus should be on SQL Server configuration. First of all – by default named instance listen on dynamic port, and you actually need to have SQL Server Browser Service enabled to ensure you can connect to named instance without specifying port while using dynamic ports. But in my case it was not that as in SQL Server configuration there was explicitly specified custom port (SQL Server Configuration Manager > Protocols for %INSTANCE_NAME% > TCP/IP Properties > TCP Dynamic Ports – if you have anything other than 0 in IPAll section fir this setting you are not using dynamic ports). When your problem is dynamic ports and disabled SQL Server Browser Service error message from SSMS looks as follows:

SQL Network Interfaces, error: 26 – Error Locating Server/Instance Specified

As you can see error message explicitly tells you “Error Locating Server/Instance Specified. To fix this either set 0 for TCP Dynamic Ports setting and enable SQL Server Browser Service or specify some port number there. You sort of choosing your dependency here – either browser service (may fail to start) or custom port (may be hijacked by other service). It seems that browser service is better approach.

So in my case I was confused by expecting named instance to listen on default port which was, to put it simply, wrong expectation. Here is how you can check on which port your instance is listening:

Commands which you can use to find out your SQL Server instance ports

But obviously having access to SQL Server you can get this data from SQL Server Configuration manager too: SQL Server Configuration Manager > Protocols for %INSTANCE_NAME% > TCP/IP Properties. Just keep in mind that you need to check TCP Dynamic Ports value both for specific address and for IPAll section. But like I said in my case, the problem was not about ports. Once I found out instance port I noticed that I still cannot connect to it using telnet, just because IP address was not enabled in SQL Server Configuration Manager > Protocols for %INSTANCE_NAME% > TCP/IP Properties (meaning it had Enabled=0). I corrected that and telnet connectivity test succeeded.
Still, when I get back to SSMS I was getting the same error – “Could not open a connection to SQL Server. Microsoft SQL Server, Error: 53”. Reason? With SQL Server 2016 and latest versions of SQL Server, I keep forgetting that the latest and greatest version of SSMS still reads alias settings from x86 registry hive (meaning you need to configure SQL alias using cliconfg.exe from C:\Windows\SysWOW64) – I have a hard time getting use to it. Interestingly fully missing x86 alias triggers error message “Could not open a connection to SQL Server. Microsoft SQL Server, Error: 53” while one we configure with non existing server or instance name will give you “SQL Network Interfaces, error: 26 – Error Locating Server/Instance Specified”.

Anyhow your key takeaways from this post should be:

  • Know your instance port
  • Make sure that IP address is enabled
  • We still need to configure alias twice (x86/x64) to avoid unpleasant surprises from apps reading setting from non-configured location

I hope this post may save some troubleshooting time for someone.

Please follow and like us:
error0

“Method not found error” when attempting to add a new item into K2 Studio project

The other day I was about to build some tiny K2 test project using old-fashioned thick designer UI K2 Studio which I still prefer to use whenever I need to build some little K2 process. Unfortunately I bumped into this error:

Error message politely informs us about this:

Method not found: ‘Microsoft. Build. BuildEngine

BuildItem SourceCode. ProjectSystem. ProjectBuildItem

get_BuildItem()’.

That’s not very obvious, right? But, trust me it just complains that some DLL is not properly added to GAC, or there is a mismatch between DLL version in GAC and in some other location.

Solution? If you run K2 environment without coldfixes just run a Repair from installation media. But most likely you’ve applied some coldfix recently. Verify if files which supposed to go to GAC were added to that location correctly. Remember that some files go to K2 installation directory, while some others may go to “[Program Files]Reference Assemblies\SourceCode\v4.0\” and into GAC v4, i.e. into “C:\Windows\Microsoft.NET\assembly\GAC_MSIL\”. In my case the error was caused by the fact that SourceCode.Workflow.Authoring.dll assembly was not updated in GACv4.

When something wrong with aforementioned DLL you will also see the same error when trying to deploy something with PnD:

So in case you getting this type of error you know what to check now.

Please follow and like us:
error0

Unable to start SSRS sevice after upgrading SQL Server instance

I recently spent quite a few time trying to figure out why my SSRS service fails to start after performing upgrade from SQL Server 2012 to SQL Server 2016. Service was failing to start with quite generic error “System error 5 has occurred. Access is denied” which is a bit to broad, but after a while I looked into instance bin directory which contains log file where I seen some error indicating that configuration file is garbled somehow. At this point I revert to reinstall once again solution, and this time I noticed warning which I probably missed during initial instance upgrade attempt:

TITLE: Microsoft SQL Server 2016 Setup

——————————

The following error has occurred:

Index was out of range. Must be non-negative and less than the size of the collection.

Parameter name: index

For help, click: http://go.microsoft.com/fwlink?LinkID=20476&ProdName=Microsoft%20SQL%20Server&EvtSrc=setup.rll&EvtID=50000&ProdVer=13.0.1711.0&EvtType=0x35C10BED%25400xF538E98B

I proceed with upgrade I seen exactly the same problem – I can’t start SSRS service after upgrade completion. This warning essentially lead me to the thread on TechNet containing right solution. It seems that upgrade process somehow can’t handle SSL bindings for SSRS service or handles them differently (produces malformed XML configuration for them?) and what you need to do is remove SSL bindings for SSRS in Reporting Services Configuration Manager (see screenshot below).

Just to save some time which may be wasted on troubleshooting just remove your SSL bindings (I did it for Web Service URL & Web Portal URL) before starting upgade of your instance. You may add these bindings back after upgrade.

Please follow and like us:
error0

Restore missing K2 Server Console Mode Shortcut

Recently I’ve seen failed K2 blackpearl installation which went seemingly well but logged an error on Configuration Analysis stage for  K2 blackpearl Server Running task. The warning message was quite helpful listing all possible reasons which may cause issues for K2 service startup: disabled service or service account, wrong credentials, missing logo as a service rights for K2 service accounts or conflicting ports configuration (required ports are busy by something else). And we all know that we have detailed setup logs to check what went wrong during installation, right?

Nonetheless assuming you focused on inability to start K2 service, then as it fails even before initializing itself and its logging which would be the case in scenario with aforementioned warning in K2 blackpearl Configuration Analysis (Windows Services snap-in will show you “Service on local computer has been started then stopped” message), your first action should be attempt to run it in console mode and see what exactly is wrong/what it complains about 🙂

Unfortunately in my specific case Setup Manager also failed to create K2 Server Console Mode Shortcut, which should give you a hint that this is likely credentials issue. In case K2 Server Console Mode Shortcut is missing it can be created manually using the following command:

C:\Windows\System32\runas.exe /user:{K2ServiceDomain\Account} /savecred "C:\Program Files (x86)\K2 blackpearl\Host Server\Bin\K2HostServer.exe"

Refer to related section of K2 help for details. Simply by looking at the command above there are not a lot of things which can go wrong here – missing service executable file (definitely should lead to more warnings on the final stages of setup) or wrong credentials (those in theory can be locked during final stages of setup at the stage when it tried to register service/create shortcut). Anyhow if you run into something like this just restore console mode shortcut to investigate your problem with service startup further.

Please follow and like us:
error0