iManage WCS webinar series

|

Last year I’ve attended Autonomy ICSE course at their training centre in Cambridge, UK. All in all it was in a good experience but to my mind couple of issues with this training were poor coverage of WCS technology and even more lapidary description of FileShare (for those who wondering about Worksite Indexer they provide dedicated IDOL trainings which cover full-fledged IDOL solution on which WokrSitie Indexer is based). Whereas this was OK not to dwell much on the FileShare technology which is niche solution and still immature technology (IMHO), poor WCS coverage was a bit unexpected. Probably really in-depth (not targeted to end users) client side components course with drill down to their logic, innards and customizations is missing in Autonomy course offering too, but this is another story…

Recently HP Autonomy / ILTA arranged series of webinars on WCS which more than compensate for this gap in ICSE course (actually after watching parts 1 & 2 I realized that proper coverage of WCS would add 1 more day to ICSE course to give decent coverage of this technology). Anyway this webinar covers all the aspects of WCS and I more than satisfied with it and can recommend it to others as it really may improve your understanding of this solution.

Webinar series comprise of 3 parts (links to recordings are available on WorkSite support portal):

\nPart 1. Email Filing Server – Functional Overview (March 28, 2013)\n\nPart 2. Email Filing Server – Best Practices Design and Deployment (April 23, 2013)\n\nPart 3. Email Filing Server – Troubleshoting (May 23, 2013):\n

Some people still prefer old fashioned reading to webcasts and if you are amongst those you may read what was covered in webinar below (as I jot it down sort of speak). Please read on for Part 1 Email Filing Server – Functional Overview.

Part 1. Email Filing Server – Functional Overview

\n\n

    \n

  1. Email Filing Service\n
      \n

    1. Email Filing Worker

      2. \n

    2. Folder Marking Worker

    2. \n

  2. Mailbox Synchronization Service

    3. \n

  3. Clustering\n
      \n

    1. WorkSite Clustering Service

      2. \n

    2. Load Balancer Service

    4. \n

  4. EFS Management Console

EFS was introduced in 8.5 product line and before that heavy-lifting for email filing was done on a client side. There was a limitation in a number of messages that user can import as it would take about 1 to 3 seconds to file 1 message and mass filing of messages performed on a client side lock the client app. With EFS filing is moved to back-end and became server-server interaction, rather than client-server interaction.

WCS Architecture\n\nEmail Filing Server includes:\n\n0. WCS legacy service which we see in WorkSite server manager – it provides send and file functionality\n\n1. EFS service includes / parent for 2 processes (EmailMgmtSvc.exe – WorkSite Email Management Service):\n\n-       A) Email Filing Worker (individual and bulk email filing)\n\n-       B) Folder Marking Worker (Outlook-to-WorkSite folder synchronization / marked folder processing)\n\n2. Mailbox Synchronization Service aka Mailbox Agent (marking of emails in inbox which were filed by other users). Currently it synchronizes only the Inbox folder.\n\n3. Clustering (for load balancing and failover). Allows for scaling of EFS to enable better filing times and to provide failover.\n\n-       A) WorkSite Clustering Service\n\n-       B) Load Balancer Service\n\n4. EFS Management Console\n

Email Filing Service (EFS)

EmailMgmtSvc.exe (WorkSite Email Management Service) – parent service that controls Email Filing and Marked Folder processing.

–       Starts when Email Filing is started from Admin console

–       Monitors polling intervals for FilingWorker.exe amd MarkingWorker.exe and starts processes on schedule

–       Communicate with EmailMgmtAdmin.exe and LoadBalancer.exe to create the ValidUsers files (ValidUsers.urs and ValidUsers.txt) after user validation is complete.

–       Sends shutdown commands to FilingWorker.exe and MarkingWorker.exe when service is stopped.

Email Filing Service (EFS) – FilingWorker.exe child process of EmailMgtSvc.exe

Email Processing

–       Process that files emails queued via the WorkSite Email Management client

–       Runs on a scheduled basis (schedule is enforced by parent service, default polling interval is 150 seconds)

–       Runs 10 worker threads by default, each making an independent connection to Excange and each processing a single user’s queued messages. When it starts it pick ups the work that available in EM_REQUESTS table and process it till the end, then thread became available for processing of the next mailbox (* software only pick ups 500 jobs per user at one time, built in limitation to avoid 1 user / small group of users bottlenecking server).

FilingWorker.exe Flow

(perfect case scenario, no message has been moved* and no modifications have been made)

\n\n

    \n

  1. EM client queues an email (either via drag and drop or toolbar) and a job is created in EM_PROJECTS table. EM client – DMS transaction. Then DMS server creates job in EM_REQUESTS table which contains failing information required for actual filing.

    2. \n

  2. EM client updates Message Class of the Email to queued on the Exchange. EM Client – Exchange transaction. (custom message class “IPM.Note.Worksite.Ems.Queued” is used to denote the fact that the message has been queued on the Exchange side) At this stage user gets hourglass icon for the message, which means message has been queued.

    3. \n

  3. Filing Worker queries the EM_REQUESTS table on its scheduled interval to retrieve details including Exchange Enrty ID for the message (source) and target WorkSite folder (destination) locations of the Email. DMS – EFS transaction.

    4. \n

  4. Filing Worker connects to Exhange, locates the email by Entry ID, and downloads it. EFS – Exchange transaction. * Perfect case scenario

    5. \n

  5. Filing Worker attempts to import the downloaded message to WorkSite. EFS – DMS trsnsaction.

    6. \n

  6. Filing Worker attempts to update the Message Class to reflect whether the Email filed successfully (“IPM.Note.Worksite.Ems.Filed”). EFS – Exchange transaction.

* Exchange message Entry ID value is location specific, if the message is moved the Entry ID will change. In such cases another property PR_SEARCH_KEY used to find message in Inbox (this search is more costly than locating the message by the Enrty ID). PR_SEARCH_KEY is limited to search only 4 levels deep at the present time (if a user moved message to a folder nested more than 4 levels deep it won’t be located).

PR_SEARCH_KEY is a fallback search mechanism to attempt to lacate messages which have been moved after being filed.

Best Practice: User should be trained don’t move queued message (one marked with hourglass) until it gets a green checkmark icon.

Post Filing

\nUpon a successful import:\n

–       Updates the Message Class of the source message (also in case of “file and delete”, moves message to the Delteted Items of the maiblox owner; you steel see green checkmark on a filed item in Deleted Items)

–       Removes the queued job from the EM_REQUESTS table

Upon an unsuccessful import:

–       Updates the Message Class of the source message to IPM.Note.Ems.WorkSite.Error

–       Updates the Filing Status field of the source message with an error description (e.g. profile error)

–       Writes a detailed error to the FilingWorker log.

–       Removes the queued job from the EM_REQUESTS table (if this is an error service capable to handle gracefully, not an exception type of an error; for exceptions error is written to the log but job may not be removed from the EM_REQUESTS table)

Email Filing Service (EFS) – MarkingWorker.exe child process of EmailMgtSvc.exe, processing folders rather than individual emails

Marked Folder Processing

–       Process which is responsible for syncronizing marked folders (Outlook folder which has been linked to Worksite folders via the EM client, either through File to Worksite tab in folder properies or through the dialogue upon folder creation)

–       Runs on a scheduled basis (360 seconds is default polling interval)

–       Runs 10 worker thread by default, each making an independent connection to Exchange and each processing a single user’s marked folders at a time. (* the same built-in limitation for processing no more than 500 messages per user per cycle as with Filing Worker)

MarkingWorker.exe Flow

\n\n

    \n

  1. EM enabled client creates marked folder and link is created in EM_PROJECTS table. EM Client – DMS transaction

    2. \n

  2. Marking Worker queries the EM_PROJECTS table to retrieve details including Exchange Entry ID for a folder (source) and marked Worksite folder (destination) locations of the Email. DMS – EFS transaction.

    3. \n

  3. Marking Worker connects to Exchange folder by its Entry ID. Identifies new emails by reading the Message Class and custom properties (filing location) of each item, and downloads those identiefied. EFS – Exchange trasaction. * to track non email items stored in Exchange folders (message class other than IPM.Note type e.g. stubs, calendar items) custom Exchange property is being set to denote that is actually been imported.

    4. \n

  4. Marking Worker attempts to import all downloaded messages to WorkSite. EFS – DMS transaction.

    5. \n

  5. Marking Worker attempts to update Message Class to reflect whether the email filed successfully (only for regular emails i.e. objects if IPM.Note class items, other items are being filed but not marked with icons, just marked with propery on background for MarkingWorker but with no visual representation for user – this is by design at the moment). EFS – Exchange transaction.

Post Filing

Upon a successful email import:

–       Updates the Message Class ot the source message to “Filed” (“IPM.Note.Worksite.Ems.Filed”)

Upon an usuccessful email import:

–       Updates the Message Class of failed imports to “Error” (“IPM.Note.Worksite.Ems.Error”)

–       Updates the Filing Status field with error description

–       Writes a detailed error to the MarkingWorker log.

Updates the STATUS field in the EM_PROJECTS table to reflect a successful synchronization or a failure (for folder link, not for the message, eror only if unable access/locate folder not related to filing status of messages inside). Folder status maybe OK where as there are errors with individual emails – this is reflected by yellow icon on failed messages.

Mailbox Synchronization (new feature of 8.5 SP2 & 9.0 versions of WCS)

\nBy default (out of box) this process runs every hour, this can be customized (it doesn’t recommended to set it lower than 5 minutes).\n

Overview

–       Solution for updating filing information on Emails which have already been filed to Worksite by another user.

–       Connects directly both to Exchange and WorkSite to perform synchronization.

–       Uses SQL compact databases to store syncronization information (local to the server where Mailbox Synchronization is running).

–       New feature in WCS 8.5 SP2/9.0

–       Successor to the previous client-side Inbox Sweeper which was included in Email Management client.

–       Runs on a scheduled basis (this service runs on its own & called sweeper.exe, no parent service)

–       Runs 10 worker threads by default, each making an independent connection to Exchange and each processing one user mailbox at a time. For this service number of threads is configurable through registry.

So in total WCS in default configuration makes 30 connection to Exchange: 10 from Filing Worker, 10 from Marking Worker and 10 from Mailbox Synchronization)

Mailbox Synchronization Process – Sweeper.exe

Initial Synchronization

\n\n

    \n

  1. Connects to each user mailbox in Exchange (those in valid users list) and downloads select message properties from all messages in the user’s Inbox (creating “snapshot of Inbox”).

    2. \n

  2. Performs a WorkSite search for all emails (which the user has security access to) added within the First Time Synchronization Horizon (30 days by default), and reads Message ID stored in DOCMASTER table.

    3. \n

  3. Matches Inbox Emails with WorkSite Emails based on Message ID.

    4. \n

  4. Updates Filing Status, Filing Location, and Filing Date for matched items.

    5. \n

  5. Stores synchronization information for each user within SQL compact databases (*.sdf files) located on the server running Mailbox Agent.

Time of initial synchronization is vary quite a bit depending on user’s mailbox size, and alway longer that subsequent synchronizations.

Regular Synchronization

    \n

  1. Performs a worksite search for all emails (which the user has security access to) added since the last synchronization, and reads Message ID stored in DOCMASTER table.

    2. \n

  2. Connects to each user mailbox in Exchange and downloads select message properties from messages added to the user’s Inbox since the last synchronization.

    3. \n

  3. Matches Inbox emails with WorkSite emails based on Message ID.

    4. \n

  4. Updates Filing Status, Filing Location, and Filing Date for matched items.

    5. \n

  5. Stores synchronization information for each user within SQL compact databases (*.sdf files) located on the server running Mailbox Agent

Clustering (Automatic Load Balancing)

Overwiew

–       New feature introduced with the 8.5 SP2/9.0 release of WCS

–       In the previous 8.5 SP1 version of WCS, scaling options were limited to WorkSite database and Exchange Mailbox server filter

–       Eah member node within the cluster processes a subset of Worksite EM users

–       Allows for scaling horizontally by adding the number of nodes necessary to reach the desired level of performance.

–       Utilizes WorkSite Cluster Manager service for communication between nodes

Clustering – WorkSite Cluster Manager Service

imFmaSvc.exe (WorkSite Cluster Manager Service)

–       Facilitates communication between LoadBalancer.exe processes on each node in the WCS cluster

–       Must be installed on each node in the WCS cluster

–       Only required if Automatic Load Balancing is in use

Clustering – WorkSite Load Balancer Service

LoadBalancer.exe (WorkSite Load Balancer Service) load & validate users (making sure that user has valid email address)

–       Starts when the “Connect” button is pressed from within the Admin console (only activated when using Automatic Load Balancing wirh clustering). Service starts in background along with Email Management Service

–       Loads WorkSite users and validates them against the Exchange Address Book or AD to build a list of valid users for EFS to process. Before only AD validation was available, but Exchange Addeess Book validation if faster in most of the cases (especially for large number of users). Exchange Addeess Book validation feature is introduced in WCS 9.0 U2. By default Exchange Address Book validation is turned on, and validation type is configurable through the registry.

–       Communicates with other servers within the cluster to distribute users across member nodes, and detect when a node has been added or removed.

–       Requires WorkSite Cluster Manager service to be installed on all member nodes.

\nRegistry key for setting validation type:\n\nHKEY_LOCAL_MACHINESOFTWAREAutonomyWorksite8.0MailboxAgentExchangeServers\n\nDWORD: AddressBook\n\n1 – Use Exchange Address Book Validation\n\n0 – Use AD validation\n

EFS Management Console

\nWas available in pre 8.5 SP2 versions but with SP2 some changes were added into it.\n

EmailMgmtAdmin.exe (WCS Management Console process)

–       Central location for configuration and management of EFS services

–       Allows admin to view, reset or cancel queued job

–       Allows admin to view, reset, or disable marked folders

–       Allows admin to view and manage service logs (there are separate logs for each of the services – Email Management, Filing Worker, Marking Worker, Load Balancer, EFS Admin Conolse, Mailbox Synchronization). All logs are kept in the same location (C:Program FilesAutonomyWorkSiteServerLogs) except Mailbox Synchronization log (C:Program DataAutonomy).

–       Loads and validates WorkSite users in a single server configuration

WCS Processes Outline

I. Email Filing Service – EmailMgmtSvc.exe\n\na. Email Filing Worker – FilingWorker.exe\n\nb. Folder Marking Worker – MarkingWorker.exe\n\nII. Mailbox Synchronization Service – Sweeper.exe\n\nIII. Clustering\n\na. WorkSite Clustering Service – imFmaSvc.exe\n\nb. Load Balancer Service – LoadBalancer.exe\n\nIV. EFS Management Console – EmailMgmtAdmin.exe

Final Thoughts

Here are a couple of questions that we would like you to think about before we meet for the next session.

 

–       EFS is an asynchronous process. What value is added by increasing the number of Email Filing Servers (scaling out)?

–       We have no ability to limit the amount of email that a user may submit for processing. What challenges does this present to the design, implementation, and administration of the system.

Please look for transcript of parts 2 and 3 of this webinar in further posts:

iManage WCS webinar series – Part 2

iManage WCS webinar series – Part 3

Leave a Reply

Your email address will not be published. Required fields are marked *