Category Archives: iManage

iManage WCS webinar series – Part 3

This is the transcript of 3rd part of iManage WCS webinar series (webinar recording date May 23rd, 2013)\n\nPart 3 – Troubleshooting\n

    \n

  1. I.                    Introduction
      \n

    1. Functional Review
  2. \n

  3. II.                  Email Filing Server
      \n

    1. Monitoring queues
    2. \n

    3. Logging
    4. \n

    5. Configuration files
    6. \n

    7. Failures/Errors
  4. \n

  5. III.                Mailbox Sync
      \n

    1. Logging
    2. \n

    3. SQL Compact databases
    4. \n

    5. Control Panel/Simulator
  6. \n

  7. IV.                Summary

Introduction – Functional review\n\n \n\nEmail Filing Server\n\n \n\n-          FilingWorker.exe (process) – Processes server-side filed emails. Reads queued jobs from the EM_REQUEST table\n\n-          MarkingWorker.exe – Processes marked folders. Reads folders from the EM_PROJECTS table\n\n-          EmailMgmtSvc.exe (service) – Manages startup and shutdown of the FilingWorker and MarkingWorker processes\n\n-          EmailMgmtAdmin.exe – EFS Management Console. Validates users in a single server configuration\n\n-          LoadBalancer.exe (service) – Validates and distributes users in a clustered configuration\n\n-          imFmaSvc.exe (service) – WorkSite Cluster Manager service used for communication between nodes by LoadBalancer\n\nMailbox Sync\n\n \n\n-          Sweeper.exe (service) – Adds filing information to Inbox emails which have already been filed to WorkSite.\n\nEmail Filing Server – Monitoring Queues\n\n \n\nThis may be useful both for troubleshooting and general monitoring\n\n \n\nFiling Request Management\n\n-          Displays filing jobs queued in the EM_REQUESTS table\n\n-          Shows job details including type (email or folder) and submitted time\n\n-          Allows admin to reset or cancel jobs\n\nMarked Folder Management\n\n-          Displays marked folders stored in the EM_PROJECTS table\n\n-          Shows synchronization status and last sync time\n\n-          Allows admin to reset or disable folders\n\nEmail Filing Server – Configuration Files\n\n \n\nEFS Configuration Files\n\n \n\nAll EFS configuration files are written to …AutonomyWorkSiteServerConfig\n\n-          DmsSettingsX.config – Contains DMS connection information. One file will exist for each DMS connection made by EFS (i.e. DmsSettings0.config, DmsSettings1.config)\n\n-          ExchangeSettings.config – Contains Exchange connection information, including service account credentials used to access mailboxes.\n\n-          FilingOptions.config – Contains settings which define how profile metadata is applied to filed items\n\n-          SvcSettings.config – Contains settings which define how EFS processes behave, including number of threads and polling interval\n\nEFS ValidUsers Files\n\n \n\n-          ValidUsers.urs (contains extended information including some details on users’ mailboxes) and ValidUsers.txt (stripped down copy of .urs file) – Contains list of users which were validated for processing (created upon automated/manual load balancing)\n\n \n\n* all these config files can be backed, for example before you make any changes during troubleshooting process\n\nEFS Log Information\n\n \n\n-          All EFS log are written to …AutonomyWorkSiteServerLogs\n\n-          Individual logs are created for each process – FilingWorker.exe, MarkingWorker.exe, EmailMgmtSvc.exe, EmailMgmtAdmin.exe, LoadBalancer.exe, imFmaSvc.exe\n\n-          For each process, a new log will be created on the next startup after the existing log reaches 2MB. If the active log exceeds 40MB, a new log will be created immediately\n\n-          Each process will retain the 50 most recent logs before overwriting\n\n-          Log Filter level is set to Verbose by default\n\n-          Log monitor can be used to view log writes in real-time\n\nEmail Filing Server – Log Information\n\n \n\nWhere to Look for Logging\n\n \n\nFilingWorker.log – Contains information about filing job processing, including downloading the item(s) from Exchange and importing them into WorkSite.\n\nContains filing error messages – server side filing requests from EM_REQUESTS table, e.g. profile error etc. In additional to this log you usually see detailed error information in a filing status column on a client side.\n\nMarkingWorker.log – Contains information about marked folder processing, including downloading the folder contents from Exchange and importing item(s) into WorkSite. E.g. errors with filing of individual message within the marked folder.\n\nEmailMgmtSvc – Contains information on service startup and continuous cycling of the worker processes.\n\nContains information about general server issues, startup failures. This service control worker processes, so it’s your destination if you need to check why marking or filing worker not starting or not happening as quickly as they should be.\n\nEmailMgmtAdmin – Contains information on what is displayed in the EFS management Console. Also displays user validation information in a single server configuration.\n\nIf you are not using clustering component  with automatic load balancing then all the logic for the user validation is within this executable. When you click on a Load Balancing button on a non clustered system all the user validation happens in this console and all the output for failures and successes during that validation will be written to EmailMgmtAdmin.log\n\nLoadBalancer – Contains information on user validation and distribution in a clustered environment.\n\nContains data about users validation on clustered systems only, plus about users distribution and number of cluster nodes.\n\nimFmaSvc – Contains information on the communication between nodes in a clustered environment.\n\nMay contain errors related with joining nodes to the cluster and communication between nodes.\n\nEmail Filing Server – Reading Logs\n\n \n\nUnderstanding the FilingWorker Logs\n\n \n\nBy default, FilingWorker will launch 1 main thread and 10 worker threads (configurable). Each line in the FilingWorker log indicates the thread number which is writing the line. Each log will include the following information:\n

    \n

  1. The main thread (thread #1) will write startup information, load and validate queued jobs from the EM_REQUESTS table for valid users, and initialize worker threads.
  2. \n

  3. Each worker thread (thread #5-14) will connect to a user mailbox, attempt to process the user’s queued items (500 max)
  4. \n

  5. As worker threads complete work for each user, the main thread will check the EM_REQUESTS table for more work
  6. \n

  7. Once no additional work is available, the main thread will update filing statuses, remove completed jobs from the EM_REQUESTS table, shutdown all worker threads and kill FilingWorker.exe

\nIt is convenient to use as a log viewer something like Notepad++, which allow you to highlight thread numbers to more easily follow them and has good search capabilities.\n\nUnderstanding the MarkingWorker Logs\n\n \n\nBy default, MarkingWorker will launch 1 main thread and 10 worker threads (configurable). Each line in the MarkingWorker log indicates the thread number which is writing the line.\n\nEach log will include the following information:\n

    \n

  1. The main thread (thread #1) will write startup information, load and validate marked folder links from the EM_PROJECTS table for valid users, and initialize worker threads.
  2. \n

  3. Each worker thread (thread #5-14) will connect to a user mailbox, attempt to process the user’s marked folders. The status of each folder will be updated during processing.
  4. \n

  5. Once all users are complete, the main thread will shutdown all worker threads and kill MarkingWorker.exe

Understanding the LoadBalancer Logs\n\n \n\nThe LoadBalancer logs will capture information on user validation process. For users which are successfully added to the ValidUsers file for processing, minimal information is captured. For failures in validation, more detail will be recorded. Each log will show following:\n

    \n

  1. Write startup information
  2. \n

  3. Retrieve all enabled users with properly configured email address in WorkSite.
  4. \n

  5. Validate all users against AD or Address Book (by default Address Book validation is turned on)
  6. \n

  7. Report user distribution

Email Filing Server – Failures/Errors\n\n \n\nFilingWorker Processing\n\n \n\nSome issues which can cause filing failures include:\n\n-          Profile errors – Refer to FilingWorker log for details. Example:  Cannot Import the message: \n\nProfile Error: AttID=imProfileCustom1, Value=TEST, Desc=The field is disabled\n\n-          Filing user not validated – Refer to LoadBalancer log for clustered environments, EmailMgmtAdmin log for single server environments. Example:\n\nThe user account with the Email Address: USER@DOMAIN.COM failed agains Active Directory\n\n-          Filed item cannot be located using ENTRY_ID or PR_SEARCH_KEY – Refer to Filing Worker log for details. Example:\n\nFiling Error…System error. <spMsgStore.OpenEntry: EID=123456789> NotFound at Interwoven.WorkSite.Ems.ThFilingProcessor.ThEfsConroller.ProcessFilingRequest(ThConfigData&ConfigData,ThDMS& Dms, ThRequest& tfr, IMsgStore, IMAPISession spSession, Boolean bDoNotCheck, List’1& vDouble, ThSlot& Slot)\n\n \n\nMarkingWorker Processing\n\n \n\nSome issues which can cause marked folder failures include:\n\n-          Profile errors – Refer to MarkingWorker log for details. Example:\n\nCannot Import the message: Profile Error: AttID=imProfileClass, Value=, Desc=Field is required\n\n-          Filing user not validated – Refer to LoadBalancer log for clustered environments, EmailMgmtAdmin log for single server environments. Example:\n\nTESTUSER does not have an E-Mail address\n\n-          Folder cannot be located using ENTRY_ID – Refer to FilingWorker log for details. Example:\n\nSystem error.<spFolder.GetProps> NotFound sAddInfo: <> at Interwoven.WorkSite.Ems.ThFilingProcessor.ThEfsController.GetLatestMessages(IMAPISession spSession, IMsgStore spMsgStore, ENTRYID FolderEID, MarkedFolder& mf, List’1& vEID, ENTRYID& ParentFolderEID, String sAddMsgClass)\n\nNOTE: Marked folder status is only updated to Successful or Failed based on whether the process was able to locate the folder and retrieve the contents. Individual item filing successfully or not does not reflect the folder status.\n\nMailbox Synchronization – Log Information\n\n \n\nMailbox Synchronization Logging Information\n\n \n\n-          Synchronization Log Monitor displays Mailbox Sync log activity in real-time.\n\n-          Log files are written to the following location:\n\nWindows 2008 Server:\n\nC:ProgramDataAutonomyWorkSiteMailboxAgentLogsMirrorLogs\n\nWindows 2003 Server:\n\nC:Documents and SettingsAll UsersAutonomyWorkSiteMailboxAgentLogsMirrorLogs\n\n-          MirrorLogs directory is only written after a successful initial synchronization\n\n-          During processing, all log data is written to SweeperLog.sdf\n\n-          SweeperLog.sdf data is written to log files in MirrorLogs upon each cycle completion\n\nMailbox Synchronization – Data\n\n \n\nMailbox Synchronization SQL Compact Files\n\n \n\n-          Mailbox Synchronization maintains separate SQL Compact data files for each user in the following location:\n\nWindows 2008 Server:\n\nC:ProgramDataAutonomyWorkSiteMailboxAgentDataStorage\n\nWindows 2003 Server:\n\nC:Documents and SettingsAll UsersAutonomyWorkSiteMailboxAgentDataStorage\n\n-          SQL Compact user files maintain a MSG ID database of the Inbox contents\n\n-          Service logging is written to SweeperLog.sdf during processing\n\n \n\nResetting Sync Data\n\n \n\n-          All synch data can be reset by stopping EFS and Mailbox Synchronization services and renaming.\n\nC:ProgramDataAutonomyWorkSiteMailboxAgent. This will force the initial synch to take place again.\n\n-          Individual users can be reset by renaming individual user SDF files.\n\n \n\nMailbox Synchronization – Control Panel\n\n \n\nControl Panel Functions\n\n \n\n-          Synch – Force an immediate synchronization without waiting for the polling interval (default, out of box polling interval equals to 1 hour)\n\n-          Validate Users – Validates the users in the WorkSite group used for Mailbox Synchronization (single server configuration only, in clustered configurations by default Mailbox Synchronization ignores group and perform processing for all users each node is balanced for)\n\n-          Select Mailbox – Lists all validated mailboxes for which Mailbox Synchronization is currently processing\n\n \n\nMailbox Synchronization – Simulator\n\n \n\nMailbox Operations Simulator\n\n Mailbox_Ops_Simulator

–          The Mailbox Operations Simulator can be used to simulate individual Mailbox Synchronization activities against specific users

–          Helpful for troubleshooting problems with specific users as well as general issues with Mailbox Synchronization

–          Does not update data, only simulates activity and displays output

Summary\n\nTroubleshooting Email Filing Server\n\n \n\n-          Check the queue – Make sure the queued job or marked folder shows in queue\n\n-          Check the user – Verify that the filing user is validated on the server\n\n-          Check the logs – Look in the appropriate service log for errors\n\n \n\nTroubleshooting Mailbox Synchronization\n\n-          Check the user – Make sure user mailbox is visible\n\n-          Check the logs – Check the logs for errors\n\n-          Test – Use the Mailbox Operations Simulator

Please follow and like us:

iManage WCS webinar series – Part 2

This is the transcript of 2nd part of iManage WCS webinar series.\n\nPart 2. Email Filing Server – Best Practices Design and Deployment\n\nSizing & deployment strategies.\n\nAgenda.\n

    \n

  1. Introduction\n
      \n

    1. EFS Review
    2. \n

    3. Performance Defined
    4. \n

    5. Performance Variables
  2. \n

  3. Filing Time\n
      \n

    1. EFS Proximity to Exchange
    2. \n

    3. EFS Proximity to WorkSite DMS
    4. \n

    5. DMS Proximity to SQL
  4. \n

  5. Scaling for Improved Performance
  6. \n

  7. Summary
  8. \n

  9. Sizing scenarios\n
      \n

    1. Case Study for Central Exchange Architecture
    2. \n

    3. Case Study for Distributed Exchange Architecture

Introduction – EFS Review

Email Filing (Filing Worker) is asynchronous (i.e. filing is not real time)

User flags a message to be filed in Outlook

On the Email Filing Server, the Filing Worker process attempts to file the message on its next iteration. All messages are filed in batches.

The amount of time that Filing Worker spends on each iteration is a product of:

–       The number of messages that need to be processed

–       The amount of time it takes to process each message

An Admin will have no direct control over the number of messages that his/her users are filing.

Marked Folder Processing (Marking Worker) is asynchronous

–       User maps (marks) an Outlook folder to a WorkSite folder.

–       MarkingWorker attempts to process the folder on next iteration. The MarkingWorker processes folders in batches. All folders are checked each iteration.

–       For each folder, MarkingWorker determines the messages that haven’t been filed and processes these one at a time.

–       The amount of time that MarkingWorker spends on each iteration is a product of

\n\n

    \n

  • The number of marked folders that need to be processed
  • \n

  • The number of messages that need to be processed
  • \n

  • The amount of time it takes to process each message

–       An admin will have no direct control over the number of folders that his/her users are marking, nor does he/she have control over the number of messages that are in each Outlook folder that get’s processed

Introduction – Performance Defined

Q: What is the single most important measure of performance for Email Filing Server?

A: The amount of time it takes EFS to process a batch.

–       Messages or Marked Folders are processed in batches. The faster the batch completes, the faster the messages in the batch complete. The faster the next batch can be processed.

–       The amount of time that a message takes to file after a user has flagged the message as a WorkSite message (either by placing it in a WorkSite folder, or by putting it in a Marked Outlook folder)

Introduction – Performance Variables\n\n-       Email Filing\n

    \n

  • Number of messages being processed in this batch
  • \n

  • Single message filing time

\n-       Marked Folder Processing\n

    \n

  • Number of Marked Folders
  • \n

  • Number of messages that need to be filed from each Outlook folder
  • \n

  • Single Message filing time

\nThere are methods to modify each of these variable in order to have a positive impact on message filing time.\n\nFiling Time – Proximity of EFS to Exchange\n\nEFS proximity to Exchange has a significant impact on performance\n\n-       The time it takes EFS to file one or more queued items to WorkSite (appear in the WorkSite filing location and/or see updated filing information on the original item) after they’ve been successfully queued.\n\n-       The time it takes EFS to file items to WorkSite after being added to a Marked Folder\n\nThe performance of the following filing functions will improve with EFS located geographically close to the user’s Exchange mailbox:\n\n-       Connect to user mailboxes in Exchange\n\n-       Locate message or folder in Exchange for Filing or Marking process\n\n-       Search mapped Exchange folders to retrieve new content added to marked folders\n\n-       Copy message from Exchange to EFS server for import into WorkSite\n\n-       Update filing information values on Emails to reflect the filing status (Filed, Queued, Failed…)\n\nFiling Time – EFS proximity to WorkSite\n\nThe perfromance of the following filing functions will improve with EFS located geogrpaphically close to the DMS server:\n\n-       Load and validate filing jobs and marked folders\n\n-       Import downloaded items from EFS to WorkSite (WorkSite import process)\n\n-       Update EM_REQUESTS and EM_PROJECTS to reflect filing status\n\n-       Load and validate users\n\nFiling Time – WorkSite Proximity to SQL\n\nThe performance of the following filing functions will improve with the DMS located geographically close to the SQL server:\n\n-       Import downloaded items into WorkSite\n\n-       Update EM_REQUESTS and EM_PROJECTS to reflect filing status\n\n-       Perform duplicate detection checks (if enabled)\n\nArchitectural Tie Breakers\n\nShould EFS be geographically closer to Exchange or WorkSite DMS?\n\n-       EFS should be located geographically closer to Exchange (as MAPI is much more “chattier” protocol by comparison with WorkSite communications traffic which are optimised for WAN; ideally both Exchange & WorkSite should be close to EFS, in other cases Exchange is preferable)\n\nShould WorkSite DMS be geographically closer to EFS or SQL?\n\n-       DMS should be geographically closer to SQL.\n\n* It is not recommended to use caching/proxy for EFS connectivity.\n\nScaling – EFS Threads

–       Each EFS server is configured by default to have 10 threads for Email Filing, 10 threads for Marked Folders, and 10 threads for Mailbox Synchronization. This means that there could be 30+ concurrent connections to the DMS at any given time (The EFS Management console and Legacy EFS service also make separate connections)

–       Each thread will generate multiple transactions for the various calls the services will need to make to the DMS, including downloading queued requests, importing items, and searching the database.

–       Some transactions, such as selecting filing requests, are very lightweight when compared to imorting items.

\n* This is not out of ordinary for WorkSite servers which provide connectivity to EFS to have high transaction per minute values, but you should keep in mind that a lot of these transactions (generated by EFS) are lightweight.\n\nScaling – Add Email Filing Servers

While you have no control over the number of messages submitted by your users, you can reduce the amount of time that EFS spends on each batch by spreading this work out across additional Email Filing Servers (scaling out).

\nMethods for adding EFS servers:\n

–       Automatic Load Balancing with Clustering (preferred)

–       Single server mode (using WorkSite security groups to divide data among servers and/or database/mailbox filters)

Note: WorkSite database filters and Exchange mailbox server filters can be applied in conjunction with either of the above methods. However WorkSite security groups can be used only with the single server mode at this point. If you do enable Automatic Load Balancing with Clustering you can’t use WorkSite groups and apply those on top of clustered servers.

\nAutomatic Load Balancing\n\nBenefits of using Automatic Load Balancing with Clustering rather than single server mode\n\n-       Loads new WorkSite users (daily) automatically without admin intervention (no need to perform manual load balancing each time when new user was added)\n\n-       Provides failover protection at the EFS level\n\n-       Less administrative overhead when adding additional Email Filing Servers (no need to mess with groups as in multiple non-clustered WCS servers scenario)\n\nScaling – Add DMS Servers

EFS will increase the workload on the WorkSite system as we have made it very easy to file a large number of emails to WorkSite.

Medium and Large firms in particular may want to segregate the asynchronous work of EFS from the real time work of production DMS. This will decrease the likelihood of a negative impact to production with the increased workload.

Best practice is to have a dedicated DMS server/cluster for EFS.

Some benefits of this approach include:

–       Protect production users from the additional transactions introduced by EFS

–       Simplify maintenance and troubleshooting (isolation of EFS traffic/log events from clients traffic/log events)

It may take a few tries to determine the best number of additional DMS servers to use for this “back office” DMS server/cluster.

In medium and large environments typical ration is 2 EFS servers to 1 DMS server. Though this is very difficult to size it upfront as it largely depends on actual workload.

Summary – Exchange Architecture

Centralized vs. Distributed Exchange

–       For optimal performance, EFS should always be local to the service account mailbox.

–       For centralized Exchange environments, all EFS servers should be local to central Exchange

–       For distributed Exchange environments, EFS servers should be placed near in each Exchange location (separate service account with local mailboxes in each location for the EFS service processing locally)

–       Mailbox Server filters can be applied to each EFS server. Each server will process only the mailboxes which are located on the mailbox servers included in the applied filter.

Summary – WorkSite Architecture 

Centralized vs. Distributed WorkSite

–       For optimal performance, EFS should be local to the WorkSite DMS server/cluster and SQL database (ideal case)

–       When the EFS cannot be local to WorkSite, meshed DMS (eDMS) servers shouldn’t be used. EFS should connect directly to DMS servers/clusters which are local to the SQL database. It’s better to have EFS traffic over the WAN, than ODBC traffic, so you should place WorkSite locally to SQL and connect your EFS to this WorkSite server. Either it’s not recommended to provide EFS connectivity throgh caching/proxy DMS servers.

–       WorkSite database filters can be applied to each DMS connection on each EFS server. Each server will process only the databases included in the applied filter for each connection.

We can make connection to individual WorkSite servers and/or individual databases from each EFS server.

Design Scenario A – Exchange Distributed\n\nInitial Design:\n\nDesign Scenario A - Exchange Distributed

Chicago (Exchange CAS Array, 8 node DMS Cluster, SQL Database, 2000 WorkSite EM clients) – WAN – New York (Exchange, 2 node DMS cluster, SQL database, 500 WorkSite EM Clients)

Design with EFS:\n\n Design Scenario A - Exchange Distributed Improved

Chicago (8 node EFS Cluster, Exchange CAS Array (with added connection to Chicago Back Office DMS), 4 Node Back Office DMS Cluster, 8 Node DMS Cluster (with added connections to NY databases), SQL Database, 2000 WorkSite EM clients) WAN – New York (2 node EFS Cluster (with added connection to back office DMS Cluster in NY), Back Office DMS Server, Exchange, 2 node DMS cluster (with added connections to Chicago databases), SQL database, 500 WorkSite EM clients)

EFS servers were added to each location doing mailbox processing only for local user base. EFS servers to DMS servers ratio is 2 to 1.\n\nDesign Scenario B – Exchange Centralized\n\nInitial Design (the same as a but Exchange is centralized and located in Chicago):\n\n Design Scenario B - Exchange Centralized\n\nDesign With EFS:\n\n  Design Scenario B - Exchange Centralized Improved\n\nQuestios from last session:

–       Q: EFS is an asynchronous process. What value is added by increasing the number of Email Filing Servers (scaling out)?

–       A: It deacreases filing time by distributing EFS work across multiple servers.

–       Q: We have no ability to limit the amount of email that a user may submit for processing. What challenges does this present to the design, implementation, and administration of the system?

–       A: We must scale the system to deliver the functionality and performance required by the business. As we have no control over the amount of email coming in, this process could be iterative and may thake several “tries” before the right balance is found.

Final Thoughts

Here are a couple of questions that we would like you to think about before we meet for the next session.

–       Now that you have a good understanding of how EFS works, where do you expect problems to occur?

–       What types of problems have you encountered with EFS?

Please follow and like us:

iManage WCS webinar series

Last year I’ve attended Autonomy ICSE course at their training centre in Cambridge, UK. All in all it was in a good experience but to my mind couple of issues with this training were poor coverage of WCS technology and even more lapidary description of FileShare (for those who wondering about Worksite Indexer they provide dedicated IDOL trainings which cover full-fledged IDOL solution on which WokrSitie Indexer is based). Whereas this was OK not to dwell much on the FileShare technology which is niche solution and still immature technology (IMHO), poor WCS coverage was a bit unexpected. Probably really in-depth (not targeted to end users) client side components course with drill down to their logic, innards and customizations is missing in Autonomy course offering too, but this is another story…

Recently HP Autonomy / ILTA arranged series of webinars on WCS which more than compensate for this gap in ICSE course (actually after watching parts 1 & 2 I realized that proper coverage of WCS would add 1 more day to ICSE course to give decent coverage of this technology). Anyway this webinar covers all the aspects of WCS and I more than satisfied with it and can recommend it to others as it really may improve your understanding of this solution.

Webinar series comprise of 3 parts (links to recordings are available on WorkSite support portal):

\nPart 1. Email Filing Server – Functional Overview (March 28, 2013)\n\nPart 2. Email Filing Server – Best Practices Design and Deployment (April 23, 2013)\n\nPart 3. Email Filing Server – Troubleshoting (May 23, 2013):\n

Some people still prefer old fashioned reading to webcasts and if you are amongst those you may read what was covered in webinar below (as I jot it down sort of speak). Please read on for Part 1 Email Filing Server – Functional Overview.

Part 1. Email Filing Server – Functional Overview

\n\n

    \n

  1. Email Filing Service\n
      \n

    1. Email Filing Worker
    2. \n

    3. Folder Marking Worker
  2. \n

  3. Mailbox Synchronization Service
  4. \n

  5. Clustering\n
      \n

    1. WorkSite Clustering Service
    2. \n

    3. Load Balancer Service
  6. \n

  7. EFS Management Console

EFS was introduced in 8.5 product line and before that heavy-lifting for email filing was done on a client side. There was a limitation in a number of messages that user can import as it would take about 1 to 3 seconds to file 1 message and mass filing of messages performed on a client side lock the client app. With EFS filing is moved to back-end and became server-server interaction, rather than client-server interaction.

WCS Architecture\n\nEmail Filing Server includes:\n\n0. WCS legacy service which we see in WorkSite server manager – it provides send and file functionality\n\n1. EFS service includes / parent for 2 processes (EmailMgmtSvc.exe – WorkSite Email Management Service):\n\n-       A) Email Filing Worker (individual and bulk email filing)\n\n-       B) Folder Marking Worker (Outlook-to-WorkSite folder synchronization / marked folder processing)\n\n2. Mailbox Synchronization Service aka Mailbox Agent (marking of emails in inbox which were filed by other users). Currently it synchronizes only the Inbox folder.\n\n3. Clustering (for load balancing and failover). Allows for scaling of EFS to enable better filing times and to provide failover.\n\n-       A) WorkSite Clustering Service\n\n-       B) Load Balancer Service\n\n4. EFS Management Console\n

Email Filing Service (EFS)

EmailMgmtSvc.exe (WorkSite Email Management Service) – parent service that controls Email Filing and Marked Folder processing.

–       Starts when Email Filing is started from Admin console

–       Monitors polling intervals for FilingWorker.exe amd MarkingWorker.exe and starts processes on schedule

–       Communicate with EmailMgmtAdmin.exe and LoadBalancer.exe to create the ValidUsers files (ValidUsers.urs and ValidUsers.txt) after user validation is complete.

–       Sends shutdown commands to FilingWorker.exe and MarkingWorker.exe when service is stopped.

Email Filing Service (EFS) – FilingWorker.exe child process of EmailMgtSvc.exe

Email Processing

–       Process that files emails queued via the WorkSite Email Management client

–       Runs on a scheduled basis (schedule is enforced by parent service, default polling interval is 150 seconds)

–       Runs 10 worker threads by default, each making an independent connection to Excange and each processing a single user’s queued messages. When it starts it pick ups the work that available in EM_REQUESTS table and process it till the end, then thread became available for processing of the next mailbox (* software only pick ups 500 jobs per user at one time, built in limitation to avoid 1 user / small group of users bottlenecking server).

FilingWorker.exe Flow

(perfect case scenario, no message has been moved* and no modifications have been made)

\n\n

    \n

  1. EM client queues an email (either via drag and drop or toolbar) and a job is created in EM_PROJECTS table. EM client – DMS transaction. Then DMS server creates job in EM_REQUESTS table which contains failing information required for actual filing.
  2. \n

  3. EM client updates Message Class of the Email to queued on the Exchange. EM Client – Exchange transaction. (custom message class “IPM.Note.Worksite.Ems.Queued” is used to denote the fact that the message has been queued on the Exchange side) At this stage user gets hourglass icon for the message, which means message has been queued.
  4. \n

  5. Filing Worker queries the EM_REQUESTS table on its scheduled interval to retrieve details including Exchange Enrty ID for the message (source) and target WorkSite folder (destination) locations of the Email. DMS – EFS transaction.
  6. \n

  7. Filing Worker connects to Exhange, locates the email by Entry ID, and downloads it. EFS – Exchange transaction. * Perfect case scenario
  8. \n

  9. Filing Worker attempts to import the downloaded message to WorkSite. EFS – DMS trsnsaction.
  10. \n

  11. Filing Worker attempts to update the Message Class to reflect whether the Email filed successfully (“IPM.Note.Worksite.Ems.Filed”). EFS – Exchange transaction.

* Exchange message Entry ID value is location specific, if the message is moved the Entry ID will change. In such cases another property PR_SEARCH_KEY used to find message in Inbox (this search is more costly than locating the message by the Enrty ID). PR_SEARCH_KEY is limited to search only 4 levels deep at the present time (if a user moved message to a folder nested more than 4 levels deep it won’t be located).

PR_SEARCH_KEY is a fallback search mechanism to attempt to lacate messages which have been moved after being filed.

Best Practice: User should be trained don’t move queued message (one marked with hourglass) until it gets a green checkmark icon.

Post Filing

\nUpon a successful import:\n

–       Updates the Message Class of the source message (also in case of “file and delete”, moves message to the Delteted Items of the maiblox owner; you steel see green checkmark on a filed item in Deleted Items)

–       Removes the queued job from the EM_REQUESTS table

Upon an unsuccessful import:

–       Updates the Message Class of the source message to IPM.Note.Ems.WorkSite.Error

–       Updates the Filing Status field of the source message with an error description (e.g. profile error)

–       Writes a detailed error to the FilingWorker log.

–       Removes the queued job from the EM_REQUESTS table (if this is an error service capable to handle gracefully, not an exception type of an error; for exceptions error is written to the log but job may not be removed from the EM_REQUESTS table)

Email Filing Service (EFS) – MarkingWorker.exe child process of EmailMgtSvc.exe, processing folders rather than individual emails

Marked Folder Processing

–       Process which is responsible for syncronizing marked folders (Outlook folder which has been linked to Worksite folders via the EM client, either through File to Worksite tab in folder properies or through the dialogue upon folder creation)

–       Runs on a scheduled basis (360 seconds is default polling interval)

–       Runs 10 worker thread by default, each making an independent connection to Exchange and each processing a single user’s marked folders at a time. (* the same built-in limitation for processing no more than 500 messages per user per cycle as with Filing Worker)

MarkingWorker.exe Flow

\n\n

    \n

  1. EM enabled client creates marked folder and link is created in EM_PROJECTS table. EM Client – DMS transaction
  2. \n

  3. Marking Worker queries the EM_PROJECTS table to retrieve details including Exchange Entry ID for a folder (source) and marked Worksite folder (destination) locations of the Email. DMS – EFS transaction.
  4. \n

  5. Marking Worker connects to Exchange folder by its Entry ID. Identifies new emails by reading the Message Class and custom properties (filing location) of each item, and downloads those identiefied. EFS – Exchange trasaction. * to track non email items stored in Exchange folders (message class other than IPM.Note type e.g. stubs, calendar items) custom Exchange property is being set to denote that is actually been imported.
  6. \n

  7. Marking Worker attempts to import all downloaded messages to WorkSite. EFS – DMS transaction.
  8. \n

  9. Marking Worker attempts to update Message Class to reflect whether the email filed successfully (only for regular emails i.e. objects if IPM.Note class items, other items are being filed but not marked with icons, just marked with propery on background for MarkingWorker but with no visual representation for user – this is by design at the moment). EFS – Exchange transaction.

Post Filing

Upon a successful email import:

–       Updates the Message Class ot the source message to “Filed” (“IPM.Note.Worksite.Ems.Filed”)

Upon an usuccessful email import:

–       Updates the Message Class of failed imports to “Error” (“IPM.Note.Worksite.Ems.Error”)

–       Updates the Filing Status field with error description

–       Writes a detailed error to the MarkingWorker log.

Updates the STATUS field in the EM_PROJECTS table to reflect a successful synchronization or a failure (for folder link, not for the message, eror only if unable access/locate folder not related to filing status of messages inside). Folder status maybe OK where as there are errors with individual emails – this is reflected by yellow icon on failed messages.

Mailbox Synchronization (new feature of 8.5 SP2 & 9.0 versions of WCS)

\nBy default (out of box) this process runs every hour, this can be customized (it doesn’t recommended to set it lower than 5 minutes).\n

Overview

–       Solution for updating filing information on Emails which have already been filed to Worksite by another user.

–       Connects directly both to Exchange and WorkSite to perform synchronization.

–       Uses SQL compact databases to store syncronization information (local to the server where Mailbox Synchronization is running).

–       New feature in WCS 8.5 SP2/9.0

–       Successor to the previous client-side Inbox Sweeper which was included in Email Management client.

–       Runs on a scheduled basis (this service runs on its own & called sweeper.exe, no parent service)

–       Runs 10 worker threads by default, each making an independent connection to Exchange and each processing one user mailbox at a time. For this service number of threads is configurable through registry.

So in total WCS in default configuration makes 30 connection to Exchange: 10 from Filing Worker, 10 from Marking Worker and 10 from Mailbox Synchronization)

Mailbox Synchronization Process – Sweeper.exe

Initial Synchronization

\n\n

    \n

  1. Connects to each user mailbox in Exchange (those in valid users list) and downloads select message properties from all messages in the user’s Inbox (creating “snapshot of Inbox”).
  2. \n

  3. Performs a WorkSite search for all emails (which the user has security access to) added within the First Time Synchronization Horizon (30 days by default), and reads Message ID stored in DOCMASTER table.
  4. \n

  5. Matches Inbox Emails with WorkSite Emails based on Message ID.
  6. \n

  7. Updates Filing Status, Filing Location, and Filing Date for matched items.
  8. \n

  9. Stores synchronization information for each user within SQL compact databases (*.sdf files) located on the server running Mailbox Agent.

Time of initial synchronization is vary quite a bit depending on user’s mailbox size, and alway longer that subsequent synchronizations.

Regular Synchronization

    \n

  1. Performs a worksite search for all emails (which the user has security access to) added since the last synchronization, and reads Message ID stored in DOCMASTER table.
  2. \n

  3. Connects to each user mailbox in Exchange and downloads select message properties from messages added to the user’s Inbox since the last synchronization.
  4. \n

  5. Matches Inbox emails with WorkSite emails based on Message ID.
  6. \n

  7. Updates Filing Status, Filing Location, and Filing Date for matched items.
  8. \n

  9. Stores synchronization information for each user within SQL compact databases (*.sdf files) located on the server running Mailbox Agent

Clustering (Automatic Load Balancing)

Overwiew

–       New feature introduced with the 8.5 SP2/9.0 release of WCS

–       In the previous 8.5 SP1 version of WCS, scaling options were limited to WorkSite database and Exchange Mailbox server filter

–       Eah member node within the cluster processes a subset of Worksite EM users

–       Allows for scaling horizontally by adding the number of nodes necessary to reach the desired level of performance.

–       Utilizes WorkSite Cluster Manager service for communication between nodes

Clustering – WorkSite Cluster Manager Service

imFmaSvc.exe (WorkSite Cluster Manager Service)

–       Facilitates communication between LoadBalancer.exe processes on each node in the WCS cluster

–       Must be installed on each node in the WCS cluster

–       Only required if Automatic Load Balancing is in use

Clustering – WorkSite Load Balancer Service

LoadBalancer.exe (WorkSite Load Balancer Service) load & validate users (making sure that user has valid email address)

–       Starts when the “Connect” button is pressed from within the Admin console (only activated when using Automatic Load Balancing wirh clustering). Service starts in background along with Email Management Service

–       Loads WorkSite users and validates them against the Exchange Address Book or AD to build a list of valid users for EFS to process. Before only AD validation was available, but Exchange Addeess Book validation if faster in most of the cases (especially for large number of users). Exchange Addeess Book validation feature is introduced in WCS 9.0 U2. By default Exchange Address Book validation is turned on, and validation type is configurable through the registry.

–       Communicates with other servers within the cluster to distribute users across member nodes, and detect when a node has been added or removed.

–       Requires WorkSite Cluster Manager service to be installed on all member nodes.

\nRegistry key for setting validation type:\n\nHKEY_LOCAL_MACHINESOFTWAREAutonomyWorksite8.0MailboxAgentExchangeServers\n\nDWORD: AddressBook\n\n1 – Use Exchange Address Book Validation\n\n0 – Use AD validation\n

EFS Management Console

\nWas available in pre 8.5 SP2 versions but with SP2 some changes were added into it.\n

EmailMgmtAdmin.exe (WCS Management Console process)

–       Central location for configuration and management of EFS services

–       Allows admin to view, reset or cancel queued job

–       Allows admin to view, reset, or disable marked folders

–       Allows admin to view and manage service logs (there are separate logs for each of the services – Email Management, Filing Worker, Marking Worker, Load Balancer, EFS Admin Conolse, Mailbox Synchronization). All logs are kept in the same location (C:Program FilesAutonomyWorkSiteServerLogs) except Mailbox Synchronization log (C:Program DataAutonomy).

–       Loads and validates WorkSite users in a single server configuration

WCS Processes Outline

I. Email Filing Service – EmailMgmtSvc.exe\n\na. Email Filing Worker – FilingWorker.exe\n\nb. Folder Marking Worker – MarkingWorker.exe\n\nII. Mailbox Synchronization Service – Sweeper.exe\n\nIII. Clustering\n\na. WorkSite Clustering Service – imFmaSvc.exe\n\nb. Load Balancer Service – LoadBalancer.exe\n\nIV. EFS Management Console – EmailMgmtAdmin.exe

Final Thoughts

Here are a couple of questions that we would like you to think about before we meet for the next session.

 

–       EFS is an asynchronous process. What value is added by increasing the number of Email Filing Servers (scaling out)?

–       We have no ability to limit the amount of email that a user may submit for processing. What challenges does this present to the design, implementation, and administration of the system.

Please look for transcript of parts 2 and 3 of this webinar in further posts:

iManage WCS webinar series – Part 2

iManage WCS webinar series – Part 3

Please follow and like us: