Tag Archives: EFS

iManage WCS webinar series – Part 3

This is the transcript of 3rd part of iManage WCS webinar series (webinar recording date May 23rd, 2013)\n\nPart 3 – Troubleshooting\n

    \n

  1. I.                    Introduction
      \n

    1. Functional Review
  2. \n

  3. II.                  Email Filing Server
      \n

    1. Monitoring queues
    2. \n

    3. Logging
    4. \n

    5. Configuration files
    6. \n

    7. Failures/Errors
  4. \n

  5. III.                Mailbox Sync
      \n

    1. Logging
    2. \n

    3. SQL Compact databases
    4. \n

    5. Control Panel/Simulator
  6. \n

  7. IV.                Summary

Introduction – Functional review\n\n \n\nEmail Filing Server\n\n \n\n-          FilingWorker.exe (process) – Processes server-side filed emails. Reads queued jobs from the EM_REQUEST table\n\n-          MarkingWorker.exe – Processes marked folders. Reads folders from the EM_PROJECTS table\n\n-          EmailMgmtSvc.exe (service) – Manages startup and shutdown of the FilingWorker and MarkingWorker processes\n\n-          EmailMgmtAdmin.exe – EFS Management Console. Validates users in a single server configuration\n\n-          LoadBalancer.exe (service) – Validates and distributes users in a clustered configuration\n\n-          imFmaSvc.exe (service) – WorkSite Cluster Manager service used for communication between nodes by LoadBalancer\n\nMailbox Sync\n\n \n\n-          Sweeper.exe (service) – Adds filing information to Inbox emails which have already been filed to WorkSite.\n\nEmail Filing Server – Monitoring Queues\n\n \n\nThis may be useful both for troubleshooting and general monitoring\n\n \n\nFiling Request Management\n\n-          Displays filing jobs queued in the EM_REQUESTS table\n\n-          Shows job details including type (email or folder) and submitted time\n\n-          Allows admin to reset or cancel jobs\n\nMarked Folder Management\n\n-          Displays marked folders stored in the EM_PROJECTS table\n\n-          Shows synchronization status and last sync time\n\n-          Allows admin to reset or disable folders\n\nEmail Filing Server – Configuration Files\n\n \n\nEFS Configuration Files\n\n \n\nAll EFS configuration files are written to …AutonomyWorkSiteServerConfig\n\n-          DmsSettingsX.config – Contains DMS connection information. One file will exist for each DMS connection made by EFS (i.e. DmsSettings0.config, DmsSettings1.config)\n\n-          ExchangeSettings.config – Contains Exchange connection information, including service account credentials used to access mailboxes.\n\n-          FilingOptions.config – Contains settings which define how profile metadata is applied to filed items\n\n-          SvcSettings.config – Contains settings which define how EFS processes behave, including number of threads and polling interval\n\nEFS ValidUsers Files\n\n \n\n-          ValidUsers.urs (contains extended information including some details on users’ mailboxes) and ValidUsers.txt (stripped down copy of .urs file) – Contains list of users which were validated for processing (created upon automated/manual load balancing)\n\n \n\n* all these config files can be backed, for example before you make any changes during troubleshooting process\n\nEFS Log Information\n\n \n\n-          All EFS log are written to …AutonomyWorkSiteServerLogs\n\n-          Individual logs are created for each process – FilingWorker.exe, MarkingWorker.exe, EmailMgmtSvc.exe, EmailMgmtAdmin.exe, LoadBalancer.exe, imFmaSvc.exe\n\n-          For each process, a new log will be created on the next startup after the existing log reaches 2MB. If the active log exceeds 40MB, a new log will be created immediately\n\n-          Each process will retain the 50 most recent logs before overwriting\n\n-          Log Filter level is set to Verbose by default\n\n-          Log monitor can be used to view log writes in real-time\n\nEmail Filing Server – Log Information\n\n \n\nWhere to Look for Logging\n\n \n\nFilingWorker.log – Contains information about filing job processing, including downloading the item(s) from Exchange and importing them into WorkSite.\n\nContains filing error messages – server side filing requests from EM_REQUESTS table, e.g. profile error etc. In additional to this log you usually see detailed error information in a filing status column on a client side.\n\nMarkingWorker.log – Contains information about marked folder processing, including downloading the folder contents from Exchange and importing item(s) into WorkSite. E.g. errors with filing of individual message within the marked folder.\n\nEmailMgmtSvc – Contains information on service startup and continuous cycling of the worker processes.\n\nContains information about general server issues, startup failures. This service control worker processes, so it’s your destination if you need to check why marking or filing worker not starting or not happening as quickly as they should be.\n\nEmailMgmtAdmin – Contains information on what is displayed in the EFS management Console. Also displays user validation information in a single server configuration.\n\nIf you are not using clustering component  with automatic load balancing then all the logic for the user validation is within this executable. When you click on a Load Balancing button on a non clustered system all the user validation happens in this console and all the output for failures and successes during that validation will be written to EmailMgmtAdmin.log\n\nLoadBalancer – Contains information on user validation and distribution in a clustered environment.\n\nContains data about users validation on clustered systems only, plus about users distribution and number of cluster nodes.\n\nimFmaSvc – Contains information on the communication between nodes in a clustered environment.\n\nMay contain errors related with joining nodes to the cluster and communication between nodes.\n\nEmail Filing Server – Reading Logs\n\n \n\nUnderstanding the FilingWorker Logs\n\n \n\nBy default, FilingWorker will launch 1 main thread and 10 worker threads (configurable). Each line in the FilingWorker log indicates the thread number which is writing the line. Each log will include the following information:\n

    \n

  1. The main thread (thread #1) will write startup information, load and validate queued jobs from the EM_REQUESTS table for valid users, and initialize worker threads.
  2. \n

  3. Each worker thread (thread #5-14) will connect to a user mailbox, attempt to process the user’s queued items (500 max)
  4. \n

  5. As worker threads complete work for each user, the main thread will check the EM_REQUESTS table for more work
  6. \n

  7. Once no additional work is available, the main thread will update filing statuses, remove completed jobs from the EM_REQUESTS table, shutdown all worker threads and kill FilingWorker.exe

\nIt is convenient to use as a log viewer something like Notepad++, which allow you to highlight thread numbers to more easily follow them and has good search capabilities.\n\nUnderstanding the MarkingWorker Logs\n\n \n\nBy default, MarkingWorker will launch 1 main thread and 10 worker threads (configurable). Each line in the MarkingWorker log indicates the thread number which is writing the line.\n\nEach log will include the following information:\n

    \n

  1. The main thread (thread #1) will write startup information, load and validate marked folder links from the EM_PROJECTS table for valid users, and initialize worker threads.
  2. \n

  3. Each worker thread (thread #5-14) will connect to a user mailbox, attempt to process the user’s marked folders. The status of each folder will be updated during processing.
  4. \n

  5. Once all users are complete, the main thread will shutdown all worker threads and kill MarkingWorker.exe

Understanding the LoadBalancer Logs\n\n \n\nThe LoadBalancer logs will capture information on user validation process. For users which are successfully added to the ValidUsers file for processing, minimal information is captured. For failures in validation, more detail will be recorded. Each log will show following:\n

    \n

  1. Write startup information
  2. \n

  3. Retrieve all enabled users with properly configured email address in WorkSite.
  4. \n

  5. Validate all users against AD or Address Book (by default Address Book validation is turned on)
  6. \n

  7. Report user distribution

Email Filing Server – Failures/Errors\n\n \n\nFilingWorker Processing\n\n \n\nSome issues which can cause filing failures include:\n\n-          Profile errors – Refer to FilingWorker log for details. Example:  Cannot Import the message: \n\nProfile Error: AttID=imProfileCustom1, Value=TEST, Desc=The field is disabled\n\n-          Filing user not validated – Refer to LoadBalancer log for clustered environments, EmailMgmtAdmin log for single server environments. Example:\n\nThe user account with the Email Address: USER@DOMAIN.COM failed agains Active Directory\n\n-          Filed item cannot be located using ENTRY_ID or PR_SEARCH_KEY – Refer to Filing Worker log for details. Example:\n\nFiling Error…System error. <spMsgStore.OpenEntry: EID=123456789> NotFound at Interwoven.WorkSite.Ems.ThFilingProcessor.ThEfsConroller.ProcessFilingRequest(ThConfigData&ConfigData,ThDMS& Dms, ThRequest& tfr, IMsgStore, IMAPISession spSession, Boolean bDoNotCheck, List’1& vDouble, ThSlot& Slot)\n\n \n\nMarkingWorker Processing\n\n \n\nSome issues which can cause marked folder failures include:\n\n-          Profile errors – Refer to MarkingWorker log for details. Example:\n\nCannot Import the message: Profile Error: AttID=imProfileClass, Value=, Desc=Field is required\n\n-          Filing user not validated – Refer to LoadBalancer log for clustered environments, EmailMgmtAdmin log for single server environments. Example:\n\nTESTUSER does not have an E-Mail address\n\n-          Folder cannot be located using ENTRY_ID – Refer to FilingWorker log for details. Example:\n\nSystem error.<spFolder.GetProps> NotFound sAddInfo: <> at Interwoven.WorkSite.Ems.ThFilingProcessor.ThEfsController.GetLatestMessages(IMAPISession spSession, IMsgStore spMsgStore, ENTRYID FolderEID, MarkedFolder& mf, List’1& vEID, ENTRYID& ParentFolderEID, String sAddMsgClass)\n\nNOTE: Marked folder status is only updated to Successful or Failed based on whether the process was able to locate the folder and retrieve the contents. Individual item filing successfully or not does not reflect the folder status.\n\nMailbox Synchronization – Log Information\n\n \n\nMailbox Synchronization Logging Information\n\n \n\n-          Synchronization Log Monitor displays Mailbox Sync log activity in real-time.\n\n-          Log files are written to the following location:\n\nWindows 2008 Server:\n\nC:ProgramDataAutonomyWorkSiteMailboxAgentLogsMirrorLogs\n\nWindows 2003 Server:\n\nC:Documents and SettingsAll UsersAutonomyWorkSiteMailboxAgentLogsMirrorLogs\n\n-          MirrorLogs directory is only written after a successful initial synchronization\n\n-          During processing, all log data is written to SweeperLog.sdf\n\n-          SweeperLog.sdf data is written to log files in MirrorLogs upon each cycle completion\n\nMailbox Synchronization – Data\n\n \n\nMailbox Synchronization SQL Compact Files\n\n \n\n-          Mailbox Synchronization maintains separate SQL Compact data files for each user in the following location:\n\nWindows 2008 Server:\n\nC:ProgramDataAutonomyWorkSiteMailboxAgentDataStorage\n\nWindows 2003 Server:\n\nC:Documents and SettingsAll UsersAutonomyWorkSiteMailboxAgentDataStorage\n\n-          SQL Compact user files maintain a MSG ID database of the Inbox contents\n\n-          Service logging is written to SweeperLog.sdf during processing\n\n \n\nResetting Sync Data\n\n \n\n-          All synch data can be reset by stopping EFS and Mailbox Synchronization services and renaming.\n\nC:ProgramDataAutonomyWorkSiteMailboxAgent. This will force the initial synch to take place again.\n\n-          Individual users can be reset by renaming individual user SDF files.\n\n \n\nMailbox Synchronization – Control Panel\n\n \n\nControl Panel Functions\n\n \n\n-          Synch – Force an immediate synchronization without waiting for the polling interval (default, out of box polling interval equals to 1 hour)\n\n-          Validate Users – Validates the users in the WorkSite group used for Mailbox Synchronization (single server configuration only, in clustered configurations by default Mailbox Synchronization ignores group and perform processing for all users each node is balanced for)\n\n-          Select Mailbox – Lists all validated mailboxes for which Mailbox Synchronization is currently processing\n\n \n\nMailbox Synchronization – Simulator\n\n \n\nMailbox Operations Simulator\n\n Mailbox_Ops_Simulator

–          The Mailbox Operations Simulator can be used to simulate individual Mailbox Synchronization activities against specific users

–          Helpful for troubleshooting problems with specific users as well as general issues with Mailbox Synchronization

–          Does not update data, only simulates activity and displays output

Summary\n\nTroubleshooting Email Filing Server\n\n \n\n-          Check the queue – Make sure the queued job or marked folder shows in queue\n\n-          Check the user – Verify that the filing user is validated on the server\n\n-          Check the logs – Look in the appropriate service log for errors\n\n \n\nTroubleshooting Mailbox Synchronization\n\n-          Check the user – Make sure user mailbox is visible\n\n-          Check the logs – Check the logs for errors\n\n-          Test – Use the Mailbox Operations Simulator

Please follow and like us:

iManage WCS webinar series – Part 2

This is the transcript of 2nd part of iManage WCS webinar series.\n\nPart 2. Email Filing Server – Best Practices Design and Deployment\n\nSizing & deployment strategies.\n\nAgenda.\n

    \n

  1. Introduction\n
      \n

    1. EFS Review
    2. \n

    3. Performance Defined
    4. \n

    5. Performance Variables
  2. \n

  3. Filing Time\n
      \n

    1. EFS Proximity to Exchange
    2. \n

    3. EFS Proximity to WorkSite DMS
    4. \n

    5. DMS Proximity to SQL
  4. \n

  5. Scaling for Improved Performance
  6. \n

  7. Summary
  8. \n

  9. Sizing scenarios\n
      \n

    1. Case Study for Central Exchange Architecture
    2. \n

    3. Case Study for Distributed Exchange Architecture

Introduction – EFS Review

Email Filing (Filing Worker) is asynchronous (i.e. filing is not real time)

User flags a message to be filed in Outlook

On the Email Filing Server, the Filing Worker process attempts to file the message on its next iteration. All messages are filed in batches.

The amount of time that Filing Worker spends on each iteration is a product of:

–       The number of messages that need to be processed

–       The amount of time it takes to process each message

An Admin will have no direct control over the number of messages that his/her users are filing.

Marked Folder Processing (Marking Worker) is asynchronous

–       User maps (marks) an Outlook folder to a WorkSite folder.

–       MarkingWorker attempts to process the folder on next iteration. The MarkingWorker processes folders in batches. All folders are checked each iteration.

–       For each folder, MarkingWorker determines the messages that haven’t been filed and processes these one at a time.

–       The amount of time that MarkingWorker spends on each iteration is a product of

\n\n

    \n

  • The number of marked folders that need to be processed
  • \n

  • The number of messages that need to be processed
  • \n

  • The amount of time it takes to process each message

–       An admin will have no direct control over the number of folders that his/her users are marking, nor does he/she have control over the number of messages that are in each Outlook folder that get’s processed

Introduction – Performance Defined

Q: What is the single most important measure of performance for Email Filing Server?

A: The amount of time it takes EFS to process a batch.

–       Messages or Marked Folders are processed in batches. The faster the batch completes, the faster the messages in the batch complete. The faster the next batch can be processed.

–       The amount of time that a message takes to file after a user has flagged the message as a WorkSite message (either by placing it in a WorkSite folder, or by putting it in a Marked Outlook folder)

Introduction – Performance Variables\n\n-       Email Filing\n

    \n

  • Number of messages being processed in this batch
  • \n

  • Single message filing time

\n-       Marked Folder Processing\n

    \n

  • Number of Marked Folders
  • \n

  • Number of messages that need to be filed from each Outlook folder
  • \n

  • Single Message filing time

\nThere are methods to modify each of these variable in order to have a positive impact on message filing time.\n\nFiling Time – Proximity of EFS to Exchange\n\nEFS proximity to Exchange has a significant impact on performance\n\n-       The time it takes EFS to file one or more queued items to WorkSite (appear in the WorkSite filing location and/or see updated filing information on the original item) after they’ve been successfully queued.\n\n-       The time it takes EFS to file items to WorkSite after being added to a Marked Folder\n\nThe performance of the following filing functions will improve with EFS located geographically close to the user’s Exchange mailbox:\n\n-       Connect to user mailboxes in Exchange\n\n-       Locate message or folder in Exchange for Filing or Marking process\n\n-       Search mapped Exchange folders to retrieve new content added to marked folders\n\n-       Copy message from Exchange to EFS server for import into WorkSite\n\n-       Update filing information values on Emails to reflect the filing status (Filed, Queued, Failed…)\n\nFiling Time – EFS proximity to WorkSite\n\nThe perfromance of the following filing functions will improve with EFS located geogrpaphically close to the DMS server:\n\n-       Load and validate filing jobs and marked folders\n\n-       Import downloaded items from EFS to WorkSite (WorkSite import process)\n\n-       Update EM_REQUESTS and EM_PROJECTS to reflect filing status\n\n-       Load and validate users\n\nFiling Time – WorkSite Proximity to SQL\n\nThe performance of the following filing functions will improve with the DMS located geographically close to the SQL server:\n\n-       Import downloaded items into WorkSite\n\n-       Update EM_REQUESTS and EM_PROJECTS to reflect filing status\n\n-       Perform duplicate detection checks (if enabled)\n\nArchitectural Tie Breakers\n\nShould EFS be geographically closer to Exchange or WorkSite DMS?\n\n-       EFS should be located geographically closer to Exchange (as MAPI is much more “chattier” protocol by comparison with WorkSite communications traffic which are optimised for WAN; ideally both Exchange & WorkSite should be close to EFS, in other cases Exchange is preferable)\n\nShould WorkSite DMS be geographically closer to EFS or SQL?\n\n-       DMS should be geographically closer to SQL.\n\n* It is not recommended to use caching/proxy for EFS connectivity.\n\nScaling – EFS Threads

–       Each EFS server is configured by default to have 10 threads for Email Filing, 10 threads for Marked Folders, and 10 threads for Mailbox Synchronization. This means that there could be 30+ concurrent connections to the DMS at any given time (The EFS Management console and Legacy EFS service also make separate connections)

–       Each thread will generate multiple transactions for the various calls the services will need to make to the DMS, including downloading queued requests, importing items, and searching the database.

–       Some transactions, such as selecting filing requests, are very lightweight when compared to imorting items.

\n* This is not out of ordinary for WorkSite servers which provide connectivity to EFS to have high transaction per minute values, but you should keep in mind that a lot of these transactions (generated by EFS) are lightweight.\n\nScaling – Add Email Filing Servers

While you have no control over the number of messages submitted by your users, you can reduce the amount of time that EFS spends on each batch by spreading this work out across additional Email Filing Servers (scaling out).

\nMethods for adding EFS servers:\n

–       Automatic Load Balancing with Clustering (preferred)

–       Single server mode (using WorkSite security groups to divide data among servers and/or database/mailbox filters)

Note: WorkSite database filters and Exchange mailbox server filters can be applied in conjunction with either of the above methods. However WorkSite security groups can be used only with the single server mode at this point. If you do enable Automatic Load Balancing with Clustering you can’t use WorkSite groups and apply those on top of clustered servers.

\nAutomatic Load Balancing\n\nBenefits of using Automatic Load Balancing with Clustering rather than single server mode\n\n-       Loads new WorkSite users (daily) automatically without admin intervention (no need to perform manual load balancing each time when new user was added)\n\n-       Provides failover protection at the EFS level\n\n-       Less administrative overhead when adding additional Email Filing Servers (no need to mess with groups as in multiple non-clustered WCS servers scenario)\n\nScaling – Add DMS Servers

EFS will increase the workload on the WorkSite system as we have made it very easy to file a large number of emails to WorkSite.

Medium and Large firms in particular may want to segregate the asynchronous work of EFS from the real time work of production DMS. This will decrease the likelihood of a negative impact to production with the increased workload.

Best practice is to have a dedicated DMS server/cluster for EFS.

Some benefits of this approach include:

–       Protect production users from the additional transactions introduced by EFS

–       Simplify maintenance and troubleshooting (isolation of EFS traffic/log events from clients traffic/log events)

It may take a few tries to determine the best number of additional DMS servers to use for this “back office” DMS server/cluster.

In medium and large environments typical ration is 2 EFS servers to 1 DMS server. Though this is very difficult to size it upfront as it largely depends on actual workload.

Summary – Exchange Architecture

Centralized vs. Distributed Exchange

–       For optimal performance, EFS should always be local to the service account mailbox.

–       For centralized Exchange environments, all EFS servers should be local to central Exchange

–       For distributed Exchange environments, EFS servers should be placed near in each Exchange location (separate service account with local mailboxes in each location for the EFS service processing locally)

–       Mailbox Server filters can be applied to each EFS server. Each server will process only the mailboxes which are located on the mailbox servers included in the applied filter.

Summary – WorkSite Architecture 

Centralized vs. Distributed WorkSite

–       For optimal performance, EFS should be local to the WorkSite DMS server/cluster and SQL database (ideal case)

–       When the EFS cannot be local to WorkSite, meshed DMS (eDMS) servers shouldn’t be used. EFS should connect directly to DMS servers/clusters which are local to the SQL database. It’s better to have EFS traffic over the WAN, than ODBC traffic, so you should place WorkSite locally to SQL and connect your EFS to this WorkSite server. Either it’s not recommended to provide EFS connectivity throgh caching/proxy DMS servers.

–       WorkSite database filters can be applied to each DMS connection on each EFS server. Each server will process only the databases included in the applied filter for each connection.

We can make connection to individual WorkSite servers and/or individual databases from each EFS server.

Design Scenario A – Exchange Distributed\n\nInitial Design:\n\nDesign Scenario A - Exchange Distributed

Chicago (Exchange CAS Array, 8 node DMS Cluster, SQL Database, 2000 WorkSite EM clients) – WAN – New York (Exchange, 2 node DMS cluster, SQL database, 500 WorkSite EM Clients)

Design with EFS:\n\n Design Scenario A - Exchange Distributed Improved

Chicago (8 node EFS Cluster, Exchange CAS Array (with added connection to Chicago Back Office DMS), 4 Node Back Office DMS Cluster, 8 Node DMS Cluster (with added connections to NY databases), SQL Database, 2000 WorkSite EM clients) WAN – New York (2 node EFS Cluster (with added connection to back office DMS Cluster in NY), Back Office DMS Server, Exchange, 2 node DMS cluster (with added connections to Chicago databases), SQL database, 500 WorkSite EM clients)

EFS servers were added to each location doing mailbox processing only for local user base. EFS servers to DMS servers ratio is 2 to 1.\n\nDesign Scenario B – Exchange Centralized\n\nInitial Design (the same as a but Exchange is centralized and located in Chicago):\n\n Design Scenario B - Exchange Centralized\n\nDesign With EFS:\n\n  Design Scenario B - Exchange Centralized Improved\n\nQuestios from last session:

–       Q: EFS is an asynchronous process. What value is added by increasing the number of Email Filing Servers (scaling out)?

–       A: It deacreases filing time by distributing EFS work across multiple servers.

–       Q: We have no ability to limit the amount of email that a user may submit for processing. What challenges does this present to the design, implementation, and administration of the system?

–       A: We must scale the system to deliver the functionality and performance required by the business. As we have no control over the amount of email coming in, this process could be iterative and may thake several “tries” before the right balance is found.

Final Thoughts

Here are a couple of questions that we would like you to think about before we meet for the next session.

–       Now that you have a good understanding of how EFS works, where do you expect problems to occur?

–       What types of problems have you encountered with EFS?

Please follow and like us: