Top 7 Daily Enterprise Vault Checks

Veritas Enterprise Vault (formely Symantec) is an enterprise archiving platform that is used to archive common data targets such as Microsoft Exchange, File Servers and SharePoint Servers. Users are able to carry out powerful searches of this data in their archives or vaults.

This post is to provide customers with a list for the top things to look out for within Enterprise Vault. Top 7 (and a bit) places to look for when completing daily EV checks for Exchange Archiving (and common solutions!)

The first place to start when completing daily checks of EV is the status section within the Vault Admin Console. If, however, the console is not responding when completing these checks, ensure that all EV services are running and try reopening the Admin Console.

The Status will display any:

• Backup Issues
• Indexing Issues
• SQL Database Issues
• Storage Issues
• Archiving Issues
• Connection Issues
• Task Issues

If items are awaiting backup:

• Ensure the backup program is running correctly
• Ensure the backup scripts are being called
• Is the backup program honouring the archive bit?
• Are the partitions configured to the correct backup setting?
• Are the scripts adjusted accordingly?

Enterprise Vault Application Backup Issues:

• Are Indexes or Vault Stores / Archives getting stuck in backup mode?
• Can they be taking in and out manually?
• Are the scripts correct? And when called manually, are they performing correctly?
• Are the scripts being called as part of the backup schedule?

If Items are awaiting indexing:

• Ensure that index locations are not in backup mode
• Ensure that the Indexing Service is running
• Allow time to clear the backlog
• If the number of items is still rising, there may be an underlying issue with the Indexing Service and will require expert evaluation
• If items are awaiting deletion; expert evaluation MUST be undertaken, it is likely that this will be storage expiry related.

SQL Database Issues:

• Ensure that as part of the backup schedule, the SQL databases and logs are being backed up
• Ensure that maintenance plans are put in place on the EV databases and logs, in order to ensure they are truncated

Storage Issues:
Size Space shortage on disk is quite an obvious issue to resolve (add more space!) However, it is sometimes the case when this cannot be completed. As such, open partitions on the full drive(s) can be closed and new open partitions can be created on new drive(s). As the closed partitions are unlikely to expand by large amounts, the monitoring of these drives can be stopped.

Connection Issues:

• The Exchange configuration needs to reviewed.
• Is the provisioning task scheduled to run?
• Can EV contact the Domain, Domain Controller and/or Catalog Server?

Archiving Issues:
If the archiving task(s) are reporting that they are failing or not archiving any new data there are a number of variables that need to be examined.

• Are Tasks in Report Mode?
• Do tasks have schedules set?
• If the Exchange Server is in a DAG environment, is EV complaining about the passive node that infact does not currently have mailboxes on? If so, this message can be safely ignored. Although the message will continue to report.

Just because the Status is green, doesn’t mean there is nothing wrong
If the status is green or only has status errors that you are aware of, 99% of the time EV should be running fine. However there is always that 1% where there are underlying issues that need to be addressed.

With connection and archiving issues, the Windows Event Viewer is a fantastic place to gather more information about what is failing etc.

The event viewer is also a good place to look for any miscellaneous errors such as permission errors or provisioning issues etc. All errors and warnings should be reviewed. Unfortunately, this blog would last forever if I was to discuss all the possibilities here or even the most common events that occur, to look for or even warnings and errors that can be safely ignored! Each event will need to be analysed case-by-case.

As mentioned, the status review in the Enterprise Vault Administration Console is not the only place to look for to determine issues. Also as mentioned, the Windows Event Viewer is a great place to perform daily checks. On top of these locations, since EV 10.x, there is a reporting folder (defaultly configured to C:Program Files (x86)Enterprise VaultReports) within this, there will be folders holding the Provisioning Task data, the Mailbox Archiving data and the Journal Archiving data. The csv files located within these folders will display all the information from the completed runs of the associated task.

For example, the archiving task report displays the total number of mailboxes targeted, number of mailboxes processed, number of mailboxes with issues and then a breakdown of each mailbox.

The Provisioning Task report is a text file that displays all the provisioning groups and the number of mailboxes that have been processed, ignored or failed.

If all mailboxes are not being processed during the scheduled scans, an increase of the archiving window should be considered.

If the number of items being archived are not reaching the expected level, ensure that items are not stuck in the MSMQ queues (as later discussed) or consider targeting more message classes such as calendar appointments or targeting the deleted items folder in order to reclaim more space from Exchange.

The IIS page http://[ev_server]/enterprisevault/usage.asp displays the number of items and total size of items in the vault stores. This can give a good measure of the archiving rate in monitored regularly. This Summary page also displays the number of items awaiting backup. This number should drop to 0 every time a successful backup is run.

The usage page can also be drilled into to display item information per archive, which can be useful to ensure that mailboxes are being archived and to monitor troublesome mailboxes. Also, the usage page information can be extracted to csv files for further, offline review.

If the awaiting backup is not dropping after a successful backup, the issues in the backup issue section should be resolved.

It is highly advised that MSMQ configuration and changes should only be carried out by Enterprise Vault professionals. However, queues can be monitored to ensure the processes are being carried out correctly.

The associated A1, A3, A5 and A7 queues are particularly of interest. Each queue is dependant on the previous queue, ie. items in A7 will not clear until items in A5 clear. However, while this is the case, the working process is slightly different.

When the provisioning task has been run and the mailboxes are synchronized, the mailboxes are placed in the A7 queue.

When the archiving schedule the mailboxes are queued in A5 and items are archived and are awaiting backup.

When the backup procedure has been carried out and items have been backed up, the items are placed in the A1 queue for shortcut processing and pending items in users mailboxes are converted into accessible archived item shortcuts as the queue is cleared.

The A3 queue is for mailboxes that have been instructed to archive using the ‘Run Now’ feature.

If items ever get stuck in the A5 queue or there is a large backlog of mailboxes in the A5 queue, the Run Now feature should not be used to attempt to mitigate this as it will only add to the backlog.

A somewhat obvious check – Ensure Services and tasks are all running as required and all tasks have schedules, within the EV Servers section of the Vault Administration Console.

Maybe more a monthly review than a daily check, but SQL Reporting can be configured which amongst many other fine features, provides a rolling archival rate per period (day/week/month) which can ensure EV is archiving data as required.

Generally, if everything mentioned above is in order you should have a healthy Enterprise Vault environment. There may be other issues and as you know your user base will be the first to let you know! And we are sorry we can’t cover every scenario here, but aside from there being so many variations of issues each issue will have to be resolved on a case-by-case basis.

If you have enjoyed this document and/or it has resolved an issue or made the archiving aspect of your life easier please be sure to keep us in mind when undertaking future Enterprise Vault projects.