Troubleshooting Symantec File Reader Restarts

FileReader restarts usually occur due to timeout issues.  These timeouts are generally caused by the following:

1. Connectivity issues with the Monitor Controller or Network
2. Poorly written RegEx Rules
3. A bad email message.  Bad messages can be caused by incorrect header information or foreign characters in the message that Symantec DLP is unable to process

If the FileReader restarts itself occasionally, this is normal behavior.  However, if you are experiencing consistent FileReader restarts in your environment, there are a few things you can do to determine the cause:

1. FileReader may fail to start (and restart) if it can’t receive all the configuration information it needs. To troubleshoot the exact cause, look in the FileReader log first to identify which FileReader subsystem isn’t starting. Once it’s identified that a particular subsystem isn’t receiving its configuration, one should look in the MonitorController log to see if the corresponding subsystem has been initialized successfully. One of the common failures is inability to ignite cryptographic keys in the MonitorController because the ignition password on the disk got out of sync with the Administrator password in the database. In this case the password issue must be fixed and only after that should the MonitorController be restarted. Look at the VontuMonitorController.conf file in the config directory. Check the java heap size. If it is the default value of 128 and 256, you will probably need to increase the memory setting of the heap depending on the RAM available on the server.
2. Too many exceptions in policies. Each new exception, while improving the ability to “short-circuit” detection, also has the knock-on effect of multiplying the size of the overall detection matrix – all of which is loaded into memory when FileReader starts. While there is no exact limit on the total number of exception, a good rule of thumb is that more than 10 exceptions in a policy will start to have an impact at some point – after which the next exception may result in FileReader being unable to load.
3. Check to make sure the MessageChain.CacheSize & MessageChain.NumChains match the CPU Cores.
4. Check your policies.  Oftentimes FileReader restarts will occur because of a particular policy.  For example, if a Regex in a particular policy exceeds given thresholds (such as maximum component time), then the FileReader will restart.  Look at the log files for the “intentionally restarting process” message which identifies the message chain component causing the restart.  If this component is “Detection” the most likely cause is a poorly written regular expression.
5. Check for “bad” messages. Save the *.vpcap file that contains the message in question. You can use the file for testing without having to actually send the message again.
6. Check for locked *.vpcap files.
   1. Stop Packet Capture so that you do not get noise in the test. Start FileReader process. If the *.vpcap file gets picked up, the inductor is working. If the inductor is not working, find out why. The most common problem is that some process has a lock on the files. Other than that, collect the FileReader log and contact support.
7. If the inductor is working, the problem may be in the Layer 7 Parser or the Content Extractor. Visually inspect the FileReader log for any exceptions, warnings or severe log messages.
8. While the Content Extractor can often have problems processing various file formats it can rarely, if ever, be blamed for a FileReader restart.
9. Dying threads can cause FileReader to stop reporting heartbeats and eventually be restarted. Look in VontuMonitor.log for exceptions. Each exception in that log file is an indicator of a serious problem (Java crash or other defect) and is a likely cause of a FileReader restart.