APPLIES TO:

MAIN CONTENT

Purpose

Trace File Analyzer Collector (TFA) is a diagnostic collection utility to simplify diagnostic data collection on Oracle Clusterware/Grid Infrastructure, RAC and Single Instance Database systems.  TFA is similar to the diagcollection utility packaged with Oracle Clusterware in the fact that it collects and packages diagnostic data however TFA is MUCH more powerful than diagcollection with its ability to centralize and automate the collection of diagnostic information.  TFA provides the following key benefits:

• Increase Productivity with Reduction in time and effort in diagnostic collection
  
  • Diagnostic collection is performed with a single command executed from a single node
  • Diagnostics collected include log and trace files for Grid Infrastructure/Oracle Clusterware, ASM, RAC and the OS
  • Output is consolidated to a single node for ease of upload
• Reduce diagnostic data uploads to support by a factor of 10x or morein most cases
  • Diagnostic files are isolated and trimmed to only include data for incident time
• Reduce SR Resolution times
  • Proper diagnostics collection for incidents ensure complete diagnostic data collections thus minimizing the need for support to request additional data

How to Get Started with TFA

1. Download and Install TFA
   
   Note:  Starting with 11.2.0.4 and 12.1.0.2 Grid Infrastructure Patchset, TFA is installed with Grid Infrastructure.  However, TFA is written outside of the GI/RAC/RDBMS product lines and as such could be used for any trace data, and is version agnostic.
2.  
3. Perform Diagnostic Collection - The command below is diagcollect in its most simplistic form.  It will collect and trim log and trace files for the past 4 hours, these logs/traces include: Clusterware/GI, ASM, Database (from all databases) OS, OSWatcher, CHMOS and Procwatcher.  All diagnostic data will be trimmed to this 4 hour window and consolidated for upload to the node in which the diagcollection command was executed on.  There are multiple arguments to diagcollection to isolate time periods and product components, a full listing can be found in the Diagnostics Collection section below as well as the TFA Users Guide.
   
   # ./tfactl diagcollect
    

Requirements

Supported Platforms

TFA is supported on the following platforms:
Note: BASH shell 3.2 and JRE 1.5 or higher are required on all platforms.

• Intel Linux(Enterprise Linux, RedHat Linux, SUSE Linux)
• Linux Itanium
• zLinux
• Oracle Solaris SPARC
• Oracle Solaris x86-64
• AIX
• HPUX Itanium
• HPUX PA-RISC

Supported Oracle Releases

As previously stated TFA is written outside of the GI/RAC/RDBMS product lines and as such could be used for any trace data, and is version agnostic.

Download TFA Collector

The current version of TFA is 12.1.2.1.2.  This version of TFA and the associated TFA Users Guide is available by clicking the relevant Download link below. 

  

	Download TFA

 

	Download TFA Users Guide

 

 

The checksum of TFALite_<version>.zip file can be verified using any checksum utility and should match the cksum/md5sum output below:

$ cksum installTFALite_121212.zip
896609746 14931703 TFALite_121212.zip

Note: As mentioned in the TFA Requirements, JRE 1.5 or higher for the respective platform must be installed on ALL cluster nodes prior to installing TFA.  
If you have Grid Infrastructure installed, the JRE included with GI should be used.  If you are installing TFA on a system not running Grid Infrastructure the relevant JRE downloads are: 

• Linux, Solaris: JRE 1.5 or higher
  

• AIX:	 JRE 1.5 or higher
  

• HP-UX: JRE 1.5 or higher
  

Note: TFA Collector can also be downloaded as part of the RAC and DB Support Tools Bundle (Doc ID 1594347.1)

Configuring

 Please download and review the TFA Collector User Guide for detailed instructions on how to configure and run TFA.  A Quick Start Guide is provided in the Instructions section of this Note.

What's new?

TFA v12.1.2.1.2

• Fix to address CVE-2014-3566 SSL V3.0 Poodle vulnerability.  CVE-2014-3566 affects SSL V3.0 so all components that utlize ssl should be configured to disable SSL V3.0.  TFA uses SSL for communication between TFA daemons and their clients.  From TFA version 12.1.2.1.2 on SSL V3.0 had been disabled.

TFA v12.1.2.1.0

• TFA will now collect trace files even when they have not been allocated a type if their directory is of the right component and they have been updated within the time period requested
• There is now a tfactl shell that can be used to call commands. This is a beta feature that will be enhanced in upcoming releases to provide a flexible user interface
• TFA collections can now be stopped using ‘tfactl collection’
• Diagcollect will not return to the prompt until all nodes have finished their collections
• Critical Bug fixes

TFA v12.1.2.0.0

• Support for Oracle GI and RAC 12.1.0.2 trace files
• TFA Inventory performance improvements
• TFA Collection performance improvements
• New –oda and –odastorage flags for enhanced ODA data collections
• Collection of available oracheck and oratop reports
• More granular trace level adjustment
• Use of a 2048 bit certificate
• Critical Bug fixes

TFA v3.2

• Log Analyzer functionality
• Addition of –notrim for diagnostic collections
• Support for TFA on zLinux
• Collection of ExaWatcher data
• Sundiag collection
• Aging of TFA log files restricting maximum size
• Multiple performance enhancements
• Critical Bug fixes

TFA v3.1

• Non root user collections using TFA
• Collections from Exadata Storage Cells
• Collection of whole directories and bucket directories
• Collection of DBWLM directories and statistics
• Critical Bug fixes

TFA v2.5.1.6

• TFA pre collection inventory will only run an inventory for components to be collected
• Some file types now have the expected date format for timestamps in the file type xml. This means we do not have to determine this which improves inventory performance
• Collection of system data during diagnostic collection is now done in parallel with trace file trimming and collection.
• When TFA 2.5.1.6 is patched as part of a Grid Infrastructure installation or patch it will move the TFA_HOME to the Grid Home and the repository, database and logs to the Grid owners Oracle Base
• Initial install will now run a rediscovery of resources (databases etc) 5 minutes after installation before switching to running a six hour cycle 
• Added confirmation when patching or uninstalling to ensure only the expected nodes are affected
• Installation issues on Single Instance and 12.1 Grid Infrastructure have been corrected
• Bug Fixes

TFA v2.5.1.5

• Improved security to allow sudo running of tfactl
• TFA JVM can now run on one of a number of available ports so will not fail if the preferred port is not available. Also the JVM can run on different ports on each node as required
• All new releases and patches after 2.5.1.5 will be able to be patched from a single node across the cluster without requiring ssh or local patch installs
• TFA can now be installed in a shared file system with the repository for diagnostic collections also supported on a shared file system
• Uninstall now does not remove the repository
• TFA will now collect and correctly classify compressed OSW files
• Additional file type support
• tfactl syntax has been modified to provide a more simple interface and some of the options have been removed from the help and documentation
• Diagcollect now collects last 4 hours worth of data from the time when diagnostic collections is run
• Diagnostic collection nomonitor option, which allows diagcollect to run in the background
• Support added for local uninstall of TFA
• Installation supports deferred discovery in order to speed up installation process
• Installation supports –javahome flag for silent and deferred installs
• TFA uses CRS to automatically add nodes from a GI cluster into TFA
• When installing TFA into GI_HOME – default repository, logs and database are placed under ORACLE_BASE
• Bug Fixes

TFA v2.5.1.3

• Initial Release of TFA

Quick Start Guide

Installing TFA

1. Log into the system as the ROOT user
2. Determine the appropriate location for the TFA installation, this location must exist on all cluster nodes.  A viable location would be $ORACLE_base (for pre-11.2.0.4 and 12.1.0.1).
3. Stage the installTFALite installer on node 1.  After staging the installer you may query the available install flags via ./installTFALite -h:
   
   # ./installTFALite -h
   
      Usage for ./installTFALite
   
      ./installTFALite [-local][-deferdiscovery][-tfabase <install dir>][-javahom
   
           -local            -    Only install on the local node
           -deferediscovery  -    Discover Oracle trace directories after installati
           -tfabase          -    Install into the directory supplied
           -javahome         -    Use this directory for the JRE
           -silent           -    Do not ask any install questions
           -debug            -    Print debug tracing and do not remove TFA_HOME on
   
   
      Note: Without parameters TFA will take you through an interview process for in
            /tfa will be appended to -tfabase if it is not already there.
   
   
      
   Note:  The JRE supplied with Grid Infrastructure (eg., /u01/app/11.2.0/grid/jdk) may be used to fulfill the JRE requirement for the -javahome and would probably be the best option as it is in a predictable location.
     
4. The TFA installer may be launched without any flags allowing for an interactive/interview based installation.  For this example we will specify TFA Base and the Java Home:
   
   Note:  In this example we are installing TFA into $ORACLE_BASE and using the JDK (version 1.5) that ships with Grid Infrastructure.  Do note that TFA will be installed in a directory called tfa within TFA Base.

    
   # ./installTFALite -tfabase /u01/app/oracle -javahome /u01/app/11.2.0/grid/jdk
   
   Starting TFA installation
   
   Using JAVA_HOME : /u01/app/11.2.0/grid/jdk
   
   Running Auto Setup for TFA as user root...
   
   Would you like to do a [L]ocal only or [C]lusterwide installation ? [L|l|C|c] [C] :
   
    
   
   
5. Generally speaking it is easiest to Perform a Clusterwide installation however this does temporarily require SSH equivalency to be configured for the root user.  The TFA Installer will take care of the temporary SSH equivalency configuration and will remove that configuration once the install is complete.  If for some reason you are unable to allow for SSH equivalency as root (temporarily), you may perform a local install of TFA on every node (Please see the TFA Users Guide on how to do so).
   
   Would you like to do a [L]ocal only or [C]lusterwide installation ? [L|l|C|c] [C] : C
   
   The following installation requires temporary use of SSH.
   If SSH is not configured already then we will remove SSH
   when complete.
     Do you wish to Continue ? [Y|y|N|n] [Y] Y
   
   Installing TFA now...
   
   Discovering Nodes and Oracle resources
   
   Checking whether CRS is up and running
   
   Getting list of nodes in cluster . . . . .
   
   List of nodes in cluster
   1. cetrain01
   2. cetrain02
   
   Checking ssh user equivalency settings on all nodes in cluster
   
   Node cetrain02 is configured for ssh user equivalency for root user
   
   
6. Follow the remaining prompts to complete the installation.
   
   Searching for running databases . . . . .
   
   .
   List of running databases registered in OCR
   1. ORCL
   . .
   
   
   Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS
   
   . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
   
   
   TFA Will be Installed on the Following Nodes:
   ++++++++++++++++++++++++++++++++++++++++++++
   
   Install Nodes
   =============
   cetrain01
   cetrain02
   
   Do you wish to make changes to the Node List ? [Y/y/N/n] [N] N
   
   TFA will scan the following Directories
   ++++++++++++++++++++++++++++++++++++++++++++
   
   .-------------------------------------------------------------------------.
   |                                cetrain02                                |
   +--------------------------------------------------------------+----------+
   | Trace Directory                                              | Resource |
   +--------------------------------------------------------------+----------+
   | /u01/app/11.2.0/grid/OPatch/crs/log                          | CRS      |
   | /u01/app/11.2.0/grid/cfgtoollogs                             | INSTALL  |
   | /u01/app/11.2.0/grid/crs/log                                 | CRS      |
   ....<snip>....
   
   Installing TFA on cetrain01:
   HOST: cetrain01 TFA_HOME: /u01/app/oracle/tfa/cetrain01/tfa_home
   
   Installing TFA on cetrain02:
   HOST: cetrain02 TFA_HOME: /u01/app/oracle/tfa/cetrain02/tfa_home
   
   .-------------------------------------------------------------------------.
   | Host      | Status of TFA | PID   | Port | Version | Build ID           |
   +-----------+---------------+-------+------+---------+--------------------+
   | cetrain01 | RUNNING       | 32242 | 5000 |     3.1 | 310020140205043544 |
   | cetrain02 | RUNNING       | 22098 | 5000 |     3.1 | 310020140205043544 |
   '-----------+---------------+-------+------+---------+--------------------'
   
   Running Inventory in All Nodes...
   
   Enabling Access for Non-root Users on cetrain01...
   
   Adding default users and groups to TFA Access list...
   
   Summary of TFA Installation:
   .--------------------------------------------------------------.
   |                           cetrain01                          |
   +---------------------+----------------------------------------+
   | Parameter           | Value                                  |
   +---------------------+----------------------------------------+
   | Install location    | /u01/app/oracle/tfa/cetrain01/tfa_home |
   | Repository location | /u01/app/oracle/tfa/repository         |
   | Repository usage    | 4 MB out of 992 MB                     |
   '---------------------+----------------------------------------'
   
   .--------------------------------------------------------------.
   |                           cetrain02                          |
   +---------------------+----------------------------------------+
   | Parameter           | Value                                  |
   +---------------------+----------------------------------------+
   | Install location    | /u01/app/oracle/tfa/cetrain02/tfa_home |
   | Repository location | /u01/app/oracle/tfa/repository         |
   | Repository usage    | 1 MB out of 992 MB                     |
   '---------------------+----------------------------------------'
   
   
   TFA is successfully installed...
   
   
   

Starting and Stopping TFA

TFA runs out of init on Linux and Unix platforms so that it will be started automatically whenever a server is started.  The Name and location of the TFA init script (init.tfa) is dependent on the platform:

• Linux and Solaris: /etc/init.d/init.tfa
• Aix: /etc/init.tfa
• HP-UX: /sbin/init.d/init.tfa

Note:  All examples below are on the Linux platform, the location of the init.tfa script for other platforms is displayed above.

To start TFA execute the init.tfa script with the start argument as the root user:

 To stop TFA execute the init.tfa script with the stop argument as the root user:

 To restart TFA (stop and restart in a single command) execute the init.tfa script with the restart argument as the root user:

Diagnostics Collection

The diagnostic collection module is invoked by tfactl with the diagcollect verb.  The diagcollection module has the capability to take a number of different parameters to allow the user to control how large or detailed the required evidence set is.   It is possible to provide a specific time of an incident  or a time range for data to be collected.  It is also possible to collect whole files that have relevant data or "trim" the files in the evidence set to just a time slice of data. 

The tfactl parameters and examples for data collection can be viewed by invoking "tfactl diagcollect -h":

 In the following example, we use the -all, -from and -to switches to tfactl diagcollect which tells TFA to collect diagnostic logs of all types from midnight Feb 5th to 13:00 on Feb 5th.  The command will launch the specified diag collection on ALL cluster nodes and neatly places them in a zip file for each node under $TFA_HOME/repository on the TFA master node (node 1 in our case).

By default diagcollection does display the collection status in the foreground (as shown above), this behavior may be modified by specifying the -nomonitor flag with diagcollection:

Automatic Diagnostic Collection

TFA does have the capability to automatically perform diagnostics collection when an incident is detected by TFA.  This feature eases the diagnostic collection burdon by covers the situations where diagnostic data would have rotated past the point of this After the initial inventory is completed by TFA all files that are determined to be Alert Logs are monitored in real time so that TFA can take action when certain messages are seen. By default these logs are RDBMS alert logs, ASM alert logs and CRS alert logs. When specific strings (pre-defined in TFA) are found in the logs information on the strings is saved to the Berkeley Database and when enabled, Automatic Diagnostic Collection will be triggered. Exactly what is collected is dependent on the string match that was found but potentially trimmed versions of all O/S, CRS, ASM and RDBMS logs could be collected for each string. Clearly this type of operation could be called many times a second if not controlled so there can never be a collection generated at less than 5 minute intervals due to the TFA implementation of flood control. Also note that if the TFA repository reaches its repository max size then Automatic Diagnostic Collection will be disabled until sufficient space is free.

Managing Automatic Diagnostic Collection

To Enable or Disable Automatic Diagnostic Collection (disabled by default), the autodiagcollect TFA parameter simply needs to be set to ON or OFF.  When set to OFF (default) automatic diagnostic collection will be disabled. If set to ON, diagnostics will be collected when the required search strings (see User Guide) are detected whilst scanning the alert logs. To set automatic collection for all nodes of the TFA cluster the -c flag must be used:

The trimming of files collected by Automatic Diagnostic Collection can be controlled by the trimfiles TFA parameter.  When ON (default) files will be trimmed to only include data around the time of the event, when OFF any file that was written to at the time of the event (that also pertains to the event as determined by TFA) will be collected in its entirety. To set trimfiles for all nodes of the TFA cluster the ‘-c’ flag must be used:

TFA Repository Space Management

By default, maximum repository size is 10 GB or 50% of the filesystem size that the TFA Repository resides on (whichever one is greater). If the TFA repository filesystem free space is reduced to 1GB the TFA respository will be closed until space is freed.  There is no in-built automatic way to manage TFA repository space at this point. Please make sure to monitor TFA repository space usage. The "tfactl print config" command shows current and maximum size of repository.

 To remove files from the TFA Repostitory the "tfactl purge" command should be used with the -older #[d][h] (d is days, h is hours).  To remove diagcollections older than 30 days you would execute:

Patching/Upgrading TFA Collector

Once TFA Collector 2.5.1.6 has been installed patching is accomplished by running the install script for the new version on one node and the other nodes will be patched automatically with no requirement for ssh root user equivalence. 

Please see the TFA Collector User Guide for more details and how to patch older versions of TFA Collector.

TFA commands at a glance

Note: For detailed information regarding the avilable TFA commands please refer to the TFA Collector User Guide which is available in the Download section of this document.

Overall help for TFA can be displayed with:

Print Details on the TFA configuration:

 Add/Remove/Modify directories known to TFA:

 Purge TFA Repository data:

 Add/Remove Hosts within the TFA Configuration:

Modify TFA Parameters:

 Add and Remove TFA users/groups

Uploading Files to Service Requests in MOS

Regardless of the Diagnostic Collection method used (Manual or Automatic), tfactl diagcollect collects and stores collected diagnostics in the repository. Repository location can be obtained using the “tfactl print repository” command.  The files that reside in the repository directory are those that would be uploaded to MOS for a particular incident. If TFA is installed in Grid Infrastructure home, repository would be located in ORACLE_BASE/tfa directory:

Known Issues

1.  Upper-case TFA directory name leads to an error during install.  Example:

Installing TFA on ao1

HOST: ao1       TFA_HOME: /u01/app/TFA/ao1/tfa_home

Can't open file /u01/app/TFA/ao1/bin/tfactl: Not a directory     <---- here is the error

Removing TFA Setup files...

Fixed: version 2.5.1.6.  

Work-around for earlier version: change the directory name from TFA to tfa (lower-case)

2. Bug 18123522 : TFA SSL CERTIFICATE EXPIRY IN PUBLIC.JKS

Fixed: version 3.1