In my work for Oracle RAC Support, I have noticed that people sometimes identify the wrong background process trace files… for example, uploading an old LMON trace file instead of the current one. Here are some one-liners to identify the current background traces.

Oracle’s database trace files follow this naming convention:

<sid>_<ospid>_<processname>.trc

For foreground processes, the <processname> is “ora”. For background processes, it’s the four-letter short name by which that background process is known – for eg, PMON for Process Monitor. (Here’s a list of all the background processes in 12c.)

Whenever the instance is restarted, the OS PIDs change, so you can wind up with (for example) several tracefiles for “pmon” in the trace directory.

mysid_pmon_1234.trc 
mysid_pmon_2345.trc 
mysid_pmon_3456.trc

This can be a bit of a pain in RAC when you want to look at the traces for your six lmsN background processes – or when Oracle Support asks for them! – and find you have dozens of *lms* files to pick through.

There are two easy ways to find the background process trace files you want:

1. Grep the files for entries on a certain date

eg. to list all trace files containing any entries for 2014-07-29 11 am – 12 noon:

grep '2014-07-29 11:'  *trc | awk -F: '{print $1}' | uniq

Output:

mysid_pmon_1234.trc
mysid_diag_2345.trc
mysid_lgwr_3456.trc
mysid_lmd0_4567.trc
...

2. Grep the alert log for the OS PIDs of the background processes (excluding some non-fatal procs which are stopped and started all the time)

grep "started with pid" alert_mysid.log | egrep -v "VKRM|DW0|DM0"

Output:

PMON started with pid=2, OS id=1234
PSP0 started with pid=3, OS id=2345
... etc ...

Of course, this will give you ALL the OS PIDs of all startups in the alert log, so look at the last entry only. If there’s more than one startup, you can add the “Starting Oracle” to the output:

egrep "Starting O|started with pid" alert_mysid.log | egrep -v "VKRM|DW0|DM0"