This chapter provides reference information about system accounting.This is a list of the reference information in this chapter.runacct Script Daily Accounting Reports System Accounting Files For more information about system accounting tasks, see Chapter 9, Managing System Accounting (Tasks). The main daily accounting script, runacct, is normally invoked by the cron command outside of normal business hours. The runacct script processes connect, fee, disk, and process accounting files. This script also prepares daily and cumulative summary files for use by the prdaily and monacct scripts for billing purposes. The runacct script takes care not to damage files if errors occur. A series of protection mechanisms that are used to perform the following tasks:Recognize an error Provide intelligent diagnostics Complete processing in such a way that the runacct script can be restarted with minimal intervention This script records its progress by writing descriptive messages to the active file. Files used by the runacct script are assumed to be in the /var/adm/acct/nite directory, unless otherwise noted. All diagnostic output during the execution of the runacct script is written to the fd2log file. When the runacct script is invoked, it creates the lock and lock1 files. These files are used to prevent simultaneous execution of the runacct script. The runacct program prints an error message if these files exist when it is invoked. The lastdate file contains the month and day the runacct script was last invoked, and is used to prevent more than one execution per day. If the runacct script detects an error, the following occurs: A message is written to the console Email is sent to root and adm Locks might be removed Diagnostics are saved Execution is ended For instructions on how to restart the runacct script, see How to Restart the runacct Script.To allow the runacct script to be restarted, processing is broken down into separate re-entrant states. The statefile file is used to track the last state completed. When each state is completed, the statefile file is updated to reflect the next state. After processing for the state is complete, the statefile file is read and the next state is processed. When the runacct script reaches the CLEANUP state, it removes the locks and ends. States are executed as shown in the following table. State Description SETUP The turnacct switch command is executed to create a new pacct file. The /var/adm/pacctn process accounting files (except for the pacct file) are moved to the /var/adm/Spacctn.MMDD files. The /var/adm/wtmpx file is moved to the /var/adm/acct/nite/wtmp.MMDD file (with the current time record added on the end) and a new /var/adm/wtmp file is created. The closewtmp and utmp2wtmp programs add records to the wtmp.MMDD file and the new wtmpx file to account for users who are currently logged in. WTMPFIX The wtmpfix program checks the wtmp.MMDD file in the nite directory for accuracy. Because some date changes cause the acctcon program to fail, the wtmpfix program attempts to adjust the time stamps in the wtmpx file if a record of a date change appears. This program also deletes any corrupted entries from the wtmpx file. The fixed version of the wtmp.MMDD file is written to the tmpwtmp file. CONNECT The acctcon program is used to record connect accounting records in the file ctacct.MMDD. These records are in tacct.h format. In addition, the acctcon program creates the lineuse and reboots files. The reboots file records all the boot records found in the wtmpx file. PROCESS The acctprc program is used to convert the /var/adm/Spacctn.MMDD process accounting files into complete accounting records in the ptacctn.MMDD files. The Spacct and ptacct files are correlated by number so that if the runacct script fails, the Spacct files are not processed. MERGE The acctmerg program merges the process accounting records with the connect accounting records to form the daytacct file. FEES The acctmerg program merges ASCII tacct records from the fee file into the daytacct file. DISK The dodisk script produces the disktacct file. If the dodisk script has been run, which produces the disktacct file, the DISK program merges the file into the daytacct file and moves the disktacct file to the /tmp/disktacct.MMDD file. MERGETACCT The acctmerg program merges the daytacct file with the sum/tacct file, the cumulative total accounting file. Each day, the daytacct file is saved in the sum/tacct.MMDD file so that the sum/tacct file can be re-created if it is corrupted or lost. CMS The acctcms program is run several times. This program is first run to generate the command summary by using the Spacctn files and write the data to the sum/daycms file. The acctcms program is then run to merge the sum/daycms file with the sum/cms cumulative command summary file. Finally, the acctcms program is run to produce nite/daycms and nite/cms, the ASCII command summary files from the sum/daycms and sum/cms files, respectively. The lastlogin program is used to create the /var/adm/acct/sum/loginlog log file. This file reports when each user last logged in. If the runacct script is run after midnight, the dates showing the time last logged in by some users will be incorrect by one day. USEREXIT Any installation-dependent (local) accounting program can be run at this point. The runacct script expects this program to be called the /usr/lib/acct/runacct.local program. CLEANUP This state cleans up temporary files, runs the prdaily script and saves its output in the sum/rpt.MMDD file, removes the locks, and then exits. When restarting the runacct script in the CLEANUP state, remove the last ptacct file because this file will not be complete. The runacct shell script generates five basic reports upon each invocation. The following table describes these reports.Report Type Description Daily Report Shows terminal line utilization by tty number. Daily Usage Report Indicates usage of system resources by users (listed in order of user ID). Daily Command Summary Indicates usage of system resources by commands, listed in descending order of memory use. In other words, the command that used the most memory is listed first. This same information is reported for the month in the monthly command summary. Monthly Command Summary A cumulative summary that reflects the data accumulated since the last invocation of the monacct program. Last Login Report Shows the last time each user logged in (listed in chronological order). This report gives information about each terminal line used. The following is a sample Daily Report. Jan 16 02:30 2004 DAILY REPORT FOR venus Page 1 from Mon Jan 15 02:30:02 2004 to Tue Oan 16 02:30:01 2004 1 runacct 1 acctcon TOTAL DURATION IS 1440 MINUTES LINE MINUTES PERCENT # SESS # ON # OFF console 868 60 1 1 2 TOTALS 868 -- 1 1 2The from and to lines specify the time period reflected in the report. This time period covers the time the last Daily Report was generated to the time the current Daily Report was generated. Then, the report presents a log of system reboots, shutdowns, power failure recoveries, and any other record written to the /var/adm/wtmpx file by the acctwtmp program. For more information, see the acct1M man page.The second part of the report is a breakdown of terminal line utilization. The TOTAL DURATION tells how long the system was in multiuser mode (accessible through the terminal lines). The following table describes the data provided by the Daily Report.Column Description LINE The terminal line or access port. MINUTES The number of minutes that the line was in use during the accounting period. PERCENT The TOTAL DURATION divided by the number of MINUTES. # SESS The number of times this line or port was accessed for a login session. # ON Same as SESS. (This column no longer has meaning. Previously, this column listed the number of times that a line or port was used to log in a user.) # OFFT The number of times a user logs out and any interrupts that occur on that line. Generally, interrupts occur on a port when ttymon is first invoked after the system is brought to multiuser mode. If the # OFF exceeds the # SESS by a large factor, the multiplexer, modem, or cable is probably going bad. Or, a bad connection exists somewhere. The most common cause is an unconnected cable dangling from the multiplexer. During real time, you should monitor the /var/adm/wtmpx file because it is the file from which the connect accounting is derived. If the wtmpx file grows rapidly, execute the following command to see which tty line is the noisiest.# /usr/lib/acct/acctcon -l file < /var/adm/wtmpxIf interruption is occurring frequently, general system performance will be affected. Additionally, the wtmp file might become corrupted. To correct this problem, see How to Fix a Corrupted wtmpx File. The Daily Usage Report breaks down system resource utilization by user. A sample of this report follows. Jan 16 02:30 2004 DAILY USAGE REPORT FOR skisun Page 1 LOGIN CPU (MINS) KCORE- MINS CONNECT (MINS) DISK # OF # OF # DISK FEE UID NAME PRIME NPRIME PRIME NPRIME PRIME NPRIME BLOCKS PROCS SESS SAMPLES 0 TOTAL 72 148 11006173 51168 26230634 57792 539 330 0 2150 1 0 root 32 76 11006164 33664 26230616 22784 0 0 0 127 0 4 adm 0 0 22 51 0 0 0 420 0 0 0 101 rimmer 39 72 894385 1766020 539 330 0 1603 1 0 0The following table describes the data provided by the Daily Usage Report.Column Description UID User ID number. LOGIN NAME Login (or user) name of the user. Identifies a user who has multiple login names. CPU (MINS) Amount of time, in minutes, that the user's process used the central processing unit. Divided into PRIME and NPRIME (nonprime) utilization. The accounting system's version of this data is located in the /etc/acct/holidays file. KCORE-MINS A cumulative measure of the amount of memory in Kbyte segments per minute that a process uses while running. Divided into PRIME and NPRIME utilization. CONNECT (MINS) Amount of time, in minutes, that the a user was logged in to the system, or “real time.” Divided into PRIME and NPRIME utilization. If these numbers are high while the # OF PROCS is low, you can conclude that the user logs in first thing in the morning and hardly touches the terminal the rest of the day. DISK BLOCKS Output from the acctdusg program, which runs the disk accounting programs and merges the accounting records (daytacct). For accounting purposes, a block is 512 bytes. # OF PROCS Number of processes invoked by the user. If large numbers appear, a user might have a shell procedure that has run out of control. # OF SESS Number of times that a user logged in to the system. # DISK SAMPLES Number of times that disk accounting was run to obtain the average number of DISK BLOCKS. FEE Often unused field that represents the total accumulation of units charged against the user by the chargefee script. The Daily Command Summary report shows the system resource utilization by command. With this report, you can identify the most heavily used commands. Based on how those commands use system resources, you can then gain insight on how best to tune the system. These reports are sorted by TOTAL KCOREMIN, which is an arbitrary gauge but often useful for calculating drain on a system.A sample Daily Command Summary follows. TOTAL COMMAND SUMMARY COMMAND NUMBER TOTAL TOTAL TOTAL MEAN MEAN HOG CHARS BLOCKS NAME CMDS KCOREMIN CPU-MIN REAL-MIN SIZE-K CPU-MIN FACTOR TRNSFD READ TOTALS 2150 1334999.75 219.59 724258.50 6079.48 0.10 0.00 397338982 419448 netscape 43 2456898.50 92.03 54503.12 26695.51 2.14 0.00 947774912 225568 adeptedi 7 88328.22 4.03 404.12 21914.95 0.58 0.01 93155160 8774 dtmail 1 54919.17 5.33 17716.57 10308.94 5.33 0.00 213843968 40192 acroread 8 31218.02 2.67 17744.57 11682.66 0.33 0.00 331454464 11260 dtwm 1 16252.93 2.53 17716.57 6416.05 2.53 0.00 158662656 12848 dtterm 5 4762.71 1.30 76300.29 3658.93 0.26 0.00 33828352 11604 dtaction 23 1389.72 0.33 0.60 4196.43 0.01 0.55 18653184 539 dtsessio 1 1174.87 0.24 17716.57 4932.97 0.24 0.00 23535616 5421 dtcm 1 866.30 0.18 17716.57 4826.21 0.18 0.00 3012096 6490The following table describes the data provided by the Daily Command Summary.Column Description COMMAND NAME Name of the command. All shell procedures are lumped together under the name sh because only object modules are reported by the process accounting system. You should monitor the frequency of programs called a.out or core, or any other unexpected name. You can use the acctcom program to determine who executed an oddly named command and if superuser privileges were used. NUMBER CMDS Total number of times this command was run. TOTAL KCOREMIN Total cumulative measurement of the Kbyte segments of memory used by a process per minute of run time. TOTAL CPU-MIN Total processing time this program accumulated. TOTAL REAL-MIN Total real-time (wall-clock) minutes this program accumulated. MEAN SIZE-K Mean (average) of the TOTAL KCOREMIN over the number of invocations reflected by the NUMBER CMDS. MEAN CPU-MIN Mean (average) derived from the NUMBER CMDS and the TOTAL CPU-MIN. HOG FACTOR Total CPU time divided by elapsed time. Shows the ratio of system availability to system utilization, providing a relative measure of total available CPU time consumed by the process during its execution. CHARS TRNSFD Total number of characters transferred by the read and write system calls. Might be negative due to overflow. BLOCKS READ Total number of the physical block reads and writes that a process performed. The format of the Daily Command Summary and the Monthly Command Summary reports are virtually the same. However, the daily summary reports only on the current accounting period while the monthly summary reports on the start of the fiscal period to the current date. In other words, the monthly report is a cumulative summary that reflects the data accumulated since the last invocation of the monacct program.A sample Monthly Command Summary follows. Jan 16 02:30 2004 MONTHLY TOTAL COMMAND SUMMARY Page 1 TOTAL COMMAND SUMMARY COMMAND NUMBER TOTAL TOTAL TOTAL MEAN MEAN HOG CHARS BLOCKS NAME CMDS KCOREMIN CPU-MIN REAL-MIN SIZE-K CPU-MIN FACTOR TRNSFD READ TOTALS 42718 4398793.50 361.92 956039.00 12154.09 0.01 0.00 16100942848 825171 netscape 789 3110437.25 121.03 79101.12 25699.58 0.15 0.00 3930527232 302486 adeptedi 84 1214419.00 50.20 4174.65 24193.62 0.60 0.01 890216640 107237 acroread 145 165297.78 7.01 18180.74 23566.84 0.05 0.00 1900504064 26053 dtmail 2 64208.90 6.35 20557.14 10112.43 3.17 0.00 250445824 43280 dtaction 800 47602.28 11.26 15.37 4226.93 0.01 0.73 640057536 8095 soffice. 13 35506.79 0.97 9.23 36510.84 0.07 0.11 134754320 5712 dtwm 2 20350.98 3.17 20557.14 6419.87 1.59 0.00 190636032 14049For a description of the data provided by the Monthly Command Summary, see Daily Command Summary. This report gives the date when a particular login was last used. You can use this information to find unused logins and login directories that can be archived and deleted. A Last Login Report follows.Jan 16 02:30 2004 LAST LOGIN Page 1 01-06-12 kryten 01-09-08 protoA 01-10-14 ripley 01-07-14 lister 01-09-08 protoB 01-10-15 scutter1 01-08-16 pmorph 01-10-12 rimmer 01-10-16 scutter2 At any time, you can examine the contents of the /var/adm/pacctn files, or any file with records in the acct.h format, by using the acctcom program. If you do not specify any files and do not provide any standard input when you run this command, the acctcom command reads the pacct file. Each record read by the acctcom command represents information about a terminated process. Active processes can be examined by running the ps command. The default output of the acctcom command provides the following information:# acctcom COMMAND START END REAL CPU MEAN NAME USER TTYNAME TIME TIME (SECS) (SECS) SIZE(K) #accton root ? 02:30:01 02:30:01 0.03 0.01 304.00 turnacct adm ? 02:30:01 02:30:01 0.42 0.01 320.00 mv adm ? 02:30:01 02:30:01 0.07 0.01 504.00 utmp_upd adm ? 02:30:01 02:30:01 0.03 0.01 712.00 utmp_upd adm ? 02:30:01 02:30:01 0.01 0.01 824.00 utmp_upd adm ? 02:30:01 02:30:01 0.01 0.01 912.00 utmp_upd adm ? 02:30:01 02:30:01 0.01 0.01 920.00 utmp_upd adm ? 02:30:01 02:30:01 0.01 0.01 1136.00 utmp_upd adm ? 02:30:01 02:30:01 0.01 0.01 576.00 closewtm adm ? 02:30:01 02:30:01 0.10 0.01 664.00Field Explanation COMMAND NAME Command name (pound (#) sign if the command was executed with superuser privileges) USER User name TTYNAME tty name (listed as ? if unknown) START TIME Command execution starting time END TIME Command execution ending time REAL (SECS) Real time (in seconds) CPU (SECS) CPU time (in seconds) MEAN SIZE (K) Mean size (in Kbytes) You can obtain the following information by using acctcom command options.State of the fork/exec flag (1 for fork without exec) System exit status Hog factor Total kcore minutes CPU factor Characters transferred Blocks read The following table describes the acctcom command options.Option Description a Shows average statistics about the processes selected. The statistics are printed after the output is recorded. b Reads the files backward, showing latest commands first. This option has no effect if reading standard input. f Prints the fork/exec flag and system exit status columns. The output is an octal number. h Instead of mean memory size, shows the hog factor, which is the fraction of total available CPU time consumed by the process during its execution. Hog factor = total-CPU-time/elapsed-time. i Prints columns that contains the I/O counts in the output. k Shows total kcore minutes instead of memory size. m Shows mean core size. This size is the default. q Prints average statistics, not output records. r Shows CPU factor: user-time/(system-time + user-time). t Shows separate system and user CPU times. v Excludes column headings from the output. C sec Shows only processes with total CPU time (system plus user) that exceeds sec seconds. e time Shows processes existing at or before time, given in the format hr[:min[:sec]] E time Shows processes starting at or before time, given in the format hr[:min[:sec]]. Using the same time for both S and E, shows processes that existed at the time. g group Shows only processes that belong to group. H factor Shows only processes that exceed factor, where factor is the “hog factor” (see the h option). I chars Shows only processes that transferred more characters than the cutoff number specified by chars. l line Show only processes that belong to the terminal /dev/line. n pattern Shows only commands that match pattern (a regular expression except that “+” means one or more occurrences). o ofile Instead of printing the records, copies them in acct.h format to ofile. O sec Shows only processes with CPU system time that exceeds sec seconds. s time Show processes existing at or after time, given in the format hr[:min[:sec]]. S time Show processes starting at or after time, given in the format hr[:min[:sec]]. u user Shows only processes that belong to user. The /var/adm directory contains the active data collection files. The following table describes the accounting files in this directory. File Description dtmp Output from the acctdusg program fee Output from the chargefee program, which are the ASCII tacct records pacct Active process accounting file pacctn Process accounting files that are switched by running the turnacct script Spacctn.MMDD Process accounting files for MMDD during execution of the runacct script The /var/adm/acct directory contains the nite, sum, and fiscal directories. These directories contain the actual data collection files. For example, the nite directory contains files that are reused daily by the runacct script. A brief summary of the files in the /var/adm/acct/nite directory follows. File Description active Used by the runacct script to record progress and print warning and error messages active.MMDD Same as the active file after the runacct script detects an error cms ASCII total command summary used by the prdaily script ctacct.MMDD Connect accounting records in tacct.h format ctmp Output of acctcon1 program, which consists of connect session records in ctmp.h format (acctcon1 and acctcon2 are provided for compatibility purposes) daycms ASCII daily command summary used by the prdaily script daytacct Total accounting records for one day in tacct.h format disktacct Disk accounting records in tacct.h format, created by the dodisk script fd2log Diagnostic output during execution of the runacct script lastdate Last day the runacct script executed (in date +%m%d format) lineuse tty line usage report used by the prdaily script lock Used to control serial use of the runacct script log Diagnostic output from the acctcon program log.MMDD Same as the log file after the runacct script detects an error owtmpx Previous day's wtmpx file reboots Beginning and ending dates from the wtmpx file, and a listing of reboots statefile Used to record current state during execution of the runacct script tmpwtmp wtmpx file corrected by the wtmpfix program wtmperror Contains wtmpfix error messages wtmperror MMDD Same as the wtmperror file after the runacct script detects an error wtmpMMDD The runacct script's copy of the wtmpx file The sum directory contains the cumulative summary files updated by the runacct script and used by the monacct script. The following table summarizes the files in the /var/adm/acct/sum directory. File Description cms Total command summary file for current fiscal period in binary format cmsprev Command summary file without latest update daycms Command summary file for the day's usage in internal summary format loginlog Record of last date each user logged in; created by the lastlogin script and used in the prdaily script rprt.MMDD Saved output of prdaily script tacct Cumulative total accounting file for current fiscal period tacctprev Same as the tacct file without latest update tacct.MMDD Total accounting file for MMDD The fiscal directory contains periodic summary files that are created by the monacct script. The following table summarizes the files in the /var/adm/acct/fiscal directory. File Description cmsn Total command summary file for fiscal period n in internal summary format fiscrptn Report similar to rprtn for fiscal period n tacctn Total accounting file for fiscal period n The following table summarizes the most useful files produced by the runacct script. These files are found in the /var/adm/acct directory. File Description nite/daytacct The total accounting file for the day in tacct.h format. nite/lineuse The runacct script calls the acctcon program to gather data on terminal line usage from the /var/adm/acct/nite/tmpwtmp file and writes the data to the /var/adm/acct/nite/lineuse file. The prdaily script uses this data to report line usage. This report is especially useful for detecting bad lines. If the ratio between the number of logouts to logins is greater than three to one, the line is very likely failing. sum/cms This file is the accumulation of each day's command summaries. The accumulation restarts when the monacct script is executed. The ASCII version is the nite/cms file. sum/daycms The runacct script calls the acctcms program to process the commands used during the day to create the Daily Command Summary report and stores the data in the /var/adm/acct/sum/daycms file. The ASCII version is the /var/adm/acct/nite/daycms file. sum/loginlog The runacct script calls the lastlogin script to update the last date logged in for the logins in the /var/adm/acct/sum/loginlog file. The lastlogin command also removes from this file any logins that are no longer valid. sum/rprt.MMDD Each execution of the runacct script saves a copy of the daily report that was printed by the prdaily script. sum/tacct Contains the accumulation of each day's nite/daytacct data and is used for billing purposes. The monacct script restarts accumulating this data each month or fiscal period.