This chapter describes how to set up and maintain system accounting.This is a list of the overview information in this chapter.What is System Accounting? Setting Up System Accounting For information on using extended accounting, see Chapter 4, Extended Accounting (Overview), in System Administration Guide: Virtualization Using the Solaris Operating System.For information on the step-by-step procedures that are associated with system accounting, see System Accounting (Task Map).For reference information about the various system accounting reports, see Chapter 10, System Accounting (Reference). This section describes new or changed features in system accounting in the Solaris release.Solaris 10: Changes have been made to the internals of the load averaging, cpu usr/sys/idle, and accounting functions. Microstate accounting has replaced the old accounting mechanism and is enabled by default all of the time. As a result, you might notice slightly different process usage and timing statistics.Switching to microstate accounting provides substantially more accurate data about user processes and the amount of time they spend in various states. In addition, this information is used to generate more accurate load averages and statistics from the /proc file system. For more information, see the proc4 man page. System accounting software in the Solaris OS is a set of programs that enables you to collect and record data about user connect time, CPU time charged to processes, and disk usage. Once you collect this data, you can generate reports and charge fees for system usage.You can use system accounting on a daily or monthly basis. Or, you can tack disk usage per user.You can use the accounting programs to perform these tasks:Monitor system usage Locate and correct performance problems Maintain system security After you set up the system accounting programs, they run mostly on their own.Automatic accounting is set up by first putting the accounting startup script into root's crontab file. The accounting startup script can then be started automatically by the cron command.The following overview describes the system accounting process.Between system startup and shutdown, raw data about system use (such as user logins, running processes, and data storage) are collected in accounting files. Periodically (usually once a day), the /usr/lib/acct/runacct script processes the various accounting files and produces both cumulative summary files and daily accounting reports. Then, the /usr/lib/acct/prdaily script prints the daily reports.For more information about the runacct script, see runacct Script. Monthly, you can process and print the cumulative runacct summary files by executing the monacct script. The summary reports produced by the monacct script provide an efficient means for billing users on a monthly or other fiscal basis. The system accounting software provides C language programs and shell scripts that organize data into summary files and reports. These programs reside in the /usr/lib/acct directory. The accounting reports reside in the /var/adm/acct directory.Daily accounting can help you perform four types of auditing:Connect accounting Process accounting Disk accounting Fee calculations Connect accounting enables you to determine the following information:The length of time a user was logged in How the tty lines are being used The number of reboots on your system How many times the accounting software was turned off and on To provide this information on connect sessions, the system stores the following dataRecord of time adjustments Boot times Number of times the accounting software was turned off and on Changes in run levels The creation of user processes (login processes and init processes) The terminations of processes These records are produced from the output of system programs such as date, init, login, ttymon, and acctwtmp. They are stored in the /var/adm/wtmpx file. Entries in the wtmpx file can contain the following information: Login name Device name Process ID Entry type Time stamp that denotes when the entry was made Process accounting enables you to keep track of the following data about each process that runs on your system:User IDs and group IDs of users using the process Beginning times and elapsed times of the process CPU time for the process (user time and system time) Amount of memory used by the process Commands run by the process The tty that controls the process Every time a process terminates, the exit program collects this information and writes it to the /var/adm/pacct file. Disk accounting enables you to gather and format the following data about the files each user has on disks:User name and user ID of the user Number of blocks that are used by the user's files This data is collected by the /usr/lib/acct/dodisk shell script at intervals that are determined by the entry you add to the /var/spool/cron/crontabs/root file. In turn, the dodisk script invokes the acctdisk and acctdusg commands. These commands gather disk usage by login name.Information gathered by running the dodisk script is stored in the /var/adm/acct/nite/disktacct file. This information is overwritten the next time the dodisk script is run. Therefore, avoid running the dodisk script twice in the same day. The acctdusg command might overcharge for files that are written randomly, which can create holes in the files. This problem occurs because the acctdusg command does not read the indirect blocks of a file when determining the file size. Rather, the acctdusg command determines the file size by checking the current file size value in the file's inode. The chargefee utility stores charges for special services that are provided to a user in the /var/adm/fee file. A special service, for example, is file restoration. Each entry in the file consists of a user login name, user ID, and the fee. This file is checked by the runacct script every day, and new entries are merged into the accounting records. For instructions on running the chargefee script to bill users, see How to Bill Users. Here is a step-by-step summary of how daily accounting works: When the system is switched into multiuser mode, the /usr/lib/acct/startup program is executed. The startup program executes several other programs that invoke daily accounting. The acctwtmp program adds a “boot” record to the /var/adm/wtmpx file. In this record, the system name is shown as the user name in the wtmpx record. The following table summarizes how the raw accounting data is gathered and where it is stored. File in /var/adm Information Stored Written By Format wtmpx Connect sessions login, init Binary Changes date Binary Reboots acctwtmp Binary Shutdowns shutacct Binary pacctn Processes Kernel (when the process ends) Binary turnacct switch (which creates a new file when the old file reaches 500 blocks) Binary fee Special charges chargefee ASCII acct/nite/disktacct Disk space used dodisk Binary The turnacct script, invoked with the on option, begins process accounting. Specifically, the turnacct script executes the accton program with the /var/adm/pacct argument. The remove shell script “cleans up” the saved pacct and wtmpx files that are left in the sum directory by the runacct script. The login and init programs record connect sessions by writing records into the /var/adm/wtmpx file. Date changes (using date with an argument) are also written to the /var/adm/wtmpx file. Reboots and shutdowns using the acctwtmp command are also recorded in the /var/adm/wtmpx file. When a process ends, the kernel writes one record per process, using the acct.h format, in the /var/adm/pacct file.Every hour, the cron command executes the ckpacct script to check the size of the /var/adm/pacct file. If the file grows beyond 500 blocks (default), the turnacct switch command is executed. (The program moves the pacct file to the pacctn file and creates a new file.) The advantage of having several smaller pacct files becomes apparent when you try to restart the runacct script if a failure occurs when processing these records. The runacct script is executed by the cron command each night. The runacct script processes the accounting files to produce command summaries and usage summaries by user name. These accounting files are processed: /var/adm/pacctn, /var/adm/wtmpx, /var/adm/fee, and /var/adm/acct/nite/disktacct. The /usr/lib/acct/prdaily script is executed on a daily basis by the runacct script to write the daily accounting information in the /var/adm/acct/sum/rprtMMDD files. The monacct script should be executed on a monthly basis (or at intervals you determine, such as at the end of every fiscal period). The monacct script creates a report that is based on data stored in the sum directory that has been updated daily by the runacct script. After creating the report, the monacct script “cleans up” the sum directory to prepare the directory's files for the new runacct data. If the system is shut down by using the shutdown command, the shutacct script is executed automatically. The shutacct script writes a reason record into the /var/adm/wtmpx file and turns off process accounting. Task Description For Instructions Set up system accounting. Set up system accounting by performing the following tasks:Create the /etc/rc0.d/K22acct and /etc/rc2.d/S22acct files. Modify the /var/spool/cron/crontabs/adm and /var/spool/cron/crontabs/root crontab files. How to Set Up System Accounting Bill users. Run the /usr/lib/acct/chargefee username amount command. How to Bill Users Fix a corrupted wtmpx file. Convert the wtmpx file from binary to ASCII format. How to Fix a Corrupted wtmpx File Fix tacct errors. Run the prtacct script to check the /var/adm/acct/sum/tacctprev file. Then, patch the latest/var/adm/acct/sum/tacctMMDD file. You will need to re-create the /var/adm/acct/sum/tacct file. How to Fix tacct Errors Restart the runacct script. Remove the lastdate file and any lock files. Then, manually restart the runacct script. How to Restart the runacct Script Disable system accounting temporarily. Edit theadm crontab file to stop the ckpacct, runacct, and monacct programs from running. How to Temporarily Stop System Accounting Disable system accounting permanently. Delete the entries for the ckpacct, runacct, and monacct programs in the adm and crontab files. How to Permanently Disable System Accounting You can set up system accounting to run while the system is in multiuser mode (Run Level 2). Generally, this task involves these steps:Creating the /etc/rc0.d/K22acct and /etc/rc2.d/S22acct startup scripts Modifying the /var/spool/cron/crontabs/adm and /var/spool/cron/crontabs/root crontab files The following table describes the default accounting scripts. Purpose Accounting Script Man Page Run Frequency Checks the size of the /usr/adm/pacct log file and makes sure that it does not get too large. ckpacct acctsh1M Periodically Processes connect, disk, and fee accounting information. You can remove from this script the commands for the accounting features you do not want processed. runacct runacct1M Daily Generates fiscal accounting summary reports on a monthly basis. You can determine how often this script is run. You can remove from this script the commands for the accounting features you do not want to use. monacct acctsh1M On a fiscal basis You can choose which accounting scripts run by default. After these entries have been added to the crontab files, system accounting should run automatically. &rolestep.sgm;If necessary, install the SUNWaccr and SUNWaccu packages on your system by using the pkgadd command. Install /etc/init.d/acct as the startup script for Run Level 2. # ln /etc/init.d/acct /etc/rc2.d/S22acct Install /etc/init.d/acct as the stop script for Run Level 0.# ln /etc/init.d/acct /etc/rc0.d/K22acct Add the following lines to the adm crontab file to start the ckpacct, runacct, and monacct scripts automatically.# EDITOR=vi; export EDITOR # crontab -e adm 0 * * * * /usr/lib/acct/ckpacct 30 2 * * * /usr/lib/acct/runacct 2> /var/adm/acct/nite/fd2log 30 7 1 * * /usr/lib/acct/monacct Add the following line to the root crontab file to start the dodisk script automatically.# crontab -e 30 22 * * 4 /usr/lib/acct/dodisk Edit /etc/acct/holidays to include national holidays and local holidays.For more information, see the holidays4 man page and the example that follows. Reboot the system, or start system accounting manually by typing:# /etc/init.d/acct start This modified adm crontab contains entries for the ckpacct, runacct, and monacct scripts. #ident "@(#)adm 1.5 92/07/14 SMI" /* SVr4.0 1.2 */ # # The adm crontab file should contain startup of performance # collection if the profiling and performance feature has been # installed. 0 * * * * /usr/lib/acct/ckpacct 30 2 * * * /usr/lib/acct/runacct 2> /var/adm/acct/nite/fd2log 30 7 1 * * /usr/lib/acct/monacct This modified root crontab contains entries for the dodisk program.#ident "@(#)root 1.19 98/07/06 SMI" /* SVr4.0 1.1.3.1 */ # # The root crontab should be used to perform accounting data collection. # # 10 3 * * * /usr/sbin/logadm 15 3 * * 0 /usr/lib/fs/nfs/nfsfind 30 3 * * * [ -x /usr/lib/gss/gsscred_clean ] && /usr/lib/gss/gsscred_clean 30 22 * * 4 /usr/lib/acct/dodisk The following example shows a sample /etc/acct/holidays file. * @(#)holidays January 1, 2004 * * Prime/Nonprime Table for UNIX Accounting System * * Curr Prime Non-Prime * Year Start Start * 2004 0800 1800 * * only the first column (month/day) is significant. * * month/day Company * Holiday * 1/1 New Years Day 7/4 Indep. Day 12/25 Christmas If you provide special user services by request. Special services include restoring files or remote printing. You might want to bill users by running the chargefee utility. The chargefee utility records charges in the /var/adm/fee file. Each time the runacct utility is executed, new entries are merged into the total accounting records. See the acctsh1M man page for more information. &rolestep.sgm;Charge a user for special services. # /usr/lib/acct/chargefee username amountusernameIs the user account you want to bill. amountSpecifies the number of units to bill the user. This value is an arbitrary unit that you set to charge users based on some task such as printing or restoring a file. You would have to write a script that invokes the chargefee utility and charges a user for a specific task. In the following example, the user print_customer is charged 10 units.# /usr/lib/acct/chargefee print_customer 10 This section describes how to fix corrupted system accounting files and how to restart the runacct script.Unfortunately, system accounting is not foolproof. Occasionally, a file becomes corrupted or lost. Some files can simply be ignored or restored from backup. However, certain files must be fixed to maintain the integrity of system accounting. The wtmpx files seem to cause the most problems in the daily operation of system accounting. When the date is changed manually and the system is in multiuser mode, a set of date change records is written to the /var/adm/wtmpx file. The wtmpfix utility is designed to adjust the time stamps in the wtmp records when a date change is encountered. However, some combinations of date changes and reboots slip through the wtmpfix utility and cause the acctcon program to fail. &rolestep.sgm;Change to the /var/adm directory. Convert the wtmpx file from binary format to ASCII format.# /usr/lib/acct/fwtmp < wtmpx > wtmpx.ascii Edit the wtmpx.ascii file to delete the corrupted records. Convert the wtmpx.ascii file back to a binary file. # /usr/lib/acct/fwtmp -ic < wtmpx.ascii > wtmpxSee the fwtmp1M man page for more information. The integrity of the /var/adm/acct/sum/tacct file is important if you are charging users for system resources. Occasionally, unusual tacct records appear with negative numbers, duplicate user IDs, or a user ID of 65535. First, check the /var/adm/acct/sum/tacctprev file by using the prtacct script to print the file. If the contents look all right, patch the latest /var/adm/acct/sum/tacctMMDD file. Then, re-create the /var/adm/acct/sum/tacct file. The following steps outline a simple patch procedure. &rolestep.sgm;Change to the /var/adm/acct/sum directory. Convert the tacctMMDD file from binary format to ASCII format.# /usr/lib/acct/acctmerg -v < tacctMMDD > xtacctMMDD is pair of two-digit numbers that represent the month and day. Edit the xtacct file, removing corrupted records and writing duplicate records to another file. Convert the xtacct file from ASCII format to binary format. # /usr/lib/acct/acctmerg -i < xtacct > tacctMMDD Merge the files tacctprev and tacct.MMDD into the tacct file. # /usr/lib/acct/acctmerg < tacctprev tacctMMDD > tacct The runacct script can fail for several reasons. The following are the most common reasons: A system crash The /var directory is running out of space A corrupted wtmpx file If the active.MMDD file exists, check it first for error messages. If the active and lock files exist, check the fd2log file for any relevant messages.Run without arguments, the runacct script assumes that this invocation is the first invocation of the day. The argument MMDD is necessary if the runacct script is being restarted and specifies the month and day for which the runacct script reruns the accounting. The entry point for processing is based on the contents of the statefile file. To override the statefile file, include the desired state on the command line. For a description of the available states, see the runacct1M man page. When you run the runacct program manually, be sure to run it as user adm. Change directories to the /var/adm/acct/nite directory.$ cd /var/adm/acct/nite Remove the lastdate file and any lock* files, if any.$ rm lastdate lock*The lastdate file contains the date that the runacct program was last run. Restarting the runacct script in the next step re-creates this file. Restart the runacct script.$ /usr/lib/acct/runacct MMDD [state] 2> /var/adm/acct/nite/fd2log &MMDDIs the month and day specified by two-digit numbers. stateSpecifies a state, or starting point, where the runacct script processing should begin. You can temporarily stop system accounting or permanently disable it.&rolestep.sgm;Edit the adm crontab file to stop the ckpacct, runacct, and monacct programs from running by commenting out the appropriate lines.# EDITOR=vi; export EDITOR # crontab -e adm #0 * * * * /usr/lib/acct/ckpacct #30 2 * * * /usr/lib/acct/runacct 2> /var/adm/acct/nite/fd2log #30 7 1 * * /usr/lib/acct/monacct Edit the root crontab file to stop the dodisk program from running by commenting out the appropriate line.# crontab -e #30 22 * * 4 /usr/lib/acct/dodisk Stop the system accounting program.# /etc/init.d/acct stop (Optional) Remove the newly added comment symbols from the crontab files. Restart the system accounting program to re-enable system accounting.# /etc/init.d/acct start &rolestep.sgm;Edit the adm crontab file and delete the entries for the ckpacct, runacct, and monacct programs.# EDITOR=vi; export EDITOR # crontab -e adm Edit the root crontab file and delete the entries for the dodisk program.# crontab -e Remove the startup script for Run Level 2. # unlink /etc/rc2.d/S22acct Remove the stop script for Run Level 0.# unlink /etc/rc0.d/K22acct Stop the system accounting program.# /etc/init.d/acct stop