This chapter describes procedures for monitoring system performance by using the vmstat, iostat, df, and sar commands.For information on the procedures that are associated with monitoring system performance, see the following:Displaying System Performance Information (Task Map) Monitoring System Activities (Task Map) Task Description For Instructions Display virtual memory Statistics. Collect virtual memory statistics by using the vmstat command. How to Display Virtual Memory Statistics (vmstat) Display system event information. Display system event information by using the vmstat command with the s option How to Display System Event Information (vmstat -s) Display swapping statistics. Use the vmstat command with the S option to display swapping statistics. How to Display Swapping Statistics (vmstat -S) Display interrupts per device. Use the vmstat command with the i option to show the number of interrupts per device. How to Display Interrupts Per Device (vmstat -i) Display disk utilization. Use the iostat command to report disk input and output statistics. How to Display Disk Utilization Information (iostat) Display extended disk statistics. Use the iostat command with the xtcoption to display extended disk statistics. How to Display Extended Disk Statistics (iostat -xtc) Display disk space information. The df k command displays disk space information in Kbytes. How to Display Disk Space Information (df -k) You can use the vmstat command to report virtual memory statistics and information about system events such as CPU load, paging, number of context switches, device interrupts, and system calls. The vmstat command can also display statistics on swapping, cache flushing, and interrupts.The following table describes the fields in the vmstat command output. Category Field Name Description procs Reports on the following: r The number of kernel threads in the dispatch queue b The number of blocked kernel threads that are waiting for resources w The number of swapped out LWPs that are waiting for processing resources to finish memory Reports on usage of real memory and virtual memory: swap Available swap space free Size of the free list page Reports on page faults and paging activity, in units per second: re Pages reclaimed mf Minor faults and major faults pi Kbytes paged in po Kbytes paged out fr Kbytes freed de Anticipated memory that is needed by recently swapped-in processes sr Pages scanned by the page daemon not currently in use. If sr does not equal zero, the page daemon has been running. disk Reports the number of disk operations per second, showing data on up to four disks faults Reports the trap/interrupt rates per second: in Interrupts per second sy System calls per second cs CPU context switch rate cpu Reports on the use of CPU time: us User time sy System time id Idle time For a more detailed description of this command, see the vmstat1M man page.Collect virtual memory statistics by using the vmstat command with a time interval in seconds.$ vmstat nwhere n is the interval in seconds between reports. The following example shows the vmstat display of statistics that were gathered at five-second intervals.$ vmstat 5 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr dd f0 s1 -- in sy cs us sy id 0 0 0 863160 365680 0 3 1 0 0 0 0 0 0 0 0 406 378 209 1 0 99 0 0 0 765640 208568 0 36 0 0 0 0 0 0 0 0 0 479 4445 1378 3 3 94 0 0 0 765640 208568 0 0 0 0 0 0 0 0 0 0 0 423 214 235 0 0 100 0 0 0 765712 208640 0 0 0 0 0 0 0 3 0 0 0 412 158 181 0 0 100 0 0 0 765832 208760 0 0 0 0 0 0 0 0 0 0 0 402 157 179 0 0 100 0 0 0 765832 208760 0 0 0 0 0 0 0 0 0 0 0 403 153 182 0 0 100 0 0 0 765832 208760 0 0 0 0 0 0 0 0 0 0 0 402 168 177 0 0 100 0 0 0 765832 208760 0 0 0 0 0 0 0 0 0 0 0 402 153 178 0 0 100 0 0 0 765832 208760 0 18 0 0 0 0 0 0 0 0 0 407 165 186 0 0 100 Run the vmstat -s command to show how many system events have taken place since the last time the system was booted. $ vmstat -s 0 swap ins 0 swap outs 0 pages swapped in 0 pages swapped out 522586 total address trans. faults taken 17006 page ins 25 page outs 23361 pages paged in 28 pages paged out 45594 total reclaims 45592 reclaims from free list 0 micro (hat) faults 522586 minor (as) faults 16189 major faults 98241 copy-on-write faults 137280 zero fill page faults 45052 pages examined by the clock daemon 0 revolutions of the clock hand 26 pages freed by the clock daemon 2857 forks 78 vforks 1647 execs 34673885 cpu context switches 65943468 device interrupts 711250 traps 63957605 system calls 3523925 total name lookups (cache hits 99%) 92590 user cpu 65952 system cpu 16085832 idle cpu 7450 wait cpu Run vmstat -S to show swapping statistics.$ vmstat -S kthr memory page disk faults cpu r b w swap free si so pi po fr de sr dd f0 s1 -- in sy cs us sy id 0 0 0 862608 364792 0 0 1 0 0 0 0 0 0 0 0 406 394 213 1 0 99The swapping statistics fields are described in the following list. For a description of the other fields, see Table 13–1.siAverage number of LWPs that are swapped in per second soNumber of whole processes that are swapped out The vmstat command truncates the output of si and so fields. Use the sar command to display a more accurate accounting of swap statistics. Run the vmstat -i command to show the number of interrupts per device. The following example shows output from the vmstat -i command. $ vmstat -i interrupt total rate -------------------------------- clock 52163269 100 esp0 2600077 4 zsc0 25341 0 zsc1 48917 0 cgsixc0 459 0 lec0 400882 0 fdc0 14 0 bppc0 0 0 audiocs0 0 0 -------------------------------- Total 55238959 105 Use the iostat command to report statistics about disk input and output, and to produce measures of throughput, utilization, queue lengths, transaction rates, and service time. For a detailed description of this command, refer to the iostat1M man page.You can display disk utilization information by using the iostat command with a time interval in seconds.$ iostat 5 tty fd0 sd3 nfs1 nfs31 cpu tin tout kps tps serv kps tps serv kps tps serv kps tps serv us sy wt id 0 1 0 0 410 3 0 29 0 0 9 3 0 47 4 2 0 94The first line of output shows the statistics since the last time the system was booted. Each subsequent line shows the interval statistics. The default is to show statistics for the terminal (tty), disks (fd and sd), and CPU (cpu). The following example shows disk statistics that were gathered every five seconds. $ iostat 5 tty sd0 sd6 nfs1 nfs49 cpu tin tout kps tps serv kps tps serv kps tps serv kps tps serv us sy wt id 0 0 1 0 49 0 0 0 0 0 0 0 0 15 0 0 0 100 0 47 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 44 6 132 0 0 0 0 0 0 0 0 0 0 0 1 99 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 3 1 23 0 0 0 0 0 0 0 0 0 0 0 1 99 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100The following table describes the fields in the output of the iostat n command.Device Type Field Name Description Terminal Device Type tin Number of characters in the terminal input queue tout Number of characters in the terminal output queue Disk Device Type bps Blocks per second tps Transactions per second serv Average service time, in milliseconds CPU Device Type us In user mode sy In system mode wt Waiting for I/O id Idle Run the iostat -xtc command to display extended disk statistics. $ iostat -xtc extended device statistics tty cpu device r/s w/s kr/s kw/s wait actv svc_t %w %b tin tout us sy wt id fd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 0 100 sd0 0.0 0.0 0.4 0.4 0.0 0.0 49.5 0 0 sd6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 nfs1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 nfs49 0.0 0.0 0.0 0.0 0.0 0.0 15.1 0 0 nfs53 0.0 0.0 0.4 0.0 0.0 0.0 24.5 0 0 nfs54 0.0 0.0 0.0 0.0 0.0 0.0 6.3 0 0 nfs55 0.0 0.0 0.0 0.0 0.0 0.0 4.9 0 0 The iostat -xtc command displays a line of output for each disk. The output fields are described in the following list.r/sReads per second w/sWrites per second kr/sKbytes read per second kw/sKbytes written per second waitAverage number of transactions that are waiting for service (queue length) actvAverage number of transactions that are actively being serviced svc_tAverage service time, in milliseconds %wPercentage of time that the queue is not empty %bPercentage of time that the disk is busy Use the df command to show the amount of free disk space on each mounted disk. The usable disk space that is reported by df reflects only 90 percent of full capacity, as the reporting statistics allows for 10 percent above the total available space. This head room normally stays empty for better performance. The percentage of disk space actually reported by the df command is used space divided by usable space. If the file system exceeds 90 percent capacity, you could transfer files to a disk that is not as full by using the cp command. Alternately, you could transfer files to a tape by using the tar or cpio commands. Or, you could remove the files.For a detailed description of this command, see the df1M man page.Use the df -k command to display disk space information in Kbytes.$ df -k Filesystem kbytes used avail capacity Mounted on /dev/dsk/c0t3d0s0 192807 40231 133296 24% / The following example shows the output from the df -k command.$ df -k Filesystem kbytes used avail capacity Mounted on /dev/dsk/c0t0d0s0 254966 204319 25151 90% / /devices 0 0 0 0% /devices ctfs 0 0 0 0% /system/contract proc 0 0 0 0% /proc mnttab 0 0 0 0% /etc/mnttab swap 496808 376 496432 1% /etc/svc/volatile objfs 0 0 0 0% /system/object /dev/dsk/c0t0d0s6 3325302 3073415 218634 94% /usr fd 0 0 0 0% /dev/fd swap 496472 40 496432 1% /var/run swap 496472 40 496432 1% /tmp /dev/dsk/c0t0d0s5 13702 1745 10587 15% /opt /dev/dsk/c0t0d0s7 9450 1045 7460 13% /export/homeThe following table describes the output of the df -k command.Field Name Description kbytes Total size of usable space in the file system used Amount of space used avail Amount of space available for use capacity Amount of space used, as a percentage of the total capacity mounted on Mount point Task Description For Instructions Check file access. Display file access operation status by using the sar command with the a option. How to Check File Access (sar -a) Check buffer activity. Display buffer activity statistics by using the sar command with the b option. How to Check Buffer Activity (sar -b) Check system call statistics. Display system call statistics by using the sar command with the c option. How to Check System Call Statistics (sar -c) Check disk activity. Check disk activity by using the sar command with the d option. How to Check Disk Activity (sar -d) Check page-out and memory. Use the sar command with the g option to display page-out memory freeing activities. How to Check Page-Out and Memory (sar -g) Check kernel memory allocation. The kernel memory allocation (KMA) allows a kernel subsystem to allocate and free memory, as needed. Use the sar command with the k option to check KMA. How to Check Kernel Memory Allocation (sar -k) Check interprocess communication. Use the sar command with the m option to report interprocess communication activities. How to Check Interprocess Communication (sar -m) Check page-in activity. Use the sar command with the p option to report page-in activity. How to Check Page-In Activity (sar -p) Check queue activity. Use the sar command with the q option to check the following:Average queue length while queue is occupied Percentage of time that the queue is occupied How to Check Queue Activity (sar -q) Check unused memory. Use the sar command with the r option to report the number of memory pages and swap file disk blocks that are currently used. How to Check Unused Memory (sar -r) Check CPU utilization. Use the sar command with the u option to display CPU utilization statistics. How to Check CPU Utilization (sar -u) Check system table status. Use the sar command with the v option to report status on the following system tables:Process Inode File Shared memory record How to Check System Table Status (sar -v) Check swapping activity. Use the sar command with the w option to check swapping activity. How to Check Swapping Activity (sar -w) Check terminal activity. Use the sar command with the y option to monitor terminal device activity. How to Check Terminal Activity (sar -y) Check overall system performance. The sar A command displays statistics from all options to provide overall system performance information. How to Check Overall System Performance (sar -A) Set up automatic data collection. To set up your system to collect data automatically and to run the sar commands, do the following:Run the svcadm enable system/sar:default command Edit the /var/spool/cron/crontabs/sys file How to Set Up Automatic Data Collection Use the sar command to perform the following tasks:Organize and view data about system activity. Access system activity data on a special request basis. Generate automatic reports to measure and monitor system performance, as well as special request reports to pinpoint specific performance problems. For information on how to set up the sar command to run on your system, as well as a description of these tools, see Collecting System Activity Data Automatically (sar). For a detailed description of this command, see the sar1 man page.Display file access operation statistics with the sar -a command.$ sar -a SunOS balmyday 5.10 s10_51 sun4u 03/18/2004 00:00:00 iget/s namei/s dirbk/s 01:00:00 0 3 0 02:00:00 0 3 0 03:00:00 0 3 0 04:00:00 0 3 0 05:00:00 0 3 0 06:00:00 0 3 0 07:00:00 0 3 0 08:00:00 0 3 0 08:20:01 0 3 0 08:40:00 0 3 0 09:00:00 0 3 0 09:20:01 0 10 0 09:40:01 0 1 0 10:00:02 0 5 0 Average 0 4 0The following list describes the field names and description of operating system routines that are reported by the sar -a command.iget/sThe number of requests made for inodes that were not in the directory name look-up cache (DNLC). namei/sThe number of file system path searches per second. If namei does not find a directory name in the DNLC, it calls iget to get the inode for either a file or directory. Hence, most igets are the result of DNLC misses. dirbk/sThe number of directory block reads issued per second. The larger the reported values for these operating system routines, the more time the kernel is spending to access user files. The amount of time reflects how heavily programs and applications are using the file systems. The a option is helpful for viewing how disk-dependent an application is. Display buffer activity statistics with the sar -b command. The buffer is used to cache metadata. Metadata includes inodes, cylinder group blocks, and indirect blocks. $ sar -b 00:00:00 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s 01:00:00 0 0 100 0 0 55 0 0 The following example of sar -b command output shows that the %rcache and %wcache buffers are not causing any slowdowns. All the data is within acceptable limits.$ sar -b SunOS balmyday 5.10 s10_51 sun4u 03/18/2004 00:00:04 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s 01:00:00 0 0 100 0 0 94 0 0 02:00:01 0 0 100 0 0 94 0 0 03:00:00 0 0 100 0 0 92 0 0 04:00:00 0 1 100 0 1 94 0 0 05:00:00 0 0 100 0 0 93 0 0 06:00:00 0 0 100 0 0 93 0 0 07:00:00 0 0 100 0 0 93 0 0 08:00:00 0 0 100 0 0 93 0 0 08:20:00 0 1 100 0 1 94 0 0 08:40:01 0 1 100 0 1 93 0 0 09:00:00 0 1 100 0 1 93 0 0 09:20:00 0 1 100 0 1 93 0 0 09:40:00 0 2 100 0 1 89 0 0 10:00:00 0 9 100 0 5 92 0 0 10:20:00 0 0 100 0 0 68 0 0 10:40:00 0 1 98 0 1 70 0 0 11:00:00 0 1 100 0 1 75 0 0 Average 0 1 100 0 1 91 0 0The following table describes the buffer activities that are displayed by the b option.Field Name Description bread/s Average number of reads per second that are submitted to the buffer cache from the disk lread/s Average number of logical reads per second from the buffer cache %rcache Fraction of logical reads that are found in the buffer cache (100 % minus the ratio of bread/s to lread/s) bwrit/s Average number of physical blocks (512 blocks) that are written from the buffer cache to disk, per second lwrit/s Average number of logical writes to the buffer cache, per second %wcache Fraction of logical writes that are found in the buffer cache (100 % minus the ratio of bwrit/s to lwrit/s) pread/s Average number of physical reads, per second, that use character device interfaces pwrit/s Average number of physical write requests, per second, that use character device interfaces The most important entries are the cache hit ratios %rcache and %wcache. These entries measure the effectiveness of system buffering. If %rcache falls below 90 percent, or if %wcache falls below 65 percent, it might be possible to improve performance by increasing the buffer space. Display system call statistics by using the sar -c command.$ sar -c 00:00:00 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s 01:00:00 38 2 2 0.00 0.00 149 120 The following example shows output from the sar -c command.$ sar -c SunOS balmyday 5.10 s10_51 sun4u 03/18/2004 00:00:04 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s 01:00:00 89 14 9 0.01 0.00 2906 2394 02:00:01 89 14 9 0.01 0.00 2905 2393 03:00:00 89 14 9 0.01 0.00 2908 2393 04:00:00 90 14 9 0.01 0.00 2912 2393 05:00:00 89 14 9 0.01 0.00 2905 2393 06:00:00 89 14 9 0.01 0.00 2905 2393 07:00:00 89 14 9 0.01 0.00 2905 2393 08:00:00 89 14 9 0.01 0.00 2906 2393 08:20:00 90 14 9 0.01 0.01 2914 2395 08:40:01 90 14 9 0.01 0.00 2914 2396 09:00:00 90 14 9 0.01 0.01 2915 2396 09:20:00 90 14 9 0.01 0.01 2915 2396 09:40:00 880 207 156 0.08 0.08 26671 9290 10:00:00 2020 530 322 0.14 0.13 57675 36393 10:20:00 853 129 75 0.02 0.01 10500 8594 10:40:00 2061 524 450 0.08 0.08 579217 567072 11:00:00 1658 404 350 0.07 0.06 1152916 1144203 Average 302 66 49 0.02 0.01 57842 55544The following table describes the system call categories that are reported by the c option. Typically, reads and writes account for about half of the total system calls. However, the percentage varies greatly with the activities that are being performed by the system.Field Name Description scall/s The number of all types of system calls per second, which is generally about 30 per second on a system with 4 to 6 users. sread/s The number of read system calls per second. swrit/s The number of write system calls per second. fork/s The number of fork system calls per second, which is about 0.5 per second on a system with 4 to 6 users. This number increases if shell scripts are running. exec/s The number of exec system calls per second. If exec/s divided by fork/s is greater than 3, look for inefficient PATH variables. rchar/s The number of characters (bytes) transferred by read system calls per second. wchar/s The number of characters (bytes) transferred by write system calls per second. Display disk activity statistics with the sar -d command.$ sar -d 00:00:00 device %busy avque r+w/s blks/s avwait avserv This abbreviated example illustrates the output from the sar -d command.$ sar -d SunOS balmyday 5.10 s10_51 sun4u 03/18/2004 12:36:32 device %busy avque r+w/s blks/s avwait avserv 12:40:01 dad1 15 0.7 26 399 18.1 10.0 dad1,a 15 0.7 26 398 18.1 10.0 dad1,b 0 0.0 0 1 1.0 3.0 dad1,c 0 0.0 0 0 0.0 0.0 dad1,h 0 0.0 0 0 0.0 6.0 fd0 0 0.0 0 0 0.0 0.0 nfs1 0 0.0 0 0 0.0 0.0 nfs2 1 0.0 1 12 0.0 13.2 nfs3 0 0.0 0 2 0.0 1.9 nfs4 0 0.0 0 0 0.0 7.0 nfs5 0 0.0 0 0 0.0 57.1 nfs6 1 0.0 6 125 4.3 3.2 nfs7 0 0.0 0 0 0.0 6.0 sd1 0 0.0 0 0 0.0 5.4 ohci0,bu 0 0.0 0 0 0.0 0.0 ohci0,ct 0 0.0 0 0 0.0 0.0 ohci0,in 0 0.0 7 0 0.0 0.0 ohci0,is 0 0.0 0 0 0.0 0.0 ohci0,to 0 0.0 7 0 0.0 0.0The following table describes the disk device activities that are reported by the d option.Field Name Description device Name of the disk device that is being monitored. %busy Portion of time the device was busy servicing a transfer request. avque Average number of requests during the time the device was busy servicing a transfer request. r+w/s Number of read-and-write transfers to the device, per second. blks/s Number of 512-byte blocks that are transferred to the device, per second. avwait Average time, in milliseconds, that transfer requests wait idly in the queue. This time is measured only when the queue is occupied. avserv Average time, in milliseconds, for a transfer request to be completed by the device. For disks, this value includes seek times, rotational latency times, and data transfer times. Note that queue lengths and wait times are measured when something is in the queue. If %busy is small, large queues and service times probably represent the periodic efforts by the system to ensure that altered blocks are promptly written to the disk. Use the sar -g command to display page-out and memory freeing activities in averages. $ sar -g 00:00:00 pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf 01:00:00 0.00 0.00 0.00 0.00 0.00The output displayed by the sar -g command is a good indicator of whether more memory might be needed. Use the ps -elf command to show the number of cycles that are used by the page daemon. A high number of cycles, combined with high values for the pgfree/s and pgscan/s fields, indicates a memory shortage. The sar -g command also shows whether inodes are being recycled too quickly and causing a loss of reusable pages. The following example shows output from the sar -g command.$ sar -g SunOS balmyday 5.10 s10_51 sun4u 03/18/2004 00:00:00 pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf 01:00:00 0.00 0.00 0.00 0.00 0.00 02:00:00 0.01 0.01 0.01 0.00 0.00 03:00:00 0.00 0.00 0.00 0.00 0.00 04:00:00 0.00 0.00 0.00 0.00 0.00 05:00:00 0.00 0.00 0.00 0.00 0.00 06:00:00 0.00 0.00 0.00 0.00 0.00 07:00:00 0.00 0.00 0.00 0.00 0.00 08:00:00 0.00 0.00 0.00 0.00 0.00 08:20:01 0.00 0.00 0.00 0.00 0.00 08:40:00 0.00 0.00 0.00 0.00 0.00 09:00:00 0.00 0.00 0.00 0.00 0.00 09:20:01 0.05 0.52 1.62 10.16 0.00 09:40:01 0.03 0.44 1.47 4.77 0.00 10:00:02 0.13 2.00 4.38 12.28 0.00 10:20:03 0.37 4.68 12.26 33.80 0.00 Average 0.02 0.25 0.64 1.97 0.00The following table describes the output from the g option.Field Name Description pgout/s The number of page-out requests per second. ppgout/s The actual number of pages that are paged-out, per second. A single page-out request might involve paging-out multiple pages. pgfree/s The number of pages, per second, that are placed on the free list. pgscan/s The number of pages, per second, that are scanned by the page daemon. If this value is high, the page daemon is spending a lot of time checking for free memory. This situation implies that more memory might be needed. %ufs_ipf The percentage of ufs inodes taken off the free list by iget that had reusable pages associated with them. These pages are flushed and cannot be reclaimed by processes. Thus, this field represents the percentage of igets with page flushes. A high value indicates that the free list of inodes is page-bound, and that the number of ufs inodes might need to be increased. The KMA allows a kernel subsystem to allocate and free memory, as needed. Rather than statically allocating the maximum amount of memory it is expected to require under peak load, the KMA divides requests for memory into three categories:Small (less than 256 bytes) Large (512 bytes to 4 Kbytes) Oversized (greater than 4 Kbytes) The KMA keeps two pools of memory to satisfy small requests and large requests. The oversized requests are satisfied by allocating memory from the system page allocator. If you are checking a system that is being used to write drivers or STREAMS that use KMA resources, then the sar -k command will likely prove useful. Otherwise, you will probably not need the information it provides. Any driver or module that uses KMA resources, but does not specifically return the resources before it exits, can create a memory leak. A memory leak causes the amount of memory that is allocated by KMA to increase over time. Thus, if the alloc fields of the sar -k command increase steadily over time, there might be a memory leak. Another indication of a memory leak is failed requests. If this problem occurs, a memory leak has probably caused KMA to be unable to reserve and allocate memory. If it appears that a memory leak has occurred, you should check any drivers or STREAMS that might have requested memory from KMA and not returned it. Use the sar -k command to report on the following activities of the Kernel Memory Allocator (KMA). $ sar -k 00:00:00 sml_mem alloc fail lg_mem alloc fail ovsz_alloc fail 01:00:00 2523136 1866512 0 18939904 14762364 0 360448 0 02:00:02 2523136 1861724 0 18939904 14778748 0 360448 0 The following is an abbreviated example of sar -k output. $ sar -k SunOS balmyday 5.10 s10_51 sun4u 03/18/2004 00:00:04 sml_mem alloc fail lg_mem alloc fail ovsz_alloc fail 01:00:00 6119744 4852865 0 60243968 54334808 156 9666560 0 02:00:01 6119744 4853057 0 60243968 54336088 156 9666560 0 03:00:00 6119744 4853297 0 60243968 54335760 156 9666560 0 04:00:00 6119744 4857673 0 60252160 54375280 156 9666560 0 05:00:00 6119744 4858097 0 60252160 54376240 156 9666560 0 06:00:00 6119744 4858289 0 60252160 54375608 156 9666560 0 07:00:00 6119744 4858793 0 60252160 54442424 156 9666560 0 08:00:00 6119744 4858985 0 60252160 54474552 156 9666560 0 08:20:00 6119744 4858169 0 60252160 54377400 156 9666560 0 08:40:01 6119744 4857345 0 60252160 54376880 156 9666560 0 09:00:00 6119744 4859433 0 60252160 54539752 156 9666560 0 09:20:00 6119744 4858633 0 60252160 54410920 156 9666560 0 09:40:00 6127936 5262064 0 60530688 55619816 156 9666560 0 10:00:00 6545728 5823137 0 62996480 58391136 156 9666560 0 10:20:00 6545728 5758997 0 62996480 57907400 156 9666560 0 10:40:00 6734144 6035759 0 64389120 59743064 156 10493952 0 11:00:00 6996288 6394872 0 65437696 60935936 156 10493952 0 Average 6258044 5150556 0 61138340 55609004 156 9763900 0The following table describes the output from the k option.Field Name Description sml_mem The amount of memory, in bytes, that the KMA has available in the small memory request pool. In this pool, here a small request is less than 256 bytes. alloc The amount of memory, in bytes, that the KMA has allocated from its small memory request pool to small memory requests. fail The number of requests for small amounts of memory that failed. lg_mem The amount of memory, in bytes, that the KMA has available in the large memory request pool. In this pool, a large request is from 512 bytes to 4 Kbytes. alloc The amount of memory, in bytes, that the KMA has allocated from its large memory request pool to large memory requests. fail The number of failed requests for large amounts of memory. ovsz_alloc The amount of memory that is allocated for oversized requests, which are requests that are greater than 4 Kbytes. These requests are satisfied by the page allocator. Thus, there is no pool. fail The number of failed requests for oversized amounts of memory. Use the sar -m command to report interprocess communication activities. $ sar -m 00:00:00 msg/s sema/s 01:00:00 0.00 0.00These figures are usually zero (0.00), unless you are running applications that use messages or semaphores. The following list describes the output from the m option.msg/sThe number of message operations (sends and receives) per second sema/sThe number of semaphore operations per second The following abbreviated example shows output from the sar -m command.$ sar -m SunOS balmyday 5.10 s10_51 sun4u 03/18/2004 00:00:00 msg/s sema/s 01:00:00 0.00 0.00 02:00:02 0.00 0.00 03:00:00 0.00 0.00 04:00:00 0.00 0.00 05:00:01 0.00 0.00 06:00:00 0.00 0.00 Average 0.00 0.00 Use the sar -p command to report page-in activity, which includes protection and translation faults.$ sar -p 00:00:00 atch/s pgin/s ppgin/s pflt/s vflt/s slock/s 01:00:00 0.07 0.00 0.00 0.21 0.39 0.00 The following example shows output from the sar p command. $ sar -p SunOS balmyday 5.10 s10_51 sun4u 03/18/2004 00:00:04 atch/s pgin/s ppgin/s pflt/s vflt/s slock/s 01:00:00 0.09 0.00 0.00 0.78 2.02 0.00 02:00:01 0.08 0.00 0.00 0.78 2.02 0.00 03:00:00 0.09 0.00 0.00 0.81 2.07 0.00 04:00:00 0.11 0.01 0.01 0.86 2.18 0.00 05:00:00 0.08 0.00 0.00 0.78 2.02 0.00 06:00:00 0.09 0.00 0.00 0.78 2.02 0.00 07:00:00 0.08 0.00 0.00 0.78 2.02 0.00 08:00:00 0.09 0.00 0.00 0.78 2.02 0.00 08:20:00 0.11 0.00 0.00 0.87 2.24 0.00 08:40:01 0.13 0.00 0.00 0.90 2.29 0.00 09:00:00 0.11 0.00 0.00 0.88 2.24 0.00 09:20:00 0.10 0.00 0.00 0.88 2.24 0.00 09:40:00 2.91 1.80 2.38 4.61 17.62 0.00 10:00:00 2.74 2.03 3.08 8.17 21.76 0.00 10:20:00 0.16 0.04 0.04 1.92 2.96 0.00 10:40:00 2.10 2.50 3.42 6.62 16.51 0.00 11:00:00 3.36 0.87 1.35 3.92 15.12 0.00 Average 0.42 0.22 0.31 1.45 4.00 0.00The following table describes the reported statistics from the p option.Field Name Description atch/s The number of page faults, per second, that are satisfied by reclaiming a page currently in memory (attaches per second). Instances include reclaiming an invalid page from the free list and sharing a page of text that is currently being used by another process. An example is two or more processes that are accessing the same program text. pgin/s The number of times, per second, that file systems receive page-in requests. ppgin/s The number of pages paged in, per second. A single page-in request, such as a soft-lock request (see slock/s) or a large block size, might involve paging-in multiple pages. pflt/s The number of page faults from protection errors. Instances of protection faults indicate illegal access to a page and “copy-on-writes.” Generally, this number consists primarily of “copy-on-writes.” vflt/s The number of address translation page faults, per second. These faults are known as validity faults. Validity faults occur when a valid process table entry does not exist for a given virtual address. slock/s The number of faults, per second, caused by software lock requests that require physical I/O. An example of the occurrence of a soft-lock request is the transfer of data from a disk to memory. The system locks the page that is to receive the data so that the page cannot be claimed and used by another process. Use the sar q command to report the following information: The Average queue length while the queue is occupied. The percentage of time that the queue is occupied. $ sar -q 00:00:00 runq-sz %runocc swpq-sz %swpoccThe following list describes the output from the q option.runq-szThe number of kernel threads in memory that are waiting for a CPU to run. Typically, this value should be less than 2. Consistently higher values mean that the system might be CPU-bound. %runoccThe percentage of time that the dispatch queues are occupied. swpq-szNo longer reported by the sar command. %swpoccNo longer reported by the sar command. The following example shows output from the sar -q command. If the %runocc value is high (greater than 90 percent) and the runq-sz value is greater than 2, the CPU is heavily loaded and response is degraded. In this case, additional CPU capacity might be required to obtain acceptable system response. $ sar -q SunOS balmyday 5.10 s10_51 sun4u 03/18/2004 00:00:04 runq-sz %runocc swpq-sz %swpocc 01:00:00 1.0 0 0.0 0 02:00:01 1.3 0 0.0 0 03:00:00 1.0 0 0.0 0 04:00:00 1.0 0 0.0 0 05:00:00 1.0 0 0.0 0 06:00:00 2.0 0 0.0 0 07:00:00 0.0 0 0.0 0 08:00:00 1.0 0 0.0 0 08:20:00 1.0 0 0.0 0 08:40:01 2.0 0 0.0 0 09:00:00 0.0 0 0.0 0 09:20:00 1.0 0 0.0 0 09:40:00 1.2 2 0.0 0 10:00:00 1.2 2 0.0 0 10:20:00 1.0 1 0.0 0 10:40:00 1.3 9 0.0 0 11:00:00 1.2 7 0.0 0 Average 1.2 1 0.0 0 Use the sar -r command to report the number of memory pages and swap-file disk blocks that are currently unused. $ sar -r 00:00:00 freemem freeswap 01:00:00 2135 401922The following list describes the output from the r option.freememThe average number of memory pages that are available to user processes over the intervals sampled by the command. Page size is machine-dependent. freeswapThe number of 512-byte disk blocks that are available for page swapping. The following example shows output from the sar -r command.$ sar -r SunOS balmyday 5.10 s10_51 sun4u 03/18/2004 00:00:04 freemem freeswap 01:00:00 44717 1715062 02:00:01 44733 1715496 03:00:00 44715 1714746 04:00:00 44751 1715403 05:00:00 44784 1714743 06:00:00 44794 1715186 07:00:00 44793 1715159 08:00:00 44786 1714914 08:20:00 44805 1715576 08:40:01 44797 1715347 09:00:00 44761 1713948 09:20:00 44802 1715478 09:40:00 41770 1682239 10:00:00 35401 1610833 10:20:00 34295 1599141 10:40:00 33943 1598425 11:00:00 30500 1561959 Average 43312 1699242 Use the sar -u command to display CPU utilization statistics. $ sar -u 00:00:00 %usr %sys %wio %idle 01:00:00 0 0 0 100The sar command without any options is equivalent to the sar -u command. At any given moment, the processor is either busy or idle. When busy, the processor is in either user mode or system mode. When idle, the processor is either waiting for I/O completion or “sitting still” with no work to do. The following list describes output from the u option.%usrLists the percentage of time that the processor is in user mode %sysLists the percentage of time that the processor is in system mode %wioLists the percentage of time that the processor is idle and waiting for I/O completion %idleLists the percentage of time that the processor is idle and not waiting for I/O A high %wio value generally means that a disk slowdown has occurred. The following example shows output from the sar -u command. $ sar -u SunOS balmyday 5.10 s10_51 sun4u 03/18/2004 00:00:04 %usr %sys %wio %idle 01:00:00 0 0 0 100 02:00:01 0 0 0 100 03:00:00 0 0 0 100 04:00:00 0 0 0 100 05:00:00 0 0 0 100 06:00:00 0 0 0 100 07:00:00 0 0 0 100 08:00:00 0 0 0 100 08:20:00 0 0 0 99 08:40:01 0 0 0 99 09:00:00 0 0 0 99 09:20:00 0 0 0 99 09:40:00 4 1 0 95 10:00:00 4 2 0 94 10:20:00 1 1 0 98 10:40:00 18 3 0 79 11:00:00 25 3 0 72 Average 2 0 0 98 Use the sar -v command to report the status of the process table, inode table, file table, and shared memory record table. $ sar -v 00:00:00 proc-sz ov inod-sz ov file-sz ov lock-sz 01:00:00 43/922 0 2984/4236 0 322/322 0 0/0 The following abbreviated example shows output from the sar -v command. This example shows that all tables are large enough to have no overflows. These tables are all dynamically allocated based on the amount of physical memory.$ sar -v SunOS balmyday 5.10 s10_51 sun4u 03/18/2004 00:00:04 proc-sz ov inod-sz ov file-sz ov lock-sz 01:00:00 69/8010 0 3476/34703 0 0/0 0 0/0 02:00:01 69/8010 0 3476/34703 0 0/0 0 0/0 03:00:00 69/8010 0 3476/34703 0 0/0 0 0/0 04:00:00 69/8010 0 3494/34703 0 0/0 0 0/0 05:00:00 69/8010 0 3494/34703 0 0/0 0 0/0 06:00:00 69/8010 0 3494/34703 0 0/0 0 0/0 07:00:00 69/8010 0 3494/34703 0 0/0 0 0/0 08:00:00 69/8010 0 3494/34703 0 0/0 0 0/0 08:20:00 69/8010 0 3494/34703 0 0/0 0 0/0 08:40:01 69/8010 0 3494/34703 0 0/0 0 0/0 09:00:00 69/8010 0 3494/34703 0 0/0 0 0/0 09:20:00 69/8010 0 3494/34703 0 0/0 0 0/0 09:40:00 74/8010 0 3494/34703 0 0/0 0 0/0 10:00:00 75/8010 0 4918/34703 0 0/0 0 0/0 10:20:00 72/8010 0 4918/34703 0 0/0 0 0/0 10:40:00 71/8010 0 5018/34703 0 0/0 0 0/0 11:00:00 77/8010 0 5018/34703 0 0/0 0 0/0 Output from the v option is described in the following table.Field Name Description proc-sz The number of process entries (proc structures) that are currently being used, or allocated, in the kernel. inod-sz The total number of inodes in memory compared to the maximum number of inodes that are allocated in the kernel. This number is not a strict high watermark. The number can overflow. file-sz The size of the open system file table. The sz is given as 0, because space is allocated dynamically for the file table. ov The overflows that occur between sampling points for each table. lock-sz The number of shared memory record table entries that are currently being used, or allocated, in the kernel. The sz is given as 0 because space is allocated dynamically for the shared memory record table. Use the sar -w command to report swapping and switching activity.$ sar -w 00:00:00 swpin/s bswin/s swpot/s bswot/s pswch/s 01:00:00 0.00 0.0 0.00 0.0 22The following list describes target values and observations related to the sar w command output.swpin/sThe number of LWP transfers into memory per second. bswin/sThe number of blocks transferred for swap-ins per second. /* (float)PGTOBLK(xx->cvmi.pgswapin) / sec_diff */ swpot/sThe average number of processes that are swapped out of memory per second. If the number is greater than 1, you might need to increase memory. bswot/sThe number of blocks that are transferred for swap-outs per second. pswch/sThe number of kernel thread switches, per second. All process swap-ins include process initialization. The following example shows output from the sar -w command. $ sar -w SunOS balmyday 5.10 s10_51 sun4u 03/18/2004 00:00:04 swpin/s bswin/s swpot/s bswot/s pswch/s 01:00:00 0.00 0.0 0.00 0.0 132 02:00:01 0.00 0.0 0.00 0.0 133 03:00:00 0.00 0.0 0.00 0.0 133 04:00:00 0.00 0.0 0.00 0.0 134 05:00:00 0.00 0.0 0.00 0.0 133 06:00:00 0.00 0.0 0.00 0.0 133 07:00:00 0.00 0.0 0.00 0.0 132 08:00:00 0.00 0.0 0.00 0.0 131 08:20:00 0.00 0.0 0.00 0.0 133 08:40:01 0.00 0.0 0.00 0.0 132 09:00:00 0.00 0.0 0.00 0.0 132 09:20:00 0.00 0.0 0.00 0.0 132 09:40:00 0.00 0.0 0.00 0.0 335 10:00:00 0.00 0.0 0.00 0.0 601 10:20:00 0.00 0.0 0.00 0.0 353 10:40:00 0.00 0.0 0.00 0.0 747 11:00:00 0.00 0.0 0.00 0.0 804 Average 0.00 0.0 0.00 0.0 198 Use the sar -y command to monitor terminal device activities. $ sar -y 00:00:00 rawch/s canch/s outch/s rcvin/s xmtin/s mdmin/s 01:00:00 0 0 0 0 0 0If you have a lot of terminal I/O, you can use this report to determine if any bad lines exist. The activities recorded are defined in the following list.rawch/sInput characters (raw queue) per second canch/sInput characters that are processed by canon (canonical queue) per second outch/sOutput characters (output queue) per second rcvin/sReceiver hardware interrupts per second xmtin/sTransmitter hardware interrupts per second mdmin/sModem interrupts per second The number of modem interrupts per second (mdmin/s) should be close to zero. The receive and transmit interrupts per second (xmtin/s and rcvin/s) should be less than or equal to the number of incoming or outgoing characters, respectively. If not, check for bad lines. The following example shows output from the sar -y command.$ sar -y SunOS balmyday 5.10 s10_51 sun4u 03/18/2004 00:00:04 rawch/s canch/s outch/s rcvin/s xmtin/s mdmin/s 01:00:00 0 0 0 0 0 0 02:00:01 0 0 0 0 0 0 03:00:00 0 0 0 0 0 0 04:00:00 0 0 0 0 0 0 05:00:00 0 0 0 0 0 0 06:00:00 0 0 0 0 0 0 07:00:00 0 0 0 0 0 0 08:00:00 0 0 0 0 0 0 08:20:00 0 0 0 0 0 0 08:40:01 0 0 0 0 0 0 09:00:00 0 0 0 0 0 0 09:20:00 0 0 0 0 0 0 09:40:00 0 0 1 0 0 0 10:00:00 0 0 37 0 0 0 10:20:00 0 0 0 0 0 0 10:40:00 0 0 3 0 0 0 11:00:00 0 0 3 0 0 0 Average 0 0 1 0 0 0 Use the sar -A command to display statistics from all options to provide a view of overall system performance.This command provides a more global perspective. If data from more than a single time segment is shown, the report includes averages. Three commands are involved in the automatic collection of system activity data: sadc, sa1, and sa2. The sadc data collection utility periodically collects data on system activity and saves the data in a file in binary format, one file for each 24-hour period. You can set up the sadc command to run periodically (usually once each hour), and whenever the system boots to multiuser mode. The data files are placed in the /var/adm/sa directory. Each file is named sadd, where dd is the current date. The format of the command is as follows: /usr/lib/sa/sadc [t n] [ofile]The command samples n times with an interval of t seconds, which should be greater than five seconds between samples. This command then writes to the binary ofile file, or to standard output.The sadc command should be run at system boot time to record the statistics from when the counters are reset to zero. To make sure that the sadc command is run at boot time, the svcadm enable system/sar:default command writes a record to the daily data file. The command entry has the following format:/usr/bin/su sys -c "/usr/lib/sa/sadc /var/adm/sa/sa`date +%d`" To generate periodic records, you need to run the sadc command regularly. The simplest way to do so is to uncomment the following lines in the /var/spool/cron/crontabs/sys file: # 0 * * * 0-6 /usr/lib/sa/sa1 # 20,40 8-17 * * 1-5 /usr/lib/sa/sa1 # 5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -AThe sys crontab entries do the following:The first two crontab entries cause a record to be written to the /var/adm/sa/sadd file every 20 minutes from 8 a.m. to 5 p.m., Monday through Friday, and every hour on the hour otherwise. The third entry writes a record to the /var/adm/sa/sardd file hourly, Monday through Friday, and includes all sar options. You can change these defaults to meet your needs. Another shell script, sa2, produces reports rather than binary data files. The sa2 command invokes the sar command and writes the ASCII output to a report file. The sar command can be used either to gather system activity data itself or to report what has been collected in the daily activity files that are created by the sadc command. The sar command has the following formats: sar [-aAbcdgkmpqruvwy] [-o file] t [n] sar [-aAbcdgkmpqruvwy] [-s time] [-e time] [-i sec] [-f file]The following sar command samples cumulative activity counters in the operating system every t seconds, n times. The t should be five seconds or greater. Otherwise, the command itself might affect the sample. You must specify a time interval in which to take the samples. Otherwise, the command operates according to the second format. The default value of n is 1. The following example takes two samples separated by 10 seconds. If the o option were specified, samples are saved in binary format.$ sar -u 10 2Other important information about the sar command includes the following:With no sampling interval or number of samples specified, the sar command extracts data from a previously recorded file. This file is either the file specified by the f option or, by default, the standard daily activity file, /var/adm/sa/sadd, for the most recent day. The s and e options define the starting time and the ending time for the report. Starting and ending times are of the form hh[:mm[:ss]], where hh, mm, and ss represent hours, minutes, and seconds. The i option specifies, in seconds, the intervals between record selection. If the i option is not included, all intervals that are found in the daily activity file are reported. The following table lists the sar options and their actions. Option Actions a Checks file access operations b Checks buffer activity c Checks system calls d Checks activity for each block device g Checks page-out and memory freeing k Checks kernel memory allocation m Checks interprocess communication nv Checks system table status p Checks swap and dispatch activity q Checks queue activity r Checks unused memory u Checks CPU utilization w Checks swapping and switching volume y Checks terminal activity A Reports overall system performance, which is the same as entering all options. Using no option is equivalent to calling the sar command with the u option. &rolestep.sgm;Run the svcadm enable system/sar:default command.This version of the sadc command writes a special record that marks the time when the counters are reset to zero (boot time). Edit the /var/spool/cron/crontabs/sys crontab file.Do not edit a crontab file directly. Instead, use the crontab e command to make changes to an existing crontab file. # crontab -e sys Uncomment the following lines:0 * * * 0-6 /usr/lib/sa/sa1 20,40 8-17 * * 1-5 /usr/lib/sa/sa1 5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -AFor more information, see the crontab1 man page.