1************************************************************************** 2* The following are additional notes on the cputimes command. 3* 4* $Id: cputimes_notes.txt,v 1.1.1.1 2015/09/30 22:01:08 christos Exp $ 5* 6* COPYRIGHT: Copyright (c) 2007 Brendan Gregg. 7************************************************************************** 8 9 10* How cputimes works 11 12cputimes measures time consumed by the kernel, idle therads and processes, 13by tracking the activity of the schedular. In particular we track on-cpu 14and off-cpu events for kernel therads, measuring the timestamps at each event. 15 16 17* Why cputimes? 18 19If you are interested in how much time processes are consuming, the data 20given by "prstat" or "prstat -m" is fine. However there is no easy way to 21see kernel consumed time, which is the idea behind cputimes. 22 23 24* What does it mean? 25 26The output shows categories of threads by the sum of time, in nanoseconds. 27 28A nanosecond is 10^-9, or 0.000000001 of a second. This program uses 29nanoseconds as units, but does not have nanosecond accuracy. It would be 30reasonable to assume that this has microsecond accuracy (10^-6), so in 31practise ignore the last three digits of the times. 32 33The sections reported are, 34 35PROCESSES - the sum of all the process time on the CPU. 36KERNEL - the sum of the time spent in the kernel. 37IDLE - the time the kernel spent in the idle thread, waiting for some work. 38 39If your system isn't doing much, then the idle time will be quite large. If 40your system is running many applications, then there may be no idle time 41at all - instead most of the time appearing under processes. 42 43 44* When is there a problem? 45 46Expect to see most of the time in processes or idle, depending on how busy 47your server is. Seeing a considerable amout of time in kernel would 48definately be interesting. 49 50The kernel generally doesn't use much CPU time, usually less than 5%. 51If it were using more, that may indicate heavy activity from an interrupt 52thread, or activity caused by DTrace. 53 54For example, 55 56 # cputimes 1 57 2005 Apr 27 23:49:32, 58 THREADS TIME (ns) 59 IDLE 28351679 60 KERNEL 436022725 61 PROCESS 451304688 62 63In this sample the kernel is using a massive amount of the CPUs, around 47%. 64This sample was taken during heavy network utilisation, the time consumed 65by the TCP/IP and network driver threads (and DTrace). The "intrstat" command 66could be used for further analysis of the interrupt threads responsible 67for servicing the network interface. 68 69 70* Problems with cputimes 71 72The way cputimes measures schedular activity turns out to be a lot of work. 73There are many scheduling events per second where one thread steps onto a 74CPU and another leaves. It turns out that cputimes itself causes some degree 75of kernel load. 76 77Here we run 1 cputimes, 78 79 # cputimes 1 80 2005 May 15 12:00:41, 81 THREADS TIME (ns) 82 KERNEL 12621985 83 PROCESS 982751579 84 2005 May 15 12:00:42, 85 THREADS TIME (ns) 86 KERNEL 12267577 87 PROCESS 983513765 88 [...] 89 90Now a second cputimes is run at the same time, 91 92 # cputimes 1 93 2005 May 15 12:02:06, 94 THREADS TIME (ns) 95 KERNEL 17366426 96 PROCESS 978804165 97 2005 May 15 12:02:07, 98 THREADS TIME (ns) 99 KERNEL 17614829 100 PROCESS 978671601 101 [...] 102 103And now a third, 104 105 # cputimes 1 106 2005 May 15 12:03:09, 107 THREADS TIME (ns) 108 KERNEL 21303089 109 PROCESS 974925124 110 2005 May 15 12:03:10, 111 THREADS TIME (ns) 112 KERNEL 21222992 113 PROCESS 975152727 114 [...] 115 116Each extra cputimes is consuming an extra 4 to 5 ms of the CPU as kernel time. 117Around 0.5%. This can be used as an estimate of the kernel load caused by 118running cputimes, and a similar strategy could be used to measure the kernel 119load of other DTrace scripts. 120 121However the following CPU characteristics must be taken into consideration, 122 123 # psrinfo -v 124 Status of virtual processor 0 as of: 05/15/2005 12:06:05 125 on-line since 04/30/2005 13:32:32. 126 The i386 processor operates at 867 MHz, 127 and has an i387 compatible floating point processor. 128 129as well as the type of activity that was also running on the system, which 130cputimes was monitoring (frequency of scheduling events). 131 132A system with a slower CPU will use a larger proportion of kernel time to 133perform the same tasks. Also, a system that is context switching more 134(switching between different processes) is likely to consume more kernel time 135as well. 136 137 138 139