I was investigating gathering performance data on (oracle) linux servers recently and came across Performance Co-Pilot (PCP) . I have come across this product regularly in the past, but it seemed somewhat abstract to me, and I never ran into any actual usage. And we got sar for linux performance data and for the Oracle database we got oswatcher (and it’s exadata cousin exawatcher) and TFA right? How wrong I was.
First let me explain a few things.
There is no magic involved in performance measurement and tuning. What this aims to say is that most performance gathering frameworks use the same performance data sources as regular tools use. The are tools that can go beyond the regular data sources, like systemtap , which actually can be used as a source for PCP (but is beyond the scope of this article) and perf to name a few. This means that whatever tool you use, like: mpstat, iostat, top or sar, they all display the same cpu information. It might disguised in a different way, but really it’s all the same information.
This means that’s it’s not about the information itself, but about how you can interact with historical data. The majority of the performance tools simply do not have a facility to store measurements. That’s perfectly fine, and that wasn’t ever a design criterion when they were written. This is true for tools like mpstat, iostat, top to name a few. However in a lot of cases having volatile data is a real pain. Of course you can save the output to a file, but this very quickly becomes a challenge once you have a few files of sar, iostat, etc.
This is where sar comes in. Sar is installed on linux using the sysstat rpm package, and actually does store historical data! Sar stores it’s data in /var/log/sa, and stores sar archives historically based on the HISTORY setting in /etc/sysconfig/sysstat. In the past I seen this been set to 8, a check on a current oracle linux 6 machine shows HISTORY been set to 28. Also in the past the maximal retention time was 31, because the day of the month number would overwrite the same day of the previous month. This issue too seems to be solved, the comments in the /etc/sysconfig/sysstat file says that if the value is set higher than 28, files are kept in month-based directories.
The historical data sar stores are measurements with a ten minute interval. The reason for this is how sa1 (a helper utility for storing sar data) is invoked. This is done via a cron setting in /etc/cron.d/sysstat:
# Run system activity accounting tool every 10 minutes*/10 * * * * root /usr/lib64/sa/sa1 1 1
# 0 * * * * root /usr/lib64/sa/sa1 600 6 &
# Generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A
If you speak cron, you see that the second row instructs cron to run sa1 taking one sample every ten minutes (which is put in /var/log/saNN, and on line 5 to run sa2, which creates a (text)report called /var/log/sarNN, and does maintenance based on the HISTORY setting.
This is brilliant when coming from having no data at all. In fact, I have used this in the past and am using this with success to look into measurements looking for anomalies. However, it’s not very granular, it’s per ten minutes. Also, getting data is by extracting data from the /var/log/sa/saNN file (sar -f /var/log/sa/saNN -s hh:mm:ss -e hh:mm:ss (switches for data sources, like cpu% (u), swapping in/out (W), etc)). More frequently I just ask for the /var/log/sa/sarNN (text) report, which contains all sar measurements.
So far we are all happy. Performance data can be queried ad-hoc, and there’s a history with per 10 minute samples. But how about when you do test things on your system? In most of the cases I am involved with benchmarking and proof of concepts, we run things and actually want data which is more granular than 10 minutes. In fact, a single test might not even take 10 minutes and not be visible in the sar history at all! There is a solution for that too: setup sar, iostat, mpstat, etc. each with output into a file in the background, and then run whatever you want to run, and kill the performance collectors afterwards. Sounds great, but if you’ve ever tried that, it becomes a bit complex quite soon.
This is where performance co-pilot (PCP) comes in. PCP is an open-source tool that is available in the ‘latest’ repository of oracle linux 6 and 7, which means it can simply be installed using yum. Once installed, you can extract the very same data earlier mentioned tools get from a linux system using the pcp tools:
CPU information:
[opc@deploy1 ~]$ pmrep :sar-u%user %nice %sys %iowait %steal %idle
13:00:41 N/A N/A N/A N/A N/A N/A
13:00:42 0.47 0.00 0.11 0.00 0.00 99.41
13:00:43 0.96 0.00 0.12 0.00 0.00 98.89
13:00:44 0.21 0.00 0.06 0.00 0.00 99.72
13:00:45 1.15 0.00 0.17 0.00 0.00 98.65
13:00:46 0.03 0.00 0.01 0.00 0.00 99.96
13:00:47 1.22 0.00 0.14 0.00 0.00 98.58
13:00:48 0.58 0.00 0.18 0.00 0.00 99.23
13:00:49 0.98 0.00 0.12 0.00 0.00 98.85
13:00:50 0.18 0.00 0.04 0.00 0.00 99.77
13:00:51 1.15 0.00 0.18 0.00 0.00 98.65
13:00:52 0.07 0.00 0.01 0.00 0.00 99.91
13:00:53 1.19 0.00 0.15 0.00 0.00 98.63
Memory information:
[opc@deploy1 ~]$ pmrep :memfree(mb) bufhead(mb) pagecache(mb kernelstack( slab(mb) pagetables(m mapped(mb) anon(mb) shm(mb) swapuse(mb) hugefree(mb) hugeused(mb)
13:08:33 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
13:08:34 237199 320 17056 16 1113 23 62 1048 27 0 0 0
13:08:35 237258 320 17056 16 1113 24 62 991 27 0 0 0
13:08:36 237200 320 17056 16 1113 24 62 1046 27 0 0 0
13:08:37 237255 320 17056 16 1113 24 62 991 27 0 0 0
13:08:38 237212 320 17056 16 1113 24 62 1034 27 0 0 0
Network:
[opc@deploy1 ~]$ pmrep :nw-pktpck/in pck/in pck/in err/in err/in err/in drp/in drp/in drp/in pck/out pck/out pck/out err/out err/out err/out drp/out drp/out drp/out
eth0 eth1 lo eth0 eth1 lo eth0 eth1 lo eth0 eth1 lo eth0 eth1 lo eth0 eth1 lo
13:08:58 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
13:08:59 10.972 0.000 29.924 0.000 0.000 0.000 0.000 0.000 0.000 16.957 0.000 29.924 0.000 0.000 0.000 0.000 0.000 0.000
13:09:00 0.996 0.000 103.63 0.000 0.000 0.000 0.000 0.000 0.000 0.996 0.000 103.627 0.000 0.000 0.000 0.000 0.000 0.000
13:09:01 0.996 0.000 29.892 0.000 0.000 0.000 0.000 0.000 0.000 0.996 0.000 29.892 0.000 0.000 0.000 0.000 0.000 0.000
13:09:02 0.996 0.000 94.653 0.000 0.000 0.000 0.000 0.000 0.000 0.996 0.000 94.653 0.000 0.000 0.000 0.000 0.000 0.000
Disk:
[opc@deploy1 ~]$ pmrep :disk-iopsriops wiops iops
sda sda sda
13:09:23 N/A N/A N/A
13:09:24 0 0 0
13:09:25 0 0 0
13:09:26 0 0 0
13:09:27 0 3 3
13:09:28 0 0 0
As you can see, all these statistics where provided using a utility called ‘pmrep’ (reporter utility). They were fetched on runtime. I can go back in time when I use the archive that the pcp logger creates and use -S (starttime):
[opc@deploy1 ~]$ pmrep -a /var/log/pcp/pmlogger/deploy1/20170113.00.10 -S @-1min :sar-u%user %nice %sys %iowait %steal %idle
13:22:36 N/A N/A N/A N/A N/A N/A
13:22:37 0.65 0.00 0.11 0.00 0.00 99.22
13:22:38 0.65 0.00 0.11 0.00 0.00 99.22
13:22:39 0.65 0.00 0.11 0.00 0.00 99.22
13:22:40 0.65 0.00 0.11 0.00 0.00 99.22
13:22:41 0.65 0.00 0.11 0.00 0.00 99.22
13:22:42 0.65 0.00 0.11 0.00 0.00 99.22
13:22:43 0.65 0.00 0.11 0.00 0.00 99.22
13:22:44 0.65 0.00 0.11 0.00 0.00 99.22
13:22:45 0.65 0.00 0.11 0.00 0.00 99.22
13:22:46 0.65 0.00 0.11 0.00 0.00 99.22
13:22:47 0.65 0.00 0.11 0.00 0.00 99.22
13:22:48 0.65 0.00 0.11 0.00 0.00 99.22
The archive is stored in /var/log/pcp/pmlogger/hostname/YYYYMMDD.00.10 for the current day, the older archives are also stored as YYYYMMDD, but without ‘.00.10’ added to it. Above the syntax for using an archive and specifying a relative time is shown (-S for start, then a number and a time specification like sec, min, hour). Of course you can specify absolute times too, like ‘-S @13:22 -T @13:25’. -T is the until time.
Did you notice the measurements read from the archive above are per second?? The metrics are actually measured per minute (this specified in /var/lib/pcp/config/pmlogger/config.default), the pmrep tool extrapolates the values per minute. I am not really happy about that, but you can use ‘-u’ to disable per second extrapolation.
When doing investigation, you often want all data from a certain point in time because quite often you don’t know upfront what exact you are going to encounter. This is also true for troubleshooting, proof of concept testing, performance analysis, benchmarking, etc. You want to be able to do multidimensional analysis after the investigated event, probably even offline. This is a point where PCP truly shines. It easy to create an archive of only the relevant time using the ‘pmlogextract’ utility:
[opc@deploy1 ~]$ pmlogextract -S 12:30 -T 12:40 /var/log/pcp/pmlogger/deploy1/20170113.00.10 thisThe above pmlogextract command reads the (current, there’s .00.10 attached) archive /var/log/pcp/pmlogger/deploy1/20170113.00.10 and creates the archive ‘this’ with all measurements between 12:30 and 12:40. A PCP archive consists of three files:
[opc@deploy1 ~]$ ls -ls this*236 -rw-rw-r--. 1 opc opc 238212 Jan 13 14:05 this.0
4 -rw-rw-r--. 1 opc opc 192 Jan 13 14:05 this.index
20 -rw-rw-r--. 1 opc opc 18456 Jan 13 14:05 this.meta
This archive is endian-independent, and can be transferred to your laptop for analysis (provided you got the PCP executables on it of course). Please mind an archive contains a subset of all available performance metrics, in order to save space. My default installation has 1620 metrics available, there are 280 stored in the archive.
But it doesn’t stop here. PCP not only installs a logger daemon (pmlogger) and a couple of agents to extract data (pmdaroot, pmdaproc, pmdaxfs, pmdalinux), but also the ‘pmcd’ daemon (pcp collector deamon) listening on port 44321. This means you can fetch metrics over the network:
[opc@deploy1 ~]$ pmrep -h oracleserver.local :memfree(mb) bufhead(mb) pagecache(mb kernelstack( slab(mb) pagetables(m mapped(mb) anon(mb) shm(mb) swapuse(mb) hugefree(mb) hugeused(mb)
15:25:18 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
15:25:19 507244 165 5436 19 499 165 1267 817 3015 0 0 0
15:25:20 507244 165 5436 19 499 165 1267 817 3015 0 0 0
15:25:21 507244 165 5436 19 499 165 1267 817 3015 0 0 0
15:25:22 507244 165 5436 19 499 165 1267 817 3015 0 0 0
15:25:23 507244 165 5436 19 499 165 1267 817 3015 0 0 0
15:25:24 507244 165 5436 19 499 165 1267 817 3015 0 0 0
This is running my memory overview report fetching current data from the server ‘oracleserver.local’! Sadly, currently is fetching historical data using pmrep via the ‘pmcd’ daemon not supported.
The last point I wanted to focus the attention on is that gathering PCP collected performance data on a central host is built-in in PCP using the logger functionality. Provided the collection host can access the remote host on tcp/44321, it’s really simple (example for above mentioned oracleserver.local host). This is all that needs to be done on the collection host. Nothing needs to be done on the host that needs to provide the data:
[root@deploy1 ~]# echo 'oracleserver.local n n PCP_LOG_DIR/pmlogger/oracleserver.local -r -T24h10m -c config.oracleserver.local' >> /etc/pcp/pmlogger/control[root@deploy1 ~]# service pmlogger restart
The installation of PCP via the oracle linux 6 or 7 repository is really simple (provided you enabled the ‘latest’ repository):
[root@deploy1 ~]# yum install pcp pcp-system-tools[root@deploy1 ~]# chkconfig pmcd on
[root@deploy1 ~]# chkconfig pmlogger on
[root@deploy1 ~]# service pmcd start
[root@deploy1 ~]# service pmlogger start
ps. I gotten a message ‘Error: Failed to create avahi client: Daemon not running’ when executing ‘service pmcd start’, but it does not seem to impact using PCP.
A final remark: the pmrep reports (:sar-u, :mem, :nw-pkt, disk-iops) mentioned are NOT reports that are in the default installation of PCP. Create a file in the current working directory of the process executing pmrep named pmrep.conf, and use the following gist to fill the file with my report definitions. Actually, it’s really easy to create a report, and there plenty of examples with the ones I created. If the pmrep.conf file is not in the current working directory, it needs to be specific using -c /path/to/pmrep.conf.