In my last article I had explained the various CPU c-states in detail and how you can disable the CPU states, and different commands which can show you the memory usage per process in Linux. These can help you troubleshoot performance related issues.
In this article I will show the usage of powerstat to measure power in watts for various tuned profile and also with some load.
I will be using Red Hat Enterprise Linux 7 (RHEL 7) for demonstrating the commands.
Install powerstat
If powerstat is available at your repository then you can install the same using
# yum install powerstat
Or you can always download this tool based on your environment and install it locally.
# rpm -Uvh /tmp/powerstat-0.02.17-10.1.x86_64.rpm warning: /tmp/powerstat-0.02.17-10.1.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID ee454f98: NOKEY Preparing... ################################# [100%] Updating / installing... 1:powerstat-0.02.17-10.1 ################################# [100%]
IMPORTANT NOTE: The power usage will fluctuate mostly when a CPU/Memory/Disk IO eating process transitions from being idle to running multiple times. As based on the time taken for this transition i.e. coming from deeper CPU c-state to C) takes time and impacts the power usage. If a process is all the time in running state then the power usage will be constant.
With profile throughput-performance
Case 1: System is Idle
Before starting let us measure the power usage when the system is completely idle
In the below example I am taking 60 samples or power usage taken per second.
# powerstat -R -c -z Running for 60.0 seconds (60 samples at 1.0 second intervals). Power measurements will start in 0 seconds time. Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts 09:51:23 0.0 0.0 0.1 99.9 0.0 1 576 718 0 0 0 29.51 09:51:24 0.0 0.0 0.1 99.9 0.0 1 620 615 0 0 0 29.37 09:51:25 0.1 0.0 0.0 99.9 0.0 1 791 605 0 0 0 29.66 09:51:26 0.1 0.0 0.0 99.9 0.0 1 698 632 0 0 0 29.47 09:51:27 0.0 0.0 0.1 99.9 0.0 1 591 606 0 0 0 29.43 09:51:28 0.1 0.0 0.1 99.8 0.0 1 2757 1173 1 0 1 30.54 09:51:29 0.3 0.0 0.3 99.4 0.0 1 8237 1670 0 0 2 37.66 09:51:30 0.1 0.0 0.0 99.9 0.0 1 667 660 0 0 0 29.51 09:51:31 0.1 0.0 0.0 99.9 0.0 1 664 648 0 0 0 29.51 09:51:32 0.0 0.0 0.1 99.9 0.0 1 595 589 0 0 0 29.43 09:51:33 0.0 0.0 0.0 100.0 0.0 1 604 623 0 0 0 29.40 09:51:34 0.1 0.0 0.0 99.9 0.0 1 581 570 0 0 0 29.36 09:51:35 0.0 0.0 0.1 99.9 0.0 1 606 628 0 0 0 29.43 09:51:36 0.1 0.0 0.1 99.9 0.0 1 702 665 0 0 0 29.41 09:51:37 0.0 0.0 0.0 100.0 0.0 1 578 583 0 0 0 29.37 09:51:38 0.0 0.0 0.0 100.0 0.0 1 566 557 0 0 0 29.26 09:51:39 0.1 0.0 0.1 99.9 0.0 1 661 691 0 0 0 29.49 09:51:40 0.1 0.0 0.0 99.9 0.0 1 732 706 0 0 0 29.41 09:51:41 0.0 0.0 0.1 99.9 0.0 1 645 641 0 0 0 29.41 09:51:42 0.0 0.0 0.0 100.0 0.0 1 558 534 0 0 0 29.35 09:51:43 0.1 0.0 0.0 99.9 0.0 1 553 544 0 0 0 29.33 09:51:44 0.0 0.0 0.0 100.0 0.0 1 619 609 0 0 0 29.36 09:51:45 0.0 0.0 0.1 99.9 0.0 1 570 569 0 0 0 29.39 09:51:46 0.0 0.0 0.0 100.0 0.0 1 679 636 0 0 0 29.45 09:51:47 0.1 0.0 0.0 99.9 0.0 1 547 540 0 0 0 29.36 09:51:48 0.0 0.0 0.1 99.9 0.0 1 686 760 1 0 1 29.49 09:51:49 0.1 0.0 0.0 99.9 0.0 1 616 600 0 0 0 29.38 09:51:50 0.1 0.0 0.1 99.8 0.0 1 864 897 12 12 5 29.69 09:51:51 0.1 0.0 0.1 99.9 0.0 1 714 723 1 1 5 29.47 09:51:52 0.0 0.0 0.0 100.0 0.0 1 596 577 0 0 0 29.40 09:51:53 0.0 0.0 0.0 100.0 0.0 1 599 583 0 0 0 29.35 09:51:54 0.0 0.0 0.1 99.9 0.0 1 574 560 0 0 0 29.30 09:51:55 0.1 0.0 0.0 99.9 0.0 1 588 592 0 0 0 29.36 09:51:56 0.0 0.0 0.0 100.0 0.0 1 702 683 0 0 0 29.47 09:51:57 0.1 0.0 0.1 99.9 0.0 1 559 566 0 0 0 29.36 Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts 09:51:58 0.1 0.0 0.1 99.9 0.0 1 2596 1047 0 0 0 30.44 09:51:59 0.3 0.0 0.2 99.5 0.0 1 8247 1642 1 0 0 33.62 09:52:00 0.1 0.0 0.0 99.9 0.0 1 801 740 2 0 2 29.52 09:52:01 0.0 0.0 0.0 100.0 0.0 1 685 671 0 0 1 29.44 09:52:02 0.0 0.0 0.1 99.9 0.0 1 569 550 0 0 1 29.28 09:52:03 0.1 0.0 0.0 99.9 0.0 1 605 636 0 0 0 29.35 09:52:04 0.0 0.0 0.1 99.9 0.0 1 628 600 0 0 0 29.30 09:52:05 0.1 0.0 0.0 99.9 0.0 1 752 730 0 0 0 29.58 09:52:06 0.1 0.0 0.1 99.9 0.0 1 692 636 0 0 0 29.36 09:52:07 0.0 0.0 0.1 99.9 0.0 1 557 560 0 0 0 29.37 09:52:08 0.0 0.0 0.0 100.0 0.0 1 677 707 1 0 1 29.38 09:52:09 0.1 0.0 0.1 99.8 0.0 1 675 717 0 0 0 29.43 09:52:10 0.0 0.0 0.1 99.9 0.0 1 696 663 0 0 0 29.35 09:52:11 0.1 0.0 0.0 99.9 0.0 1 658 640 0 0 0 29.35 09:52:12 0.2 0.0 0.2 99.6 0.0 1 642 662 0 0 0 29.83 09:52:13 0.0 0.0 0.1 99.9 0.0 1 561 563 0 0 0 29.31 09:52:14 0.1 0.0 0.0 99.9 0.0 1 584 583 0 0 0 29.27 09:52:15 0.0 0.0 0.0 100.0 0.0 1 614 682 0 0 0 29.47 09:52:16 0.1 0.0 0.1 99.9 0.0 1 667 626 0 0 0 29.42 09:52:17 0.0 0.0 0.0 100.0 0.0 1 555 547 0 0 0 29.33 09:52:18 0.0 0.0 0.0 100.0 0.0 1 610 618 0 0 0 29.28 09:52:19 0.1 0.0 0.1 99.9 0.0 1 624 629 0 0 0 29.32 09:52:20 0.0 0.0 0.0 100.0 0.0 1 697 666 0 0 0 29.34 09:52:21 0.1 0.0 0.0 99.9 0.0 1 629 618 0 0 0 29.34 09:52:22 0.0 0.0 0.1 99.9 0.0 1 555 530 0 0 0 29.28 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Average 0.0 0.0 0.0 99.9 0.0 1.0 957.9 680.3 0.3 0.2 0.3 29.65 GeoMean 0.0 0.0 0.0 99.9 0.0 1.0 723.3 659.7 0.0 0.0 0.0 29.63 StdDev 0.1 0.0 0.1 0.1 0.0 0.0 1402.9 211.4 1.6 1.5 1.0 1.19 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Minimum 0.0 0.0 0.0 99.4 0.0 1.0 547.0 530.0 0.0 0.0 0.0 29.26 Maximum 0.3 0.0 0.3 100.0 0.0 1.0 8247.0 1670.0 12.0 12.0 5.0 37.66 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Summary: CPU: 29.65 Watts on average with standard deviation 1.19 Note: power read from RAPL domains: dram, dram, package-0, package-1. These readings do not cover all the hardware in this device. C-State Resident Count Latency C6-HSW 99.085% 42217 133 C3-HSW 0.318% 9200 33 C1E-HSW 0.170% 4761 10 C1-HSW 0.246% 3428 2 POLL 0.001% 3 0 C0 0.181%
So on my blade the average power usage when the system is idle is ~30W.
As you see when the system is dile the CPU goes into deeper sleep state hence saving power.
Case 2: When the system is under load
To reproduce this environment I will use "stress" tool to put some load on my CPU, memory and disks.
I will be running below command in one of the terminal to put some load on the node
# stress -c 18 -i 13 -m 3 stress: info: [5200] dispatching hogs: 18 cpu, 13 io, 3 vm, 0 hdd
Next check the power usage
# powerstat -R -c -z Running for 60.0 seconds (60 samples at 1.0 second intervals). Power measurements will start in 0 seconds time. Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts 09:58:01 78.6 0.0 21.4 0.0 0.0 35 5132 18480 2 0 3 137.36 09:58:02 78.9 0.0 21.1 0.0 0.0 27 5028 18275 0 0 1 137.70 09:58:03 79.8 0.0 20.2 0.0 0.0 30 4792 17996 0 0 0 137.81 09:58:04 75.2 0.0 24.8 0.0 0.0 36 5568 18005 0 0 0 136.29 09:58:05 77.2 0.0 22.8 0.0 0.0 32 5981 18444 0 0 0 137.00 09:58:06 76.1 0.0 23.9 0.0 0.0 34 5681 17843 0 0 0 136.73 09:58:07 77.1 0.0 22.9 0.0 0.0 35 4727 17743 1 0 0 137.35 09:58:08 72.5 0.0 27.5 0.0 0.0 33 5803 18002 0 0 0 136.11 09:58:09 74.9 0.0 25.1 0.0 0.0 35 5751 17756 1 0 1 136.75 09:58:10 71.7 0.0 28.3 0.0 0.0 30 6368 18299 0 0 0 135.94 09:58:11 70.8 0.0 29.2 0.0 0.0 38 6079 18402 0 0 0 135.44 09:58:12 71.5 0.0 28.5 0.0 0.0 36 6201 18146 0 0 0 135.58 09:58:13 76.0 0.0 24.0 0.0 0.0 28 5391 18074 0 0 0 137.09 09:58:14 74.9 0.0 25.1 0.0 0.0 35 5834 18167 0 0 0 136.93 09:58:15 72.9 0.0 27.1 0.0 0.0 37 6104 17988 0 0 0 136.23 09:58:16 70.5 0.0 29.5 0.0 0.0 30 6029 17959 0 0 0 135.77 09:58:17 73.9 0.0 26.1 0.0 0.0 34 5782 18211 1 0 0 137.25 09:58:18 73.9 0.0 26.1 0.0 0.0 28 4833 17817 0 0 0 137.08 09:58:19 74.4 0.0 25.6 0.0 0.0 38 5664 18141 0 0 0 137.02 09:58:20 74.2 0.0 25.8 0.0 0.0 37 6218 18179 0 0 0 136.47 09:58:21 76.9 0.0 23.1 0.0 0.0 30 6020 18442 0 0 0 137.69 09:58:22 74.7 0.0 25.3 0.0 0.0 32 4904 17974 0 0 0 137.22 09:58:23 77.2 0.0 22.8 0.0 0.0 38 5344 17879 0 0 0 137.94 09:58:24 73.9 0.0 26.1 0.0 0.0 35 6255 18041 0 0 0 137.16 09:58:25 73.9 0.0 26.1 0.0 0.0 35 5744 18089 0 0 0 137.06 09:58:26 74.8 0.0 25.2 0.0 0.0 33 5669 18022 0 0 0 137.52 09:58:27 71.7 0.0 28.3 0.0 0.0 36 5451 17969 0 0 0 136.69 09:58:28 74.8 0.0 25.2 0.0 0.0 36 5789 18018 1 0 0 137.51 09:58:29 76.6 0.0 23.4 0.0 0.0 32 8520 18747 8 0 1 137.83 09:58:30 74.2 0.0 25.8 0.0 0.0 34 7467 18298 2 0 0 137.44 09:58:31 68.9 0.0 31.1 0.0 0.0 36 5975 18074 0 0 0 135.63 09:58:32 75.0 0.0 25.0 0.0 0.0 35 5031 17742 0 0 0 137.87 09:58:33 74.6 0.0 25.4 0.0 0.0 31 4711 17499 1 0 0 137.88 09:58:34 70.6 0.0 29.4 0.0 0.0 36 5922 17785 0 0 0 136.76 09:58:35 73.3 0.0 26.7 0.0 0.0 33 5390 17845 0 0 0 137.48 Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts 09:58:36 70.7 0.0 29.3 0.0 0.0 37 6319 18248 0 0 0 136.09 09:58:37 71.5 0.0 28.5 0.0 0.0 38 6036 18126 0 0 0 136.75 09:58:38 73.1 0.0 26.9 0.0 0.0 31 5788 18040 0 0 0 136.98 09:58:39 74.6 0.0 25.4 0.0 0.0 36 5287 17844 0 0 0 137.73 09:58:40 78.6 0.0 21.4 0.0 0.0 34 6193 17913 0 0 0 138.11 09:58:41 75.5 0.0 24.5 0.0 0.0 36 5973 18012 0 0 0 137.70 09:58:42 75.6 0.0 24.4 0.0 0.0 36 5548 17802 0 0 0 137.91 09:58:43 77.1 0.0 22.9 0.0 0.0 29 5514 17557 0 0 0 137.98 09:58:44 77.5 0.0 22.5 0.0 0.0 36 5243 17797 0 0 0 138.10 09:58:45 75.4 0.0 24.6 0.0 0.0 38 6221 17947 38 34 27 137.92 09:58:46 73.9 0.0 26.1 0.0 0.0 35 6462 18384 38 36 48 137.11 09:58:47 73.2 0.0 26.8 0.0 0.0 35 6237 18120 0 0 0 136.74 09:58:48 77.7 0.0 22.3 0.0 0.0 27 5607 18152 1 0 0 138.17 09:58:49 74.9 0.0 25.1 0.0 0.0 37 5490 17965 0 0 1 137.79 09:58:50 76.9 0.0 23.1 0.0 0.0 36 5116 17777 12 12 5 138.42 09:58:51 74.9 0.0 25.1 0.0 0.0 38 5208 17733 1 1 5 138.06 09:58:52 74.1 0.0 25.9 0.0 0.0 33 5093 17786 0 0 0 137.58 09:58:53 73.8 0.0 26.2 0.0 0.0 37 5785 17906 0 0 0 137.43 09:58:54 71.6 0.0 28.4 0.0 0.0 33 5277 17565 0 0 0 137.07 09:58:55 68.8 0.0 31.2 0.0 0.0 29 5674 17579 0 0 0 136.17 09:58:56 72.2 0.0 27.8 0.0 0.0 31 5552 17699 0 0 0 137.55 09:58:57 71.7 0.0 28.3 0.0 0.0 31 5371 17603 0 0 0 137.45 09:58:58 72.0 0.0 28.0 0.0 0.0 28 5472 18050 0 0 0 137.22 09:58:59 72.5 0.0 27.5 0.0 0.0 30 8640 18744 2 0 0 137.22 09:59:00 73.8 0.0 26.2 0.0 0.0 36 6647 18271 6 0 0 137.41 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Average 74.3 0.0 25.7 0.0 0.0 33.8 5781.9 18016.2 1.9 1.4 1.5 137.17 GeoMean 74.3 0.0 25.6 0.0 0.0 33.6 5741.1 18014.2 0.0 0.0 0.0 137.17 StdDev 2.4 0.0 2.4 0.0 0.0 3.1 727.8 268.8 7.0 6.4 7.0 0.71 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Minimum 68.8 0.0 20.2 0.0 0.0 27.0 4711.0 17499.0 0.0 0.0 0.0 135.44 Maximum 79.8 0.0 31.2 0.0 0.0 38.0 8640.0 18747.0 38.0 36.0 48.0 138.42 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Summary: CPU: 137.17 Watts on average with standard deviation 0.71 Note: power read from RAPL domains: dram, dram, package-0, package-1. These readings do not cover all the hardware in this device. C-State Resident Count Latency C6-HSW 0.000% 3 133 C3-HSW 0.000% 4 33 C1E-HSW 0.000% 5 10 C1-HSW 0.000% 2 2 POLL 0.000% 0 0 C0 100.000%
As you see the CPU are always at running state "C0" and the power usage is around ~137W
With profile latency-performance
In this tuned profile we set the force_latency value to "1" hence the CPU c-state will not go lower than C1 even when the system is idle. Hence the power usage will always be compartiviely high as the system never sleeps in real.
Case 1: System is Idle
Before starting let us measure the power usage when the system is completely idle
# powerstat -R -c -z Running for 60.0 seconds (60 samples at 1.0 second intervals). Power measurements will start in 0 seconds time. Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts 10:06:41 0.1 0.0 0.0 99.9 0.0 1 641 731 0 0 0 72.39 10:06:42 0.0 0.0 0.0 100.0 0.0 1 561 518 0 0 0 71.48 10:06:43 0.0 0.0 0.1 99.9 0.0 1 566 534 0 0 0 71.17 10:06:44 0.0 0.0 0.0 100.0 0.0 1 610 570 0 0 0 71.75 10:06:45 0.6 0.0 1.0 98.4 0.0 1 2224 2340 75 70 75 72.49 10:06:46 0.0 0.0 0.0 100.0 0.0 1 633 594 0 0 0 71.41 10:06:47 0.1 0.0 0.0 99.9 0.0 1 622 585 0 0 0 71.43 10:06:48 0.0 0.0 0.1 99.9 0.0 1 728 806 1 0 1 71.49 10:06:49 0.0 0.0 0.0 100.0 0.0 1 631 599 0 0 0 71.49 10:06:50 0.1 0.0 0.1 99.9 0.0 1 845 928 10 12 6 71.44 10:06:51 0.0 0.0 0.1 99.9 0.1 1 723 772 1 1 5 71.99 10:06:52 0.0 0.0 0.0 100.0 0.0 1 603 571 0 0 0 71.46 10:06:53 0.0 0.0 0.0 100.0 0.0 1 560 541 0 0 0 71.40 10:06:54 0.0 0.0 0.0 100.0 0.0 1 594 566 0 0 0 71.34 10:06:55 0.1 0.0 0.0 99.9 0.0 1 580 561 0 0 0 71.53 10:06:56 0.0 0.0 0.0 100.0 0.0 1 658 639 0 0 0 71.96 10:06:57 0.0 0.0 0.1 99.9 0.0 1 593 555 0 0 0 71.41 10:06:58 0.1 0.0 0.0 99.9 0.0 1 2367 823 0 0 0 71.46 10:06:59 0.1 0.0 0.0 99.9 0.0 1 9397 1464 0 0 0 71.89 10:07:00 0.0 0.0 0.1 99.9 0.0 1 796 734 2 0 2 71.70 10:07:01 0.0 0.0 0.0 100.0 0.0 1 656 632 0 0 0 71.52 10:07:02 0.1 0.0 0.0 99.9 0.0 1 563 543 0 0 1 73.34 10:07:03 0.0 0.0 0.1 99.9 0.0 1 600 645 0 0 0 71.81 10:07:04 0.0 0.0 0.0 100.0 0.0 1 630 604 0 0 0 71.77 10:07:05 0.0 0.0 0.1 99.9 0.0 1 754 720 0 0 0 72.06 10:07:06 0.1 0.0 0.0 99.9 0.0 1 666 656 0 0 0 71.59 10:07:07 0.0 0.0 0.0 100.0 0.0 1 601 557 0 0 0 71.53 10:07:08 0.0 0.0 0.1 99.9 0.0 1 683 696 1 0 1 71.52 10:07:09 0.1 0.0 0.0 99.9 0.0 1 669 676 0 0 0 71.55 10:07:10 0.0 0.0 0.0 100.0 0.0 1 691 652 0 0 0 71.43 10:07:11 0.0 0.0 0.1 99.9 0.1 1 669 654 0 0 0 71.58 10:07:12 0.1 0.0 0.0 99.9 0.0 1 583 552 0 0 0 71.54 10:07:13 0.0 0.0 0.0 100.0 0.0 1 580 559 0 0 0 72.71 10:07:14 0.0 0.0 0.0 100.0 0.0 1 614 580 0 0 0 72.01 10:07:15 0.0 0.0 0.0 100.0 0.0 1 553 521 0 0 0 72.33 Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts 10:07:16 0.0 0.0 0.0 100.0 0.0 1 672 654 0 0 0 73.04 10:07:17 0.0 0.0 0.0 100.0 0.0 1 612 551 0 0 0 72.55 10:07:18 0.0 0.0 0.0 100.0 0.0 1 610 606 0 0 0 72.39 10:07:19 0.1 0.0 0.1 99.9 0.0 1 599 567 0 0 0 72.69 10:07:20 0.0 0.0 0.0 100.0 0.0 1 715 675 0 0 0 72.71 10:07:21 0.0 0.0 0.0 100.0 0.0 1 634 606 0 0 0 72.86 10:07:22 0.1 0.0 0.0 99.9 0.0 1 586 539 0 0 0 72.86 10:07:23 0.0 0.0 0.0 100.0 0.0 1 553 520 0 0 0 72.83 10:07:24 0.0 0.0 0.0 100.0 0.0 1 621 589 0 0 0 72.71 10:07:25 0.0 0.0 0.0 100.0 0.0 1 565 531 0 0 0 72.65 10:07:26 0.0 0.0 0.1 99.9 0.0 1 595 570 0 0 0 72.73 10:07:27 0.2 0.0 0.1 99.7 0.0 1 607 607 0 0 0 73.01 10:07:28 0.1 0.0 0.0 99.9 0.0 1 2479 1010 1 0 2 72.77 10:07:29 0.1 0.0 0.1 99.8 0.0 1 9091 1576 1 0 1 72.98 10:07:30 0.0 0.0 0.0 100.0 0.0 1 675 655 0 0 0 72.91 10:07:31 0.1 0.0 0.0 99.9 0.0 1 619 610 0 0 0 72.69 10:07:32 0.0 0.0 0.1 99.9 0.0 1 645 656 0 0 0 72.70 10:07:33 0.0 0.0 0.0 100.0 0.0 1 579 642 0 0 0 72.64 10:07:34 0.0 0.0 0.0 100.0 0.0 1 600 606 0 0 0 72.77 10:07:35 0.0 0.0 0.0 100.0 0.0 1 593 583 0 0 0 72.85 10:07:36 0.1 0.0 0.0 99.9 0.0 1 650 634 0 0 0 72.85 10:07:37 0.0 0.0 0.0 100.0 0.0 1 609 592 0 0 0 72.85 10:07:38 0.1 0.0 0.0 99.9 0.0 1 602 580 0 0 0 72.82 10:07:39 0.1 0.0 0.0 99.9 0.0 1 682 699 0 0 0 72.75 10:07:40 0.0 0.0 0.1 99.9 0.0 1 729 704 0 0 0 72.80 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Average 0.0 0.0 0.0 99.9 0.0 1.0 1006.1 687.3 1.5 1.4 1.6 72.17 GeoMean 0.0 0.0 0.0 99.9 0.0 1.0 736.1 655.9 0.0 0.0 0.0 72.17 StdDev 0.1 0.0 0.1 0.2 0.0 0.0 1576.6 284.0 9.7 9.1 9.6 0.62 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Minimum 0.0 0.0 0.0 98.4 0.0 1.0 553.0 518.0 0.0 0.0 0.0 71.17 Maximum 0.6 0.0 1.0 100.0 0.1 1.0 9397.0 2340.0 75.0 70.0 75.0 73.34 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Summary: CPU: 72.17 Watts on average with standard deviation 0.62 Note: power read from RAPL domains: dram, dram, package-0, package-1. These readings do not cover all the hardware in this device. C-State Resident Count Latency C6-HSW 0.000% 0 133 C3-HSW 0.000% 0 33 C1E-HSW 0.000% 0 10 C1-HSW 99.229% 60985 2 POLL 0.654% 124 0 C0 0.117%
As expected the power usage when the system is idle here is ~72W while for throughoput-performance the same was ~30W.
Also if you notice the CPU c-state always stays at C1-HSW while for throughput-performance the active CPU c-state was C6-HSW when the system was idle.
Case 2: When the system is under load
To reproduce this environment I will use "stress" tool to put some load on my CPU, memory and disks.
I will be running below command in one of the terminal to put some load on the node
# stress -c 18 -i 13 -m 3 stress: info: [5200] dispatching hogs: 18 cpu, 13 io, 3 vm, 0 hdd
Next check the power usage
# powerstat -R -c -z Running for 60.0 seconds (60 samples at 1.0 second intervals). Power measurements will start in 0 seconds time. Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts 10:11:51 58.8 0.0 41.1 0.0 0.1 33 10351 20261 1 1 5 129.77 10:11:52 56.2 0.0 43.7 0.0 0.1 32 10623 20220 0 0 0 128.95 10:11:53 55.9 0.0 44.1 0.0 0.0 35 10580 20338 0 0 0 128.79 10:11:54 56.8 0.0 43.1 0.1 0.0 35 10163 20119 0 0 0 129.28 10:11:55 57.2 0.0 42.7 0.0 0.1 35 9818 19848 0 0 0 129.88 10:11:56 57.4 0.0 42.6 0.0 0.0 35 10231 20053 0 0 0 129.86 10:11:57 55.1 0.0 44.9 0.0 0.0 35 10232 19938 0 0 0 129.36 10:11:58 55.1 0.0 44.9 0.0 0.1 35 10220 19926 0 0 0 129.20 10:11:59 56.7 0.0 43.2 0.1 0.0 35 16106 22753 4 0 0 129.59 10:12:00 55.1 0.0 44.8 0.1 0.0 35 10626 20228 0 0 0 129.72 10:12:01 54.1 0.0 45.9 0.0 0.0 34 10345 20101 2 0 3 129.55 10:12:02 54.1 0.0 45.9 0.0 0.0 35 9651 19712 0 0 1 129.78 10:12:03 54.7 0.0 45.3 0.0 0.0 35 9673 19700 0 0 0 129.59 10:12:04 55.1 0.0 44.9 0.0 0.1 36 10372 20192 0 0 0 129.66 10:12:05 56.3 0.0 43.7 0.0 0.0 33 10352 20294 0 0 0 130.25 10:12:06 57.8 0.0 42.1 0.0 0.1 35 9449 19577 0 0 0 130.90 10:12:07 57.9 0.0 42.0 0.0 0.1 35 9944 19799 0 0 0 131.01 10:12:08 55.9 0.0 43.9 0.1 0.1 28 10218 20024 0 0 0 130.53 10:12:09 54.6 0.0 45.4 0.0 0.1 35 9825 20164 1 0 1 130.14 10:12:10 54.0 0.0 46.0 0.0 0.0 35 9992 20225 0 0 0 129.88 10:12:11 53.7 0.0 46.3 0.0 0.0 35 10535 20295 0 0 0 129.64 10:12:12 53.6 0.0 46.4 0.0 0.0 35 10095 20100 0 0 0 130.04 10:12:13 55.1 0.0 44.9 0.0 0.0 35 10112 19923 0 0 0 130.79 10:12:14 55.5 0.0 44.5 0.0 0.0 35 10231 20114 0 0 0 131.04 10:12:15 54.9 0.0 45.1 0.0 0.0 35 11026 20462 0 0 0 130.56 10:12:16 56.5 0.0 43.5 0.0 0.0 34 10282 20241 0 0 0 130.94 10:12:17 57.9 0.0 42.1 0.0 0.1 35 10454 20099 0 0 0 131.68 10:12:18 58.4 0.0 41.5 0.0 0.1 35 10297 20173 0 0 0 131.85 10:12:19 54.6 0.0 45.4 0.0 0.0 35 10406 20257 0 0 0 130.47 10:12:20 54.3 0.0 45.7 0.0 0.0 35 10616 20393 0 0 0 130.86 10:12:21 55.1 0.0 44.9 0.0 0.0 35 10865 20403 0 0 0 130.87 10:12:22 55.1 0.0 44.8 0.0 0.1 35 10344 20006 0 0 0 130.90 10:12:23 54.8 0.0 45.2 0.0 0.0 35 10103 19905 0 0 0 131.05 10:12:24 54.9 0.0 45.1 0.0 0.0 35 10600 20325 0 0 0 131.05 10:12:25 56.2 0.0 43.8 0.0 0.0 35 10811 20359 0 0 0 131.50 Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts 10:12:26 55.6 0.0 44.4 0.0 0.0 34 10703 20446 0 0 0 130.99 10:12:27 55.9 0.0 44.0 0.1 0.0 34 10696 20486 0 0 0 131.49 10:12:28 56.7 0.0 43.2 0.0 0.1 35 11010 20683 0 0 0 131.85 10:12:29 55.8 0.0 44.2 0.0 0.0 35 16858 23145 3 0 1 131.48 10:12:30 55.6 0.0 44.4 0.0 0.0 35 12076 21155 1 0 0 131.51 10:12:31 55.5 0.0 44.5 0.0 0.0 35 11212 20780 0 0 0 131.57 10:12:32 54.8 0.0 45.1 0.1 0.0 27 10940 22796 0 0 0 131.57 10:12:33 57.0 0.0 42.9 0.0 0.1 35 10804 20561 0 0 0 132.34 10:12:34 55.7 0.0 44.3 0.0 0.0 35 10930 20540 0 0 0 131.40 10:12:35 57.2 0.0 42.8 0.0 0.1 35 10617 20446 0 0 0 132.26 10:12:36 55.1 0.0 44.8 0.1 0.0 31 10129 19919 0 0 0 132.80 10:12:37 55.1 0.0 44.8 0.0 0.1 34 10462 20136 0 0 0 132.04 10:12:38 55.2 0.0 44.8 0.0 0.0 37 9821 20056 0 0 0 132.07 10:12:39 54.7 0.0 45.3 0.0 0.0 35 10056 20125 0 0 0 131.71 10:12:40 53.8 0.0 46.1 0.0 0.1 35 10318 20053 0 0 0 131.87 10:12:41 55.1 0.0 44.8 0.1 0.0 35 9804 19796 0 0 0 132.06 10:12:42 55.4 0.0 44.6 0.1 0.0 35 9184 19393 0 0 0 132.49 10:12:43 54.5 0.0 45.5 0.0 0.0 35 10046 19917 0 0 0 132.02 10:12:44 54.6 0.0 45.4 0.0 0.1 35 9985 19979 0 0 0 131.96 10:12:45 57.2 0.0 42.8 0.1 0.0 38 10564 20350 15 13 3 132.70 10:12:46 55.6 0.0 44.4 0.0 0.0 35 12089 20722 60 57 72 132.09 10:12:47 55.8 0.0 44.1 0.0 0.1 35 9824 19830 1 0 0 132.35 10:12:48 54.4 0.0 45.6 0.0 0.0 34 9621 19643 0 0 0 131.95 10:12:49 56.9 0.0 42.9 0.1 0.1 35 10055 20139 1 0 1 132.79 10:12:50 55.8 0.0 44.2 0.0 0.1 35 10229 19948 11 12 5 132.34 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Average 55.6 0.0 44.3 0.0 0.0 34.6 10559.7 20292.8 1.7 1.4 1.5 130.98 GeoMean 55.6 0.0 44.3 0.0 0.0 34.5 10504.4 20282.4 0.0 0.0 0.0 130.97 StdDev 1.2 0.0 1.2 0.0 0.0 1.6 1216.1 671.7 8.0 7.6 9.2 1.08 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Minimum 53.6 0.0 41.1 0.0 0.0 27.0 9184.0 19393.0 0.0 0.0 0.0 128.79 Maximum 58.8 0.0 46.4 0.1 0.1 38.0 16858.0 23145.0 60.0 57.0 72.0 132.80 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Summary: CPU: 130.98 Watts on average with standard deviation 1.08 Note: power read from RAPL domains: dram, dram, package-0, package-1. These readings do not cover all the hardware in this device. C-State Resident Count Latency C6-HSW 0.000% 0 133 C3-HSW 0.000% 0 33 C1E-HSW 0.000% 0 10 C1-HSW 0.037% 1194 2 POLL 0.000% 5 0 C0 99.963%
So here the results are almost same as with throughput-performance, since the CPU usage was always at C0 state.
Some more examples of using powerstat
NOTE: By default if powerstat is executed without any argument then it will collect 60 samples at 1 second intervals.
Enable all sampling collection
This will give collective output with "C-State Statistics", "Average CPU Frequency", "Thermal Zone Temperatures" and "Power Histogram"
# powerstat -a -R 1 60 Running for 60.0 seconds (60 samples at 1.0 second intervals). Power measurements will start in 0 seconds time. Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts CPU Freq GPU W 11:20:40 0.1 0.0 0.0 99.9 0.0 1 728 830 0 0 0 76.13 2.60 GHz -N/A- 11:20:41 0.0 0.0 0.0 100.0 0.0 1 635 632 0 0 0 75.78 2.60 GHz -N/A- 11:20:42 0.0 0.0 0.0 100.0 0.0 1 564 563 0 0 0 75.51 2.60 GHz -N/A- 11:20:43 0.0 0.0 0.0 100.0 0.0 1 533 528 0 0 0 75.74 2.60 GHz -N/A- 11:20:44 0.0 0.0 0.1 99.9 0.0 1 666 670 0 0 0 75.84 2.60 GHz -N/A- 11:20:45 0.1 0.0 0.0 99.9 0.0 1 586 577 0 0 0 76.07 2.60 GHz -N/A- 11:20:46 0.0 0.0 0.0 100.0 0.0 1 558 544 0 0 0 75.75 2.60 GHz -N/A- 11:20:47 0.0 0.0 0.0 100.0 0.0 1 625 595 0 0 0 75.75 2.60 GHz -N/A- 11:20:48 0.0 0.0 0.0 100.0 0.0 1 603 617 0 0 0 75.98 2.60 GHz -N/A- 11:20:49 0.7 0.0 1.1 98.2 0.0 1 2368 2441 75 70 75 77.36 2.60 GHz -N/A- 11:20:50 0.1 0.0 0.1 99.8 0.0 1 940 1046 11 12 6 76.44 2.60 GHz -N/A- 11:20:51 0.0 0.0 0.0 100.0 0.0 1 678 758 1 1 5 76.04 2.60 GHz -N/A- 11:20:52 0.0 0.0 0.1 99.9 0.0 1 580 573 0 0 0 75.96 2.60 GHz -N/A- 11:20:53 0.0 0.0 0.0 100.0 0.0 1 546 541 0 0 0 76.24 2.60 GHz -N/A- 11:20:54 0.1 0.0 0.0 99.9 0.0 1 651 646 0 0 0 76.36 2.60 GHz -N/A- 11:20:55 0.0 0.0 0.0 100.0 0.0 1 594 595 0 0 0 76.05 2.60 GHz -N/A- 11:20:56 0.0 0.0 0.0 100.0 0.0 1 589 585 0 0 0 76.32 2.60 GHz -N/A- 11:20:57 0.0 0.0 0.0 100.0 0.0 1 609 579 0 0 0 76.16 2.60 GHz -N/A- 11:20:58 0.1 0.0 0.0 99.9 0.0 1 1380 596 0 0 0 75.84 2.60 GHz -N/A- 11:20:59 0.3 0.0 0.1 99.6 0.0 1 9912 1934 0 0 0 76.59 2.60 GHz -N/A- 11:21:00 0.0 0.0 0.0 100.0 0.0 1 810 741 2 0 2 76.14 2.60 GHz -N/A- 11:21:01 0.1 0.0 0.1 99.9 0.0 1 624 632 0 0 0 75.82 2.60 GHz -N/A- 11:21:02 0.0 0.0 0.0 100.0 0.0 1 561 545 0 0 1 75.98 2.60 GHz -N/A- 11:21:03 0.0 0.0 0.0 100.0 0.0 1 573 587 0 0 0 75.94 2.60 GHz -N/A- 11:21:04 0.0 0.0 0.0 100.0 0.0 1 633 642 0 0 0 75.73 2.60 GHz -N/A- 11:21:05 0.1 0.0 0.1 99.9 0.0 1 758 719 0 0 0 75.75 2.60 GHz -N/A- 11:21:06 0.0 0.0 0.0 100.0 0.0 1 614 614 0 0 0 75.87 2.60 GHz -N/A- 11:21:07 0.0 0.0 0.0 100.0 0.0 1 649 604 0 0 0 76.27 2.60 GHz -N/A- 11:21:08 0.0 0.0 0.0 100.0 0.0 1 570 571 0 0 0 75.95 2.60 GHz -N/A- 11:21:09 0.1 0.0 0.0 99.9 0.0 1 697 766 0 0 0 76.10 2.60 GHz -N/A- 11:21:10 0.1 0.0 0.0 99.9 0.0 1 708 741 1 0 1 76.20 2.60 GHz -N/A- 11:21:11 0.0 0.0 0.1 99.9 0.0 1 675 687 0 0 0 75.96 2.60 GHz -N/A- 11:21:12 0.0 0.0 0.0 100.0 0.0 1 615 612 0 0 0 75.96 2.60 GHz -N/A- 11:21:13 0.1 0.0 0.0 99.9 0.0 1 537 531 0 0 0 76.22 2.60 GHz -N/A- 11:21:14 0.0 0.0 0.0 100.0 0.0 1 603 606 0 0 0 75.96 2.60 GHz -N/A- Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts CPU Freq GPU W 11:21:15 0.0 0.0 0.0 100.0 0.0 1 571 562 0 0 0 76.38 2.60 GHz -N/A- 11:21:16 0.0 0.0 0.0 100.0 0.0 1 605 631 0 0 0 76.12 2.60 GHz -N/A- 11:21:17 0.1 0.0 0.0 99.9 0.0 1 627 596 0 0 0 76.13 2.60 GHz -N/A- 11:21:18 0.0 0.0 0.0 100.0 0.0 1 557 566 0 0 0 76.32 2.60 GHz -N/A- 11:21:19 0.0 0.0 0.0 100.0 0.0 1 643 651 0 0 0 76.15 2.60 GHz -N/A- 11:21:20 0.0 0.0 0.0 100.0 0.0 1 664 635 0 0 0 76.05 2.60 GHz -N/A- 11:21:21 0.0 0.0 0.1 99.9 0.0 1 609 620 0 0 0 76.15 2.60 GHz -N/A- 11:21:22 0.1 0.0 0.0 99.9 0.0 1 562 565 0 0 0 75.90 2.60 GHz -N/A- 11:21:23 0.0 0.0 0.0 100.0 0.0 1 556 547 0 0 0 76.20 2.60 GHz -N/A- 11:21:24 0.0 0.0 0.0 100.0 0.0 1 617 615 0 0 0 76.17 2.60 GHz -N/A- 11:21:25 0.0 0.0 0.0 100.0 0.0 1 569 560 0 0 0 76.21 2.60 GHz -N/A- 11:21:26 0.0 0.0 0.0 100.0 0.0 1 557 552 0 0 0 76.17 2.60 GHz -N/A- 11:21:27 0.1 0.0 0.0 99.9 0.0 1 875 651 0 0 0 77.02 2.60 GHz -N/A- 11:21:28 0.0 0.0 0.1 99.9 0.0 1 1458 635 0 0 0 77.37 2.60 GHz -N/A- 11:21:29 0.2 0.0 0.1 99.7 0.0 1 9915 1926 0 0 1 77.43 2.60 GHz -N/A- 11:21:30 0.0 0.0 0.1 99.9 0.0 1 720 752 1 0 1 76.88 2.60 GHz -N/A- 11:21:31 0.1 0.0 0.0 99.9 0.0 1 641 631 0 0 0 76.79 2.60 GHz -N/A- 11:21:32 0.0 0.0 0.0 99.9 0.1 1 610 587 0 0 0 76.31 2.60 GHz -N/A- 11:21:33 0.0 0.0 0.0 100.0 0.0 1 580 593 0 0 0 76.34 2.60 GHz -N/A- 11:21:34 0.0 0.0 0.0 100.0 0.0 1 587 600 0 0 0 76.54 2.60 GHz -N/A- 11:21:35 0.0 0.0 0.0 100.0 0.0 1 596 584 0 0 0 76.11 2.60 GHz -N/A- 11:21:36 0.0 0.0 0.0 100.0 0.0 1 613 670 0 0 0 76.09 2.60 GHz -N/A- 11:21:37 0.0 0.0 0.0 100.0 0.0 1 652 629 0 0 0 76.05 2.60 GHz -N/A- 11:21:38 0.1 0.0 0.0 99.9 0.0 1 550 543 0 0 0 76.30 2.60 GHz -N/A- 11:21:39 0.1 0.0 0.0 99.9 0.0 1 742 767 0 0 0 76.41 2.60 GHz -N/A- -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ --------- ------ Average 0.0 0.0 0.0 99.9 0.0 1.0 994.1 701.9 1.5 1.4 1.5 76.19 2.60 GHz -N/A- GeoMean 0.0 0.0 0.0 99.9 0.0 1.0 720.0 661.9 0.0 0.0 0.0 76.19 2.60 GHz 0.00 StdDev 0.1 0.0 0.1 0.2 0.0 0.0 1678.5 336.3 9.7 9.1 9.6 0.39 0.00 Hz -N/A- -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ --------- ------ Minimum 0.0 0.0 0.0 98.2 0.0 1.0 533.0 528.0 0.0 0.0 0.0 75.51 2.60 GHz -N/A- Maximum 0.7 0.0 1.1 100.0 0.1 1.0 9915.0 2441.0 75.0 70.0 75.0 77.43 2.60 GHz -N/A- -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ --------- ------ Summary: CPU: 76.19 Watts on average with standard deviation 0.39 GPU: 0.00 Watts on average with standard deviation 0.00 Note: power read from RAPL domains: dram, dram, package-0, package-1. These readings do not cover all the hardware in this device. Note: No thermal zones on this device. C-State Resident Count Latency C6-HSW 0.000% 0 133 C3-HSW 0.000% 0 33 C1E-HSW 0.000% 0 10 C1-HSW 99.629% 61118 2 POLL 0.251% 126 0 C0 0.120%
Histogram (of 60 power measurements)
Range (Watts) Count 75.514 - 75.705 1 ## 75.706 - 75.897 11 ######################## 75.898 - 76.089 13 ############################ 76.090 - 76.281 18 ######################################## 76.282 - 76.473 9 #################### 76.474 - 76.665 2 #### 76.666 - 76.858 1 ## 76.859 - 77.050 2 #### 77.051 - 77.242 0 77.243 - 77.434 3 ######
Histogram (of 60 CPU utilization measurements)
Range (%CPU) Count 0.000 - 0.174 56 ######################################## 0.175 - 0.349 2 # 0.350 - 0.524 1 0.525 - 0.699 0 0.700 - 0.875 0 0.876 - 1.050 0 1.051 - 1.225 0 1.226 - 1.400 0 1.401 - 1.575 0 1.576 - 1.750 1
Range is zero, cannot produce histogram of CPU average frequencies
Enable Histogram
This will give you a history of CPU power usage summary at the end of execution
# powerstat -R -H Running for 60.0 seconds (60 samples at 1.0 second intervals). Power measurements will start in 0 seconds time. Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts 11:03:14 54.7 0.0 45.2 0.1 0.1 35 10273 20083 0 0 0 130.07 11:03:15 56.2 0.0 43.8 0.0 0.0 35 10391 19978 0 0 0 130.79 11:03:16 55.0 0.0 45.0 0.0 0.0 35 10916 20086 0 0 0 129.94 11:03:17 55.6 0.0 44.4 0.0 0.0 35 10268 19892 0 0 0 130.44 11:03:18 55.2 0.0 44.6 0.1 0.1 35 10415 20122 0 0 0 130.73 11:03:19 53.9 0.0 46.1 0.0 0.0 35 10077 20076 0 0 0 129.84 11:03:20 54.2 0.0 45.8 0.0 0.0 35 10044 19819 0 0 0 130.06 11:03:21 55.0 0.0 45.0 0.0 0.0 35 9424 19463 0 0 0 130.44 11:03:22 56.0 0.0 43.7 0.1 0.3 34 10477 24073 0 0 0 130.84 11:03:23 55.5 0.0 44.4 0.0 0.1 35 9703 19740 0 0 0 130.59 11:03:24 55.3 0.0 44.6 0.1 0.0 35 10275 20064 0 0 0 130.42 11:03:25 54.6 0.0 45.4 0.0 0.0 35 9623 19753 0 0 0 130.27 11:03:26 54.9 0.0 45.0 0.0 0.1 35 9444 19531 0 0 0 130.27 11:03:27 56.9 0.0 43.1 0.0 0.0 35 10328 20064 0 0 0 130.95 11:03:28 56.8 0.0 43.2 0.1 0.0 45 13479 21303 0 0 0 130.82 11:03:29 55.2 0.0 44.8 0.0 0.0 35 13593 21262 2 0 2 130.19 11:03:30 57.6 0.0 42.3 0.0 0.1 35 10620 20209 0 0 0 131.09 11:03:31 55.3 0.0 44.7 0.0 0.0 33 10575 20194 0 0 0 130.47 11:03:32 55.9 0.0 44.1 0.0 0.0 35 10215 19983 0 0 0 130.64 11:03:33 54.8 0.0 45.0 0.1 0.1 35 10678 20224 1 0 0 130.10 11:03:34 55.3 0.0 44.7 0.0 0.0 35 10376 20045 0 0 0 130.26 11:03:35 53.8 0.0 46.2 0.0 0.0 34 10199 19931 0 0 0 129.78 11:03:36 57.4 0.0 42.6 0.0 0.0 35 10411 20188 0 0 0 131.12 11:03:37 55.8 0.0 44.2 0.1 0.0 35 10492 20164 0 0 0 130.74 11:03:38 55.3 0.0 44.7 0.0 0.0 35 10354 20118 0 0 0 130.57 11:03:39 56.2 0.0 43.8 0.1 0.0 34 10263 20001 0 0 0 130.69 11:03:40 55.0 0.0 45.0 0.0 0.0 35 10755 20312 0 0 0 130.26 11:03:41 54.1 0.0 45.8 0.0 0.1 35 10629 20274 0 0 0 129.91 11:03:42 55.0 0.0 45.0 0.0 0.0 35 10498 20084 0 0 0 130.17 11:03:43 55.2 0.0 44.8 0.0 0.0 35 10981 20251 0 0 0 130.17 11:03:44 55.0 0.0 45.0 0.0 0.0 35 10618 20277 0 0 0 130.03 11:03:45 55.6 0.0 44.3 0.1 0.0 35 10486 20196 0 0 1 130.19 11:03:46 56.9 0.0 43.0 0.1 0.1 33 10032 19766 0 0 0 131.07 11:03:47 55.3 0.0 44.7 0.0 0.0 35 10079 19938 0 0 0 130.63 11:03:48 58.6 0.0 41.2 0.1 0.1 35 10364 20035 0 0 0 131.62 Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts 11:03:49 56.9 0.0 43.0 0.1 0.0 37 11002 20277 6 1 1 130.82 11:03:50 55.4 0.0 44.6 0.0 0.1 35 11145 20446 6 11 5 129.96 11:03:51 56.4 0.0 43.6 0.1 0.0 35 9995 19933 1 1 5 130.73 11:03:52 54.2 0.0 45.7 0.1 0.1 36 10269 20055 0 0 0 129.88 11:03:53 56.7 0.0 43.3 0.0 0.0 35 10882 20435 0 0 0 130.83 11:03:54 55.4 0.0 44.6 0.0 0.0 35 10314 20226 0 0 0 130.27 11:03:55 58.2 0.0 41.8 0.0 0.0 35 10202 19926 0 0 0 131.20 11:03:56 56.1 0.0 43.8 0.0 0.1 35 10320 19916 0 0 0 130.65 11:03:57 54.7 0.0 45.2 0.0 0.1 35 10573 19918 0 0 0 130.09 11:03:58 57.3 0.0 42.6 0.1 0.1 42 13602 21503 0 0 0 130.76 11:03:59 55.9 0.0 44.1 0.0 0.0 35 13752 21402 2 0 0 130.33 11:04:00 56.6 0.0 43.4 0.0 0.0 35 11798 20576 2 0 2 130.67 11:04:01 56.6 0.0 43.3 0.1 0.1 35 10919 20352 0 0 1 130.67 11:04:02 56.2 0.0 43.7 0.0 0.1 35 10133 19869 0 0 1 130.84 11:04:03 55.7 0.0 44.1 0.1 0.1 35 10205 19979 0 0 0 130.25 11:04:04 56.3 0.0 43.7 0.0 0.1 35 9981 19789 0 0 0 130.83 11:04:05 54.5 0.0 45.4 0.1 0.0 35 10732 20289 0 0 0 129.98 11:04:06 55.2 0.0 44.8 0.0 0.0 35 9902 20018 0 0 0 130.06 11:04:07 55.0 0.0 45.0 0.0 0.0 35 10204 20154 0 0 0 129.85 11:04:08 56.4 0.0 43.6 0.0 0.0 35 10115 20042 0 0 0 130.42 11:04:09 55.3 0.0 44.6 0.0 0.1 35 10158 20099 1 0 1 130.17 11:04:10 55.3 0.0 44.6 0.0 0.1 35 10543 20257 0 0 0 130.69 11:04:11 56.0 0.0 44.0 0.0 0.0 35 10389 20045 0 0 0 130.45 11:04:12 56.8 0.0 43.1 0.0 0.1 35 10443 20042 0 0 0 130.74 11:04:13 55.8 0.0 44.2 0.0 0.0 35 10332 20128 0 0 0 130.41 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Average 55.7 0.0 44.2 0.0 0.0 35.2 10587.8 20219.6 0.3 0.2 0.3 130.46 GeoMean 55.7 0.0 44.2 0.0 0.0 35.2 10554.2 20210.5 0.0 0.0 0.0 130.46 StdDev 1.0 0.0 1.0 0.0 0.0 1.6 895.6 630.3 1.2 1.4 1.0 0.39 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Minimum 53.8 0.0 41.2 0.0 0.0 33.0 9424.0 19463.0 0.0 0.0 0.0 129.78 Maximum 58.6 0.0 46.2 0.1 0.3 45.0 13752.0 24073.0 6.0 11.0 5.0 131.62 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Summary: CPU: 130.46 Watts on average with standard deviation 0.39 Note: power read from RAPL domains: dram, dram, package-0, package-1. These readings do not cover all the hardware in this device.
Histogram (of 60 power measurements)
Range (Watts) Count 129.781 - 129.964 7 ####################### 129.965 - 130.148 7 ####################### 130.149 - 130.333 12 ######################################## 130.334 - 130.517 7 ####################### 130.518 - 130.702 9 ############################## 130.703 - 130.886 12 ######################################## 130.887 - 131.071 2 ###### 131.072 - 131.255 3 ########## 131.256 - 131.440 0 131.441 - 131.624 1 ###
Histogram (of 60 CPU utilization measurements)
Range (%CPU) Count 99.875 - 99.886 2 # 99.887 - 99.899 0 99.900 - 99.911 0 99.912 - 99.924 0 99.925 - 99.937 14 ############# 99.938 - 99.949 2 # 99.950 - 99.962 0 99.963 - 99.974 0 99.975 - 99.987 0 99.988 - 99.999 42 ########################################
Show process activity log
This will show process fork/exec/exit activity log at the end of execution
# powerstat -R -s Running for 60.0 seconds (60 samples at 1.0 second intervals). Power measurements will start in 0 seconds time. Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts 11:25:57 59.1 0.0 40.8 0.1 0.0 35 10831 20371 1 0 0 130.01 11:25:58 56.2 0.0 43.8 0.0 0.0 34 10823 20321 0 0 0 129.30 11:25:59 56.4 0.0 43.6 0.0 0.0 35 16537 23175 0 0 0 129.61 11:26:00 54.9 0.0 45.0 0.0 0.1 28 10874 20393 0 0 0 129.18 11:26:01 55.2 0.0 44.8 0.0 0.0 34 10502 20094 2 0 2 129.16 11:26:02 54.9 0.0 45.1 0.1 0.0 33 10324 20108 0 0 1 129.12 11:26:03 55.7 0.0 44.2 0.1 0.1 35 10398 20258 0 0 0 129.72 11:26:04 59.2 0.0 40.7 0.0 0.1 35 10425 20210 0 0 0 131.04 11:26:05 58.5 0.0 41.4 0.1 0.0 35 10390 20415 0 0 0 130.61 11:26:06 56.2 0.0 43.8 0.0 0.0 35 10614 20343 0 0 0 130.01 11:26:07 58.0 0.0 42.0 0.0 0.1 35 10156 19989 0 0 0 130.69 11:26:08 55.9 0.0 44.1 0.1 0.0 35 10094 19997 0 0 0 130.20 11:26:09 57.5 0.0 42.4 0.0 0.1 35 10218 20028 0 0 0 130.47 11:26:10 54.0 0.0 46.0 0.0 0.0 35 10515 20430 1 0 1 129.29 11:26:11 54.8 0.0 45.2 0.0 0.0 35 10422 20120 0 0 0 129.65 11:26:12 55.5 0.0 44.5 0.0 0.0 35 10787 20148 0 0 0 129.78 11:26:13 56.1 0.0 43.9 0.0 0.0 35 10414 20213 0 0 0 130.23 11:26:14 56.1 0.0 43.9 0.0 0.1 29 10418 20060 0 0 0 130.29 11:26:15 54.8 0.0 45.1 0.1 0.1 35 10151 19862 0 0 0 130.03 11:26:16 55.2 0.0 44.8 0.0 0.0 34 10010 19894 0 0 0 130.47 11:26:17 55.4 0.0 44.4 0.0 0.2 35 10319 19964 0 0 0 130.78 11:26:18 56.5 0.0 43.4 0.1 0.0 35 9874 19770 0 0 0 130.77 11:26:19 53.8 0.0 46.2 0.0 0.0 35 9957 19754 0 0 0 129.76 11:26:20 54.3 0.0 45.7 0.0 0.0 35 10574 20074 0 0 0 130.05 11:26:21 55.2 0.0 44.8 0.0 0.0 35 10042 19996 0 0 0 130.51 11:26:22 56.3 0.0 43.6 0.0 0.1 35 9875 19979 0 0 0 130.91 11:26:23 57.0 0.0 42.9 0.0 0.1 35 10501 20254 0 0 0 131.22 11:26:24 54.5 0.0 45.5 0.1 0.0 35 11144 20257 0 0 0 130.60 11:26:25 56.0 0.0 43.9 0.0 0.1 35 10462 19960 0 0 0 130.70 11:26:26 54.7 0.0 45.3 0.0 0.0 35 10603 20407 0 0 0 130.52 11:26:27 56.4 0.0 43.6 0.0 0.0 35 11158 20557 0 0 0 130.92 11:26:28 55.0 0.0 45.0 0.0 0.0 35 11611 20603 0 0 0 130.26 11:26:29 55.4 0.0 44.6 0.0 0.0 34 17089 23325 2 0 0 130.35 11:26:30 59.6 0.0 40.2 0.1 0.1 36 10700 20369 1 0 1 132.11 11:26:31 56.8 0.0 43.2 0.0 0.0 35 11308 20668 0 0 0 131.48 Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts 11:26:32 57.0 0.0 43.0 0.0 0.0 35 10580 20293 0 0 0 131.47 11:26:33 57.8 0.0 42.2 0.0 0.1 35 10600 20266 0 0 0 131.81 11:26:34 55.4 0.0 44.6 0.0 0.0 33 10096 20099 0 0 0 131.18 11:26:35 55.5 0.0 44.4 0.0 0.1 35 10216 19940 0 0 0 131.61 11:26:36 56.0 0.0 43.9 0.1 0.0 35 10450 20028 0 0 0 132.05 11:26:37 55.0 0.0 45.0 0.0 0.0 35 10059 20038 0 0 0 131.37 11:26:38 55.9 0.0 44.0 0.0 0.1 30 10002 19780 0 0 0 131.87 11:26:39 56.7 0.0 43.2 0.1 0.0 35 11244 20582 0 0 0 132.10 11:26:40 54.3 0.0 45.6 0.0 0.1 35 10629 20460 0 0 0 131.23 11:26:41 54.2 0.0 45.8 0.0 0.0 35 10485 20307 0 0 0 131.22 11:26:42 55.3 0.0 44.7 0.0 0.0 35 10366 20110 0 0 0 131.66 11:26:43 54.0 0.0 46.0 0.0 0.0 36 10699 20285 0 0 0 131.16 11:26:44 55.2 0.0 44.8 0.0 0.0 35 10653 20134 0 0 0 131.67 11:26:45 55.8 0.0 44.0 0.0 0.1 35 10034 19971 0 0 0 131.97 11:26:46 55.0 0.0 45.0 0.0 0.0 35 9779 19686 0 0 0 131.82 11:26:47 58.5 0.0 41.4 0.1 0.1 34 12355 19748 3 0 0 132.45 11:26:48 54.8 0.0 45.2 0.0 0.1 35 10302 20153 0 0 0 131.79 11:26:49 54.5 0.0 45.4 0.1 0.0 34 12477 20890 75 70 74 131.86 11:26:50 56.2 0.0 43.6 0.1 0.1 35 10330 20245 13 12 7 132.56 11:26:51 55.1 0.0 44.9 0.0 0.0 35 10296 20063 1 1 5 131.89 11:26:52 56.6 0.0 43.4 0.0 0.1 35 10929 20268 0 0 0 132.40 11:26:53 55.0 0.0 45.0 0.0 0.0 35 10798 20423 0 0 0 131.79 11:26:54 54.2 0.0 45.8 0.0 0.0 35 10524 20104 0 0 0 131.68 11:26:55 55.7 0.0 44.2 0.1 0.0 35 10006 19986 0 0 0 132.22 11:26:56 55.2 0.0 44.7 0.1 0.1 35 10475 20188 0 0 0 132.04 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Average 55.8 0.0 44.1 0.0 0.0 34.6 10741.7 20273.5 1.6 1.4 1.5 130.91 GeoMean 55.8 0.0 44.1 0.0 0.0 34.5 10686.1 20265.2 0.0 0.0 0.0 130.91 StdDev 1.3 0.0 1.4 0.0 0.0 1.4 1236.9 602.1 9.7 9.1 9.5 0.94 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Minimum 53.8 0.0 40.2 0.0 0.0 28.0 9779.0 19686.0 0.0 0.0 0.0 129.12 Maximum 59.6 0.0 46.2 0.1 0.2 36.0 17089.0 23325.0 75.0 70.0 74.0 132.56 -------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------ Summary: CPU: 130.91 Watts on average with standard deviation 0.94 Note: power read from RAPL domains: dram, dram, package-0, package-1. These readings do not cover all the hardware in this device.
Log of fork()/exec()/exit() calls:
11:25:56 fork: parent pid=2174 -> clone pid=19697 (ssOMAgent) 11:26:00 fork: parent pid=2174 -> clone pid=19698 (ssOMAgent) 11:26:00 fork: parent pid=2174 -> clone pid=19699 (ssOMAgent) 11:26:00 exit: pid=19698 exit_code=0 (ssOMAgent) 11:26:00 exit: pid=19699 exit_code=0 (ssOMAgent) 11:26:01 exit: pid=19627 exit_code=0 () 11:26:09 fork: parent pid=2057 -> fork pid=19700 (/usr/bin/postgres) 11:26:09 exit: pid=19700 exit_code=0 (/usr/bin/postgres) 11:26:29 fork: parent pid=4412 -> clone pid=19701 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/bin/java) 11:26:29 fork: parent pid=4412 -> clone pid=19702 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/bin/java) 11:26:29 fork: parent pid=2057 -> fork pid=19703 (/usr/bin/postgres) 11:26:29 exit: pid=19703 exit_code=0 (/usr/bin/postgres) 11:26:46 fork: parent pid=2174 -> clone pid=19704 (ssOMAgent) 11:26:46 fork: parent pid=2174 -> clone pid=19705 (ssOMAgent) 11:26:46 fork: parent pid=2174 -> clone pid=19706 (ssOMAgent) 11:26:48 fork: parent pid=1914 -> fork pid=19707 (/usr/bin/monit) 11:26:48 exec: pid=19707 (/usr/bin/monit) 11:26:48 exec: pid=19707 (/usr/bin/monit) 11:26:48 fork: parent pid=1914 -> fork pid=19708 (/usr/bin/monit) 11:26:48 fork: parent pid=19707 -> fork pid=19709 (/bin/sh) 11:26:48 exit: pid=19709 exit_code=0 (/bin/sh) 11:26:48 fork: parent pid=19707 -> fork pid=19710 (/bin/sh) 11:26:48 exec: pid=19708 (/usr/bin/monit) 11:26:48 exec: pid=19708 (/usr/bin/monit) 11:26:48 fork: parent pid=1914 -> fork pid=19711 (/usr/bin/monit) 11:26:48 fork: parent pid=19710 -> fork pid=19712 (/bin/sh) 11:26:48 fork: parent pid=19710 -> fork pid=19713 (/bin/sh) 11:26:48 fork: parent pid=19708 -> fork pid=19714 (/bin/sh) 11:26:48 exec: pid=19711 (/usr/bin/monit) 11:26:48 exec: pid=19711 (/usr/bin/monit) 11:26:48 exit: pid=19714 exit_code=0 (/bin/sh) 11:26:48 fork: parent pid=19708 -> fork pid=19715 (/bin/sh) 11:26:48 fork: parent pid=19711 -> fork pid=19716 (/bin/sh) 11:26:48 exec: pid=19713 (/bin/sh) 11:26:48 exec: pid=19712 (/bin/sh) 11:26:48 exec: pid=19712 (/bin/sh) 11:26:48 fork: parent pid=19712 -> fork pid=19717 (/bin/sh) 11:26:48 fork: parent pid=19716 -> fork pid=19718 (/bin/sh) 11:26:48 fork: parent pid=19716 -> fork pid=19719 (/bin/sh) 11:26:48 fork: parent pid=19716 -> fork pid=19720 (/bin/sh) 11:26:48 fork: parent pid=19716 -> fork pid=19721 (/bin/sh) 11:26:48 fork: parent pid=19716 -> fork pid=19722 (/bin/sh) 11:26:48 fork: parent pid=19715 -> fork pid=19723 (/bin/sh) 11:26:48 fork: parent pid=19715 -> fork pid=19724 (/bin/sh) 11:26:48 exec: pid=19722 (/bin/sh) 11:26:48 exec: pid=19723 (/bin/sh) 11:26:48 exec: pid=19723 (/bin/sh) 11:26:48 exec: pid=19721 (/bin/sh) 11:26:48 fork: parent pid=19723 -> fork pid=19725 (/bin/sh) 11:26:48 exec: pid=19717 (/bin/sh) 11:26:48 exit: pid=19717 exit_code=0 (/bin/sh) 11:26:48 fork: parent pid=19712 -> fork pid=19726 (/bin/sh) 11:26:48 exec: pid=19719 (/bin/sh) 11:26:48 exec: pid=19718 (/bin/sh) 11:26:48 exec: pid=19724 (/bin/sh) 11:26:48 exec: pid=19720 (/bin/sh) 11:26:48 exec: pid=19726 (/bin/sh) 11:26:48 exit: pid=19726 exit_code=0 (/bin/sh) 11:26:48 fork: parent pid=19712 -> fork pid=19727 (/bin/sh) 11:26:48 exec: pid=19725 (/bin/sh) 11:26:48 exit: pid=19725 exit_code=0 (/bin/sh) 11:26:48 fork: parent pid=19723 -> fork pid=19728 (/bin/sh) 11:26:48 fork: parent pid=19728 -> fork pid=19729 (/bin/sh) 11:26:48 exit: pid=19719 exit_code=0 (/bin/sh) 11:26:48 exit: pid=19720 exit_code=256 (/bin/sh) 11:26:48 exit: pid=19721 exit_code=256 (/bin/sh) 11:26:48 exit: pid=19722 exit_code=0 (/bin/sh) 11:26:48 exit: pid=19718 exit_code=0 (/bin/sh) 11:26:48 exit: pid=19716 exit_code=0 (/bin/sh) 11:26:48 fork: parent pid=19711 -> fork pid=19730 (/bin/sh) 11:26:48 exec: pid=19727 (/bin/sh) 11:26:48 exit: pid=19727 exit_code=0 (/bin/sh) 11:26:48 fork: parent pid=19712 -> fork pid=19731 (/bin/sh) 11:26:48 fork: parent pid=19731 -> fork pid=19732 (/bin/sh) 11:26:48 fork: parent pid=19731 -> fork pid=19733 (/bin/sh) 11:26:48 fork: parent pid=19730 -> fork pid=19734 (/bin/sh) 11:26:48 fork: parent pid=19730 -> fork pid=19735 (/bin/sh) 11:26:48 fork: parent pid=19730 -> fork pid=19736 (/bin/sh) 11:26:48 fork: parent pid=19730 -> fork pid=19737 (/bin/sh) 11:26:48 fork: parent pid=19730 -> fork pid=19738 (/bin/sh) 11:26:48 exec: pid=19729 (/bin/sh) 11:26:48 exit: pid=19729 exit_code=0 (/bin/sh) 11:26:48 exit: pid=19728 exit_code=0 (/bin/sh) 11:26:48 fork: parent pid=19723 -> fork pid=19739 (/bin/sh) 11:26:48 exec: pid=19735 (/bin/sh) 11:26:48 exec: pid=19732 (/bin/sh) 11:26:48 exec: pid=19736 (/bin/sh) 11:26:48 exec: pid=19733 (/bin/sh) 11:26:48 exec: pid=19738 (/bin/sh) 11:26:48 exec: pid=19739 (/bin/sh) 11:26:48 exec: pid=19737 (/bin/sh) 11:26:48 exec: pid=19734 (/bin/sh) 11:26:48 exit: pid=19733 exit_code=0 (/bin/sh) 11:26:48 exit: pid=19732 exit_code=0 (/bin/sh) 11:26:48 exit: pid=19731 exit_code=0 (/bin/sh) 11:26:48 fork: parent pid=19712 -> fork pid=19740 (/bin/sh) 11:26:48 fork: parent pid=19739 -> fork pid=19741 (su) 11:26:48 exec: pid=19740 (/bin/sh) 11:26:48 exit: pid=19740 exit_code=0 (/bin/sh) 11:26:48 exit: pid=19712 exit_code=0 (/bin/sh) 11:26:48 exit: pid=19713 exit_code=256 (/bin/sh) 11:26:48 exit: pid=19710 exit_code=256 (/bin/sh) 11:26:48 fork: parent pid=19707 -> fork pid=19742 (/bin/sh) 11:26:48 exec: pid=19741 (su) 11:26:48 exec: pid=19741 (su) 11:26:48 exec: pid=19741 (su) 11:26:48 fork: parent pid=19742 -> fork pid=19743 (/bin/sh) 11:26:48 fork: parent pid=19742 -> fork pid=19744 (/bin/sh) 11:26:48 exec: pid=19744 (/bin/sh) 11:26:48 exit: pid=19734 exit_code=0 (/bin/sh) 11:26:48 exit: pid=19735 exit_code=0 (/bin/sh) 11:26:48 exit: pid=19736 exit_code=256 (/bin/sh) 11:26:48 exit: pid=19737 exit_code=256 (/bin/sh) 11:26:48 exit: pid=19738 exit_code=0 (/bin/sh) 11:26:48 exit: pid=19730 exit_code=0 (/bin/sh) 11:26:48 fork: parent pid=19711 -> fork pid=19745 (/bin/sh) 11:26:48 exec: pid=19741 (su) 11:26:49 fork: parent pid=19745 -> fork pid=19746 (/bin/sh) 11:26:49 fork: parent pid=19745 -> fork pid=19747 (/bin/sh) 11:26:49 fork: parent pid=19745 -> fork pid=19748 (/bin/sh) 11:26:49 fork: parent pid=19745 -> fork pid=19749 (/bin/sh) 11:26:49 fork: parent pid=19745 -> fork pid=19750 (/bin/sh) 11:26:49 exec: pid=19743 (/bin/sh) 11:26:49 exec: pid=19743 (/bin/sh) 11:26:49 fork: parent pid=19743 -> fork pid=19751 (/bin/sh) 11:26:49 exec: pid=19746 (/bin/sh) 11:26:49 exec: pid=19749 (/bin/sh) 11:26:49 exec: pid=19750 (/bin/sh) 11:26:49 exec: pid=19748 (/bin/sh) 11:26:49 exec: pid=19751 (/bin/sh) 11:26:49 exec: pid=19747 (/bin/sh) 11:26:49 exit: pid=19751 exit_code=0 (/bin/sh) 11:26:49 fork: parent pid=19743 -> fork pid=19752 (/bin/sh) 11:26:49 exit: pid=19746 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19748 exit_code=256 (/bin/sh) 11:26:49 exit: pid=19747 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19749 exit_code=256 (/bin/sh) 11:26:49 exit: pid=19750 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19745 exit_code=0 (/bin/sh) 11:26:49 fork: parent pid=19711 -> fork pid=19753 (/bin/sh) 11:26:49 fork: parent pid=19753 -> fork pid=19754 (/bin/sh) 11:26:49 fork: parent pid=19753 -> fork pid=19755 (/bin/sh) 11:26:49 fork: parent pid=19753 -> fork pid=19756 (/bin/sh) 11:26:49 fork: parent pid=19753 -> fork pid=19757 (/bin/sh) 11:26:49 fork: parent pid=19753 -> fork pid=19758 (/bin/sh) 11:26:49 fork: parent pid=19741 -> fork pid=19759 (python) 11:26:49 exec: pid=19752 (/bin/sh) 11:26:49 exit: pid=19752 exit_code=0 (/bin/sh) 11:26:49 exec: pid=19754 (/bin/sh) 11:26:49 fork: parent pid=19743 -> fork pid=19760 (/bin/sh) 11:26:49 exec: pid=19756 (/bin/sh) 11:26:49 exec: pid=19758 (/bin/sh) 11:26:49 exec: pid=19757 (/bin/sh) 11:26:49 exec: pid=19759 (python) 11:26:49 fork: parent pid=19759 -> fork pid=19761 (sh) 11:26:49 exit: pid=19754 exit_code=0 (/bin/sh) 11:26:49 exec: pid=19760 (/bin/sh) 11:26:49 exit: pid=19760 exit_code=0 (/bin/sh) 11:26:49 fork: parent pid=19743 -> fork pid=19762 (/bin/sh) 11:26:49 exec: pid=19755 (/bin/sh) 11:26:49 exit: pid=19755 exit_code=256 (/bin/sh) 11:26:49 exit: pid=19756 exit_code=256 (/bin/sh) 11:26:49 exit: pid=19757 exit_code=256 (/bin/sh) 11:26:49 exit: pid=19758 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19753 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19711 exit_code=0 (/usr/bin/monit) 11:26:49 exec: pid=19761 (sh) 11:26:49 fork: parent pid=19762 -> fork pid=19763 (/bin/sh) 11:26:49 fork: parent pid=19762 -> fork pid=19764 (/bin/sh) 11:26:49 exit: pid=19761 exit_code=0 (sh) 11:26:49 exit: pid=19759 exit_code=0 (python) 11:26:49 fork: parent pid=19741 -> fork pid=19765 (python) 11:26:49 exec: pid=19764 (/bin/sh) 11:26:49 exec: pid=19765 (python) 11:26:49 fork: parent pid=19765 -> fork pid=19766 (sh) 11:26:49 exec: pid=19763 (/bin/sh) 11:26:49 exec: pid=19766 (sh) 11:26:49 exit: pid=19766 exit_code=0 (sh) 11:26:49 exit: pid=19765 exit_code=0 (python) 11:26:49 fork: parent pid=19741 -> fork pid=19767 (python) 11:26:49 exit: pid=19764 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19763 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19762 exit_code=0 (/bin/sh) 11:26:49 fork: parent pid=19743 -> fork pid=19768 (/bin/sh) 11:26:49 exec: pid=19767 (python) 11:26:49 fork: parent pid=19767 -> fork pid=19769 (sh) 11:26:49 exec: pid=19768 (/bin/sh) 11:26:49 exit: pid=19768 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19743 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19744 exit_code=256 (/bin/sh) 11:26:49 exit: pid=19742 exit_code=256 (/bin/sh) 11:26:49 exit: pid=19707 exit_code=0 (/usr/bin/monit) 11:26:49 exec: pid=19769 (sh) 11:26:49 exit: pid=19769 exit_code=0 (sh) 11:26:49 exit: pid=19767 exit_code=0 (python) 11:26:49 fork: parent pid=19741 -> fork pid=19770 (python) 11:26:49 exec: pid=19770 (python) 11:26:49 fork: parent pid=19770 -> fork pid=19771 (/bin/sh) 11:26:49 fork: parent pid=19770 -> fork pid=19772 (/bin/sh) 11:26:49 fork: parent pid=19770 -> fork pid=19773 (/bin/sh) 11:26:49 fork: parent pid=19770 -> fork pid=19774 (/bin/sh) 11:26:49 fork: parent pid=19770 -> fork pid=19775 (/bin/sh) 11:26:49 fork: parent pid=19770 -> fork pid=19776 (/bin/sh) 11:26:49 exec: pid=19775 (/bin/sh) 11:26:49 exec: pid=19774 (/bin/sh) 11:26:49 exec: pid=19772 (/bin/sh) 11:26:49 exec: pid=19771 (/bin/sh) 11:26:49 exec: pid=19776 (/bin/sh) 11:26:49 exec: pid=19773 (/bin/sh) 11:26:49 exit: pid=19772 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19773 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19774 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19771 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19775 exit_code=256 (/bin/sh) 11:26:49 exit: pid=19776 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19770 exit_code=0 (python) 11:26:49 exit: pid=19741 exit_code=0 (su) 11:26:49 exit: pid=19739 exit_code=0 (/bin/sh) 11:26:49 fork: parent pid=19723 -> fork pid=19777 (/bin/sh) 11:26:49 fork: parent pid=16907 -> fork pid=19778 (kworker/u32:0) 11:26:49 fork: parent pid=19777 -> fork pid=19779 (/bin/sh) 11:26:49 exec: pid=19778 (kworker/u32:0) 11:26:49 fork: parent pid=19779 -> fork pid=19780 (/bin/sh) 11:26:49 exec: pid=19780 (/bin/sh) 11:26:49 exit: pid=19778 exit_code=0 (kworker/u32:0) 11:26:49 exit: pid=19780 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19779 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19777 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19723 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19724 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19715 exit_code=0 (/bin/sh) 11:26:49 fork: parent pid=19708 -> fork pid=19781 (/bin/sh) 11:26:49 exec: pid=19781 (/bin/sh) 11:26:49 exit: pid=19781 exit_code=0 (/bin/sh) 11:26:49 exit: pid=19708 exit_code=0 (/usr/bin/monit) 11:26:49 fork: parent pid=2057 -> fork pid=19782 (/usr/bin/postgres) 11:26:49 exit: pid=19782 exit_code=0 (/usr/bin/postgres) 11:26:49 fork: parent pid=2162 -> clone pid=19783 (ssProbeframework) 11:26:49 fork: parent pid=2162 -> clone pid=19784 (ssProbeframework) 11:26:49 fork: parent pid=2162 -> fork pid=19785 (ssProbeframework) 11:26:49 fork: parent pid=2162 -> fork pid=19786 (ssProbeframework) 11:26:49 exec: pid=19786 (ssProbeframework) 11:26:49 exec: pid=19786 (ssProbeframework) 11:26:49 fork: parent pid=19786 -> fork pid=19787 (/bin/sh) 11:26:49 exec: pid=19785 (ssProbeframework) 11:26:49 fork: parent pid=2162 -> clone pid=19788 (ssProbeframework) 11:26:49 fork: parent pid=19785 -> fork pid=19789 (/bin/sh) 11:26:50 exec: pid=19789 (/bin/sh) 11:26:50 exit: pid=19789 exit_code=0 (/bin/sh) 11:26:50 fork: parent pid=19785 -> fork pid=19790 (/bin/sh) 11:26:50 fork: parent pid=19785 -> fork pid=19791 (/bin/sh) 11:26:50 exec: pid=19791 (/bin/sh) 11:26:50 exec: pid=19791 (/bin/sh) 11:26:50 exec: pid=19791 (/bin/sh) 11:26:50 exec: pid=19790 (/bin/sh) 11:26:50 exit: pid=19790 exit_code=0 (/bin/sh) 11:26:50 exec: pid=19787 (/bin/sh) 11:26:50 exit: pid=19787 exit_code=0 (/bin/sh) 11:26:50 fork: parent pid=19786 -> fork pid=19792 (/bin/sh) 11:26:50 fork: parent pid=19786 -> fork pid=19793 (/bin/sh) 11:26:50 fork: parent pid=19786 -> fork pid=19794 (/bin/sh) 11:26:50 exit: pid=19791 exit_code=0 (/bin/sh) 11:26:50 exit: pid=19785 exit_code=0 (ssProbeframework) 11:26:50 exec: pid=19793 (/bin/sh) 11:26:50 exec: pid=19792 (/bin/sh) 11:26:50 exec: pid=19794 (/bin/sh) 11:26:51 exit: pid=19792 exit_code=0 (/bin/sh) 11:26:51 exit: pid=19793 exit_code=0 (/bin/sh) 11:26:51 exit: pid=19794 exit_code=0 (/bin/sh) 11:26:51 fork: parent pid=19786 -> fork pid=19795 (/bin/sh) 11:26:51 exec: pid=19795 (/bin/sh) 11:26:51 exit: pid=19795 exit_code=0 (/bin/sh) 11:26:51 exit: pid=19786 exit_code=0 (ssProbeframework)
I hope the article was useful.