Merge tag 'pm-4.18-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm

Pull power management updates from Rafael Wysocki:
 "These include a significant update of the generic power domains
  (genpd) and Operating Performance Points (OPP) frameworks, mostly
  related to the introduction of power domain performance levels,
  cpufreq updates (new driver for Qualcomm Kryo processors, updates of
  the existing drivers, some core fixes, schedutil governor
  improvements), PCI power management fixes, ACPI workaround for
  EC-based wakeup events handling on resume from suspend-to-idle, and
  major updates of the turbostat and pm-graph utilities.

  Specifics:

   - Introduce power domain performance levels into the the generic
     power domains (genpd) and Operating Performance Points (OPP)
     frameworks (Viresh Kumar, Rajendra Nayak, Dan Carpenter).

   - Fix two issues in the runtime PM framework related to the
     initialization and removal of devices using device links (Ulf
     Hansson).

   - Clean up the initialization of drivers for devices in PM domains
     (Ulf Hansson, Geert Uytterhoeven).

   - Fix a cpufreq core issue related to the policy sysfs interface
     causing CPU online to fail for CPUs sharing one cpufreq policy in
     some situations (Tao Wang).

   - Make it possible to use platform-specific suspend/resume hooks in
     the cpufreq-dt driver and make the Armada 37xx DVFS use that
     feature (Viresh Kumar, Miquel Raynal).

   - Optimize policy transition notifications in cpufreq (Viresh Kumar).

   - Improve the iowait boost mechanism in the schedutil cpufreq
     governor (Patrick Bellasi).

   - Improve the handling of deferred frequency updates in the schedutil
     cpufreq governor (Joel Fernandes, Dietmar Eggemann, Rafael Wysocki,
     Viresh Kumar).

   - Add a new cpufreq driver for Qualcomm Kryo (Ilia Lin).

   - Fix and clean up some cpufreq drivers (Colin Ian King, Dmitry
     Osipenko, Doug Smythies, Luc Van Oostenryck, Simon Horman, Viresh
     Kumar).

   - Fix the handling of PCI devices with the DPM_SMART_SUSPEND flag set
     and update stale comments in the PCI core PM code (Rafael Wysocki).

   - Work around an issue related to the handling of EC-based wakeup
     events in the ACPI PM core during resume from suspend-to-idle if
     the EC has been put into the low-power mode (Rafael Wysocki).

   - Improve the handling of wakeup source objects in the PM core (Doug
     Berger, Mahendran Ganesh, Rafael Wysocki).

   - Update the driver core to prevent deferred probe from breaking
     suspend/resume ordering (Feng Kan).

   - Clean up the PM core somewhat (Bjorn Helgaas, Ulf Hansson, Rafael
     Wysocki).

   - Make the core suspend/resume code and cpufreq support the RT patch
     (Sebastian Andrzej Siewior, Thomas Gleixner).

   - Consolidate the PM QoS handling in cpuidle governors (Rafael
     Wysocki).

   - Fix a possible crash in the hibernation core (Tetsuo Handa).

   - Update the rockchip-io Adaptive Voltage Scaling (AVS) driver (David
     Wu).

   - Update the turbostat utility (fixes, cleanups, new CPU IDs, new
     command line options, built-in "Low Power Idle" counters support,
     new POLL and POLL% columns) and add an entry for it to MAINTAINERS
     (Len Brown, Artem Bityutskiy, Chen Yu, Laura Abbott, Matt Turner,
     Prarit Bhargava, Srinivas Pandruvada).

   - Update the pm-graph to version 5.1 (Todd Brandt).

   - Update the intel_pstate_tracer utility (Doug Smythies)"

* tag 'pm-4.18-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm: (128 commits)
  tools/power turbostat: update version number
  tools/power turbostat: Add Node in output
  tools/power turbostat: add node information into turbostat calculations
  tools/power turbostat: remove num_ from cpu_topology struct
  tools/power turbostat: rename num_cores_per_pkg to num_cores_per_node
  tools/power turbostat: track thread ID in cpu_topology
  tools/power turbostat: Calculate additional node information for a package
  tools/power turbostat: Fix node and siblings lookup data
  tools/power turbostat: set max_num_cpus equal to the cpumask length
  tools/power turbostat: if --num_iterations, print for specific number of iterations
  tools/power turbostat: Add Cannon Lake support
  tools/power turbostat: delete duplicate #defines
  x86: msr-index.h: Correct SNB_C1/C3_AUTO_UNDEMOTE defines
  tools/power turbostat: Correct SNB_C1/C3_AUTO_UNDEMOTE defines
  tools/power turbostat: add POLL and POLL% column
  tools/power turbostat: Fix --hide Pk%pc10
  tools/power turbostat: Build-in "Low Power Idle" counters support
  tools/power turbostat: Don't make man pages executable
  tools/power turbostat: remove blank lines
  tools/power turbostat: a small C-states dump readability immprovement
  ...

8 files changed, 973 insertions, 424 deletions

diff --git a/tools/power/pm-graph/bootgraph.py b/tools/power/pm-graph/bootgraph.py
index abb4c38f029b..8ee626c0f6a5 100755
--- a/tools/power/pm-graph/bootgraph.py
+++ b/tools/power/pm-graph/bootgraph.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/python2
 #
 # Tool for analyzing boot timing
 # Copyright (c) 2013, Intel Corporation.
diff --git a/tools/power/pm-graph/sleepgraph.8 b/tools/power/pm-graph/sleepgraph.8
index 18baaf6300c9..070be2cf7f74 100644
--- a/tools/power/pm-graph/sleepgraph.8
+++ b/tools/power/pm-graph/sleepgraph.8
@@ -168,6 +168,7 @@ Create a summary page of all tests in \fIindir\fR. Creates summary.html
 in the current folder. The output page is a table of tests with
 suspend and resume values sorted by suspend mode, host, and kernel.
 Includes test averages by mode and links to the test html files.
+Use -genhtml to include tests with missing html.
 .TP
 \fB-modes\fR
 List available suspend modes.
@@ -179,6 +180,9 @@ with any options you intend to use to see if they will work.
 \fB-fpdt\fR
 Print out the contents of the ACPI Firmware Performance Data Table.
 .TP
+\fB-battery\fR
+Print out battery status and current charge.
+.TP
 \fB-sysinfo\fR
 Print out system info extracted from BIOS. Reads /dev/mem directly instead of going through dmidecode.
 .TP
diff --git a/tools/power/pm-graph/sleepgraph.py b/tools/power/pm-graph/sleepgraph.py
index 266409fb27ae..0c760478f7d7 100755
--- a/tools/power/pm-graph/sleepgraph.py
+++ b/tools/power/pm-graph/sleepgraph.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/python2
 #
 # Tool for analyzing suspend/resume timing
 # Copyright (c) 2013, Intel Corporation.
@@ -69,7 +69,7 @@ from subprocess import call, Popen, PIPE
 #        store system values and test parameters
 class SystemValues:
         title = 'SleepGraph'
-        version = '5.0'
+        version = '5.1'
         ansi = False
         rs = 0
         display = 0
@@ -240,7 +240,7 @@ class SystemValues:
         kprobes = dict()
         timeformat = '%.3f'
         cmdline = '%s %s' % \
-                        (os.path.basename(sys.argv[0]), string.join(sys.argv[1:], ' '))
+                        (os.path.basename(sys.argv[0]), ' '.join(sys.argv[1:]))
         def __init__(self):
                 self.archargs = 'args_'+platform.machine()
                 self.hostname = platform.node()
@@ -917,12 +917,18 @@ class Data:
                         self.devicegroups.append([phase])
                 self.errorinfo = {'suspend':[],'resume':[]}
         def extractErrorInfo(self):
+                elist = {
+                        'HWERROR' : '.*\[ *Hardware Error *\].*',
+                        'FWBUG'   : '.*\[ *Firmware Bug *\].*',
+                        'BUG'     : '.*BUG.*',
+                        'ERROR'   : '.*ERROR.*',
+                        'WARNING' : '.*WARNING.*',
+                        'IRQ'     : '.*genirq: .*',
+                        'TASKFAIL': '.*Freezing of tasks failed.*',
+                }
                 lf = sysvals.openlog(sysvals.dmesgfile, 'r')
                 i = 0
                 list = []
-                # sl = start line, et = error time, el = error line
-                type = 'ERROR'
-                sl = et = el = -1
                 for line in lf:
                         i += 1
                         m = re.match('[ \t]*(\[ *)(?P<ktime>[0-9\.]*)(\]) (?P<msg>.*)', line)
@@ -931,43 +937,13 @@ class Data:
                         t = float(m.group('ktime'))
                         if t < self.start or t > self.end:
                                 continue
-                        if t < self.tSuspended:
-                                dir = 'suspend'
-                        else:
-                                dir = 'resume'
+                        dir = 'suspend' if t < self.tSuspended else 'resume'
                         msg = m.group('msg')
-                        if re.match('-*\[ *cut here *\]-*', msg):
-                                type = 'WARNING'
-                                sl = i
-                        elif re.match('genirq: .*', msg):
-                                type = 'IRQ'
-                                sl = i
-                        elif re.match('BUG: .*', msg) or re.match('kernel BUG .*', msg):
-                                type = 'BUG'
-                                sl = i
-                        elif re.match('-*\[ *end trace .*\]-*', msg) or \
-                                re.match('R13: .*', msg):
-                                if et >= 0 and sl >= 0:
-                                        list.append((type, dir, et, sl, i))
-                                        self.kerror = True
-                                        sl = et = el = -1
-                                        type = 'ERROR'
-                        elif 'Call Trace:' in msg:
-                                if el >= 0 and et >= 0:
-                                        list.append((type, dir, et, el, el))
+                        for err in elist:
+                                if re.match(elist[err], msg):
+                                        list.append((err, dir, t, i, i))
                                         self.kerror = True
-                                et, el = t, i
-                                if sl < 0 or type == 'BUG':
-                                        slval = i
-                                        if sl >= 0:
-                                                slval = sl
-                                        list.append((type, dir, et, slval, i))
-                                        self.kerror = True
-                                        sl = et = el = -1
-                                        type = 'ERROR'
-                if el >= 0 and et >= 0:
-                        list.append((type, dir, et, el, el))
-                        self.kerror = True
+                                        break
                 for e in list:
                         type, dir, t, idx1, idx2 = e
                         sysvals.vprint('kernel %s found in %s at %f' % (type, dir, t))
@@ -2331,12 +2307,14 @@ class TestProps:
                 sv.suspendmode = data.stamp['mode']
                 if sv.suspendmode == 'command' and sv.ftracefile != '':
                         modes = ['on', 'freeze', 'standby', 'mem', 'disk']
-                        out = Popen(['grep', 'machine_suspend', sv.ftracefile],
-                                stderr=PIPE, stdout=PIPE).stdout.read()
-                        m = re.match('.* machine_suspend\[(?P<mode>.*)\]', out)
-                        if m and m.group('mode') in ['1', '2', '3', '4']:
-                                sv.suspendmode = modes[int(m.group('mode'))]
-                                data.stamp['mode'] = sv.suspendmode
+                        fp = sysvals.openlog(sv.ftracefile, 'r')
+                        for line in fp:
+                                m = re.match('.* machine_suspend\[(?P<mode>.*)\]', line)
+                                if m and m.group('mode') in ['1', '2', '3', '4']:
+                                        sv.suspendmode = modes[int(m.group('mode'))]
+                                        data.stamp['mode'] = sv.suspendmode
+                                        break
+                        fp.close()
                 m = re.match(self.cmdlinefmt, self.cmdline)
                 if m:
                         sv.cmdline = m.group('cmd')
@@ -2413,7 +2391,7 @@ class ProcessMonitor:
 #        markers, and/or kprobes required for primary parsing.
 def doesTraceLogHaveTraceEvents():
         kpcheck = ['_cal: (', '_cpu_down()']
-        techeck = sysvals.traceevents[:]
+        techeck = ['suspend_resume']
         tmcheck = ['SUSPEND START', 'RESUME COMPLETE']
         sysvals.usekprobes = False
         fp = sysvals.openlog(sysvals.ftracefile, 'r')
@@ -2808,7 +2786,7 @@ def parseTraceLog(live=False):
                                 # -- phase changes --
                                 # start of kernel suspend
                                 if(re.match('suspend_enter\[.*', t.name)):
-                                        if(isbegin):
+                                        if(isbegin and data.start == data.tKernSus):
                                                 data.dmesg[phase]['start'] = t.time
                                                 data.tKernSus = t.time
                                         continue
@@ -3072,13 +3050,20 @@ def parseTraceLog(live=False):
                                         sysvals.vprint('Callgraph found for task %d: %.3fms, %s' % (cg.pid, (cg.end - cg.start)*1000, name))
                                         cg.newActionFromFunction(data)
         if sysvals.suspendmode == 'command':
-                return testdata
+                return (testdata, '')
 
         # fill in any missing phases
+        error = []
         for data in testdata:
+                tn = '' if len(testdata) == 1 else ('%d' % (data.testnumber + 1))
+                terr = ''
                 lp = data.phases[0]
                 for p in data.phases:
                         if(data.dmesg[p]['start'] < 0 and data.dmesg[p]['end'] < 0):
+                                if not terr:
+                                        print 'TEST%s FAILED: %s failed in %s phase' % (tn, sysvals.suspendmode, lp)
+                                        terr = '%s%s failed in %s phase' % (sysvals.suspendmode, tn, lp)
+                                        error.append(terr)
                                 sysvals.vprint('WARNING: phase "%s" is missing!' % p)
                         if(data.dmesg[p]['start'] < 0):
                                 data.dmesg[p]['start'] = data.dmesg[lp]['end']
@@ -3106,7 +3091,7 @@ def parseTraceLog(live=False):
                         for j in range(i + 1, tc):
                                 testdata[j].mergeOverlapDevices(devlist)
                 testdata[0].stitchTouchingThreads(testdata[1:])
-        return testdata
+        return (testdata, ', '.join(error))
 
 # Function: loadKernelLog
 # Description:
@@ -3173,7 +3158,7 @@ def loadKernelLog():
         if data:
                 testruns.append(data)
         if len(testruns) < 1:
-                doError(' dmesg log has no suspend/resume data: %s' \
+                print('ERROR: dmesg log has no suspend/resume data: %s' \
                         % sysvals.dmesgfile)
 
         # fix lines with same timestamp/function with the call and return swapped
@@ -3521,68 +3506,144 @@ def createHTMLSummarySimple(testruns, htmlfile, folder):
                 .summary {border:1px solid;}\n\
                 th {border: 1px solid black;background:#222;color:white;}\n\
                 td {font: 16px "Times New Roman";text-align: center;}\n\
-                tr.alt td {background:#ddd;}\n\
-                tr.avg td {background:#aaa;}\n\
+                tr.head td {border: 1px solid black;background:#aaa;}\n\
+                tr.alt {background-color:#ddd;}\n\
+                tr.notice {color:red;}\n\
+                .minval {background-color:#BBFFBB;}\n\
+                .medval {background-color:#BBBBFF;}\n\
+                .maxval {background-color:#FFBBBB;}\n\
+                .head a {color:#000;text-decoration: none;}\n\
         </style>\n</head>\n<body>\n'
 
+        # extract the test data into list
+        list = dict()
+        tAvg, tMin, tMax, tMed = [0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [[], []]
+        iMin, iMed, iMax = [0, 0], [0, 0], [0, 0]
+        num = 0
+        lastmode = ''
+        cnt = {'pass':0, 'fail':0, 'hang':0}
+        for data in sorted(testruns, key=lambda v:(v['mode'], v['host'], v['kernel'], v['time'])):
+                mode = data['mode']
+                if mode not in list:
+                        list[mode] = {'data': [], 'avg': [0,0], 'min': [0,0], 'max': [0,0], 'med': [0,0]}
+                if lastmode and lastmode != mode and num > 0:
+                        for i in range(2):
+                                s = sorted(tMed[i])
+                                list[lastmode]['med'][i] = s[int(len(s)/2)]
+                                iMed[i] = tMed[i].index(list[lastmode]['med'][i])
+                        list[lastmode]['avg'] = [tAvg[0] / num, tAvg[1] / num]
+                        list[lastmode]['min'] = tMin
+                        list[lastmode]['max'] = tMax
+                        list[lastmode]['idx'] = (iMin, iMed, iMax)
+                        tAvg, tMin, tMax, tMed = [0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [[], []]
+                        iMin, iMed, iMax = [0, 0], [0, 0], [0, 0]
+                        num = 0
+                tVal = [float(data['suspend']), float(data['resume'])]
+                list[mode]['data'].append([data['host'], data['kernel'],
+                        data['time'], tVal[0], tVal[1], data['url'], data['result'],
+                        data['issues']])
+                idx = len(list[mode]['data']) - 1
+                if data['result'] == 'pass':
+                        cnt['pass'] += 1
+                        for i in range(2):
+                                tMed[i].append(tVal[i])
+                                tAvg[i] += tVal[i]
+                                if tMin[i] == 0 or tVal[i] < tMin[i]:
+                                        iMin[i] = idx
+                                        tMin[i] = tVal[i]
+                                if tMax[i] == 0 or tVal[i] > tMax[i]:
+                                        iMax[i] = idx
+                                        tMax[i] = tVal[i]
+                        num += 1
+                elif data['result'] == 'hang':
+                        cnt['hang'] += 1
+                elif data['result'] == 'fail':
+                        cnt['fail'] += 1
+                lastmode = mode
+        if lastmode and num > 0:
+                for i in range(2):
+                        s = sorted(tMed[i])
+                        list[lastmode]['med'][i] = s[int(len(s)/2)]
+                        iMed[i] = tMed[i].index(list[lastmode]['med'][i])
+                list[lastmode]['avg'] = [tAvg[0] / num, tAvg[1] / num]
+                list[lastmode]['min'] = tMin
+                list[lastmode]['max'] = tMax
+                list[lastmode]['idx'] = (iMin, iMed, iMax)
+
         # group test header
-        html += '<div class="stamp">%s (%d tests)</div>\n' % (folder, len(testruns))
+        desc = []
+        for ilk in sorted(cnt, reverse=True):
+                if cnt[ilk] > 0:
+                        desc.append('%d %s' % (cnt[ilk], ilk))
+        html += '<div class="stamp">%s (%d tests: %s)</div>\n' % (folder, len(testruns), ', '.join(desc))
         th = '\t<th>{0}</th>\n'
         td = '\t<td>{0}</td>\n'
+        tdh = '\t<td{1}>{0}</td>\n'
         tdlink = '\t<td><a href="{0}">html</a></td>\n'
 
         # table header
         html += '<table class="summary">\n<tr>\n' + th.format('#') +\
                 th.format('Mode') + th.format('Host') + th.format('Kernel') +\
-                th.format('Test Time') + th.format('Suspend') + th.format('Resume') +\
-                th.format('Detail') + '</tr>\n'
-
-        # test data, 1 row per test
-        avg = '<tr class="avg"><td></td><td></td><td></td><td></td>'+\
-                '<td>Average of {0} {1} tests</td><td>{2}</td><td>{3}</td><td></td></tr>\n'
-        sTimeAvg = rTimeAvg = 0.0
-        mode = ''
-        num = 0
-        for data in sorted(testruns, key=lambda v:(v['mode'], v['host'], v['kernel'], v['time'])):
-                if mode != data['mode']:
-                        # test average line
-                        if(num > 0):
-                                sTimeAvg /= (num - 1)
-                                rTimeAvg /= (num - 1)
-                                html += avg.format('%d' % (num - 1), mode,
-                                        '%3.3f ms' % sTimeAvg, '%3.3f ms' % rTimeAvg)
-                        sTimeAvg = rTimeAvg = 0.0
-                        mode = data['mode']
-                        num = 1
-                # alternate row color
-                if num % 2 == 1:
-                        html += '<tr class="alt">\n'
+                th.format('Test Time') + th.format('Result') + th.format('Issues') +\
+                th.format('Suspend') + th.format('Resume') + th.format('Detail') + '</tr>\n'
+
+        # export list into html
+        head = '<tr class="head"><td>{0}</td><td>{1}</td>'+\
+                '<td colspan=8 class="sus">Suspend Avg={2} '+\
+                '<span class=minval><a href="#s{10}min">Min={3}</a></span> '+\
+                '<span class=medval><a href="#s{10}med">Med={4}</a></span> '+\
+                '<span class=maxval><a href="#s{10}max">Max={5}</a></span> '+\
+                'Resume Avg={6} '+\
+                '<span class=minval><a href="#r{10}min">Min={7}</a></span> '+\
+                '<span class=medval><a href="#r{10}med">Med={8}</a></span> '+\
+                '<span class=maxval><a href="#r{10}max">Max={9}</a></span></td>'+\
+                '</tr>\n'
+        headnone = '<tr class="head"><td>{0}</td><td>{1}</td><td colspan=8></td></tr>\n'
+        for mode in list:
+                # header line for each suspend mode
+                num = 0
+                tAvg, tMin, tMax, tMed = list[mode]['avg'], list[mode]['min'],\
+                        list[mode]['max'], list[mode]['med']
+                count = len(list[mode]['data'])
+                if 'idx' in list[mode]:
+                        iMin, iMed, iMax = list[mode]['idx']
+                        html += head.format('%d' % count, mode.upper(),
+                                '%.3f' % tAvg[0], '%.3f' % tMin[0], '%.3f' % tMed[0], '%.3f' % tMax[0],
+                                '%.3f' % tAvg[1], '%.3f' % tMin[1], '%.3f' % tMed[1], '%.3f' % tMax[1],
+                                mode.lower()
+                        )
                 else:
-                        html += '<tr>\n'
-                html += td.format("%d" % num)
-                num += 1
-                # basic info
-                for item in ['mode', 'host', 'kernel', 'time']:
-                        val = "unknown"
-                        if(item in data):
-                                val = data[item]
-                        html += td.format(val)
-                # suspend time
-                sTime = float(data['suspend'])
-                sTimeAvg += sTime
-                html += td.format('%.3f ms' % sTime)
-                # resume time
-                rTime = float(data['resume'])
-                rTimeAvg += rTime
-                html += td.format('%.3f ms' % rTime)
-                # link to the output html
-                html += tdlink.format(data['url']) + '</tr>\n'
-        # last test average line
-        if(num > 0):
-                sTimeAvg /= (num - 1)
-                rTimeAvg /= (num - 1)
-                html += avg.format('%d' % (num - 1), mode,
-                        '%3.3f ms' % sTimeAvg, '%3.3f ms' % rTimeAvg)
+                        iMin = iMed = iMax = [-1, -1, -1]
+                        html += headnone.format('%d' % count, mode.upper())
+                for d in list[mode]['data']:
+                        # row classes - alternate row color
+                        rcls = ['alt'] if num % 2 == 1 else []
+                        if d[6] != 'pass':
+                                rcls.append('notice')
+                        html += '<tr class="'+(' '.join(rcls))+'">\n' if len(rcls) > 0 else '<tr>\n'
+                        # figure out if the line has sus or res highlighted
+                        idx = list[mode]['data'].index(d)
+                        tHigh = ['', '']
+                        for i in range(2):
+                                tag = 's%s' % mode if i == 0 else 'r%s' % mode
+                                if idx == iMin[i]:
+                                        tHigh[i] = ' id="%smin" class=minval title="Minimum"' % tag
+                                elif idx == iMax[i]:
+                                        tHigh[i] = ' id="%smax" class=maxval title="Maximum"' % tag
+                                elif idx == iMed[i]:
+                                        tHigh[i] = ' id="%smed" class=medval title="Median"' % tag
+                        html += td.format("%d" % (list[mode]['data'].index(d) + 1)) # row
+                        html += td.format(mode)                                                                         # mode
+                        html += td.format(d[0])                                                                         # host
+                        html += td.format(d[1])                                                                         # kernel
+                        html += td.format(d[2])                                                                         # time
+                        html += td.format(d[6])                                                                         # result
+                        html += td.format(d[7])                                                                         # issues
+                        html += tdh.format('%.3f ms' % d[3], tHigh[0]) if d[3] else td.format('')       # suspend
+                        html += tdh.format('%.3f ms' % d[4], tHigh[1]) if d[4] else td.format('')       # resume
+                        html += tdlink.format(d[5]) if d[5] else td.format('')          # url
+                        html += '</tr>\n'
+                        num += 1
 
         # flush the data to file
         hf = open(htmlfile, 'w')
@@ -3607,7 +3668,7 @@ def ordinal(value):
 #        testruns: array of Data objects from parseKernelLog or parseTraceLog
 # Output:
 #        True if the html file was created, false if it failed
-def createHTML(testruns):
+def createHTML(testruns, testfail):
         if len(testruns) < 1:
                 print('ERROR: Not enough test data to build a timeline')
                 return
@@ -3641,6 +3702,7 @@ def createHTML(testruns):
                 '<td class="purple">{4}Firmware Resume: {2} ms</td>'\
                 '<td class="yellow" title="time from firmware mode to return from kernel enter_state({5}) [kernel time only]">{4}Kernel Resume: {3} ms</td>'\
                 '</tr>\n</table>\n'
+        html_fail = '<table class="testfail"><tr><td>{0}</td></tr></table>\n'
 
         # html format variables
         scaleH = 20
@@ -3708,6 +3770,9 @@ def createHTML(testruns):
                                         resume_time, testdesc, stitle, rtitle)
                         devtl.html += thtml
 
+        if testfail:
+                devtl.html += html_fail.format(testfail)
+
         # time scale for potentially multiple datasets
         t0 = testruns[0].start
         tMax = testruns[-1].end
@@ -4006,6 +4071,7 @@ def addCSS(hf, sv, testcount=1, kerror=False, extra=''):
                 .blue {background:rgba(169,208,245,0.4);}\n\
                 .time1 {font:22px Arial;border:1px solid;}\n\
                 .time2 {font:15px Arial;border-bottom:1px solid;border-left:1px solid;border-right:1px solid;}\n\
+                .testfail {font:bold 22px Arial;color:red;border:1px dashed;}\n\
                 td {text-align:center;}\n\
                 r {color:#500000;font:15px Tahoma;}\n\
                 n {color:#505050;font:15px Tahoma;}\n\
@@ -4927,6 +4993,25 @@ def dmidecode(mempath, fatal=False):
                 count += 1
         return out
 
+def getBattery():
+        p = '/sys/class/power_supply'
+        bat = dict()
+        for d in os.listdir(p):
+                type = sysvals.getVal(os.path.join(p, d, 'type')).strip().lower()
+                if type != 'battery':
+                        continue
+                for v in ['status', 'energy_now', 'capacity_now']:
+                        bat[v] = sysvals.getVal(os.path.join(p, d, v)).strip().lower()
+                break
+        ac = True
+        if 'status' in bat and 'discharging' in bat['status']:
+                ac = False
+        charge = 0
+        for v in ['energy_now', 'capacity_now']:
+                if v in bat and bat[v]:
+                        charge = int(bat[v])
+        return (ac, charge)
+
 # Function: getFPDT
 # Description:
 #        Read the acpi bios tables and pull out FPDT, the firmware data
@@ -5202,8 +5287,9 @@ def getArgFloat(name, args, min, max, main=True):
 
 def processData(live=False):
         print('PROCESSING DATA')
+        error = ''
         if(sysvals.usetraceevents):
-                testruns = parseTraceLog(live)
+                testruns, error = parseTraceLog(live)
                 if sysvals.dmesgfile:
                         for data in testruns:
                                 data.extractErrorInfo()
@@ -5220,15 +5306,18 @@ def processData(live=False):
                 for data in testruns:
                         data.debugPrint()
                 sys.exit()
-
+        if len(testruns) < 1:
+                return (testruns, {'error': 'timeline generation failed'})
         sysvals.vprint('Creating the html timeline (%s)...' % sysvals.htmlfile)
-        createHTML(testruns)
+        createHTML(testruns, error)
         print('DONE')
         data = testruns[0]
         stamp = data.stamp
         stamp['suspend'], stamp['resume'] = data.getTimeValues()
         if data.fwValid:
                 stamp['fwsuspend'], stamp['fwresume'] = data.fwSuspend, data.fwResume
+        if error:
+                stamp['error'] = error
         return (testruns, stamp)
 
 # Function: rerunTest
@@ -5268,58 +5357,88 @@ def runTest(n=0):
         sysvals.sudouser(sysvals.testdir)
         sysvals.outputResult(stamp, n)
 
-def find_in_html(html, strs, div=False):
-        for str in strs:
-                l = len(str)
-                i = html.find(str)
-                if i >= 0:
+def find_in_html(html, start, end, firstonly=True):
+        n, out = 0, []
+        while n < len(html):
+                m = re.search(start, html[n:])
+                if not m:
                         break
-        if i < 0:
-                return ''
-        if not div:
-                return re.search(r'[-+]?\d*\.\d+|\d+', html[i+l:i+l+50]).group()
-        n = html[i+l:].find('</div>')
-        if n < 0:
+                i = m.end()
+                m = re.search(end, html[n+i:])
+                if not m:
+                        break
+                j = m.start()
+                str = html[n+i:n+i+j]
+                if end == 'ms':
+                        num = re.search(r'[-+]?\d*\.\d+|\d+', str)
+                        str = num.group() if num else 'NaN'
+                if firstonly:
+                        return str
+                out.append(str)
+                n += i+j
+        if firstonly:
                 return ''
-        return html[i+l:i+l+n]
+        return out
 
 # Function: runSummary
 # Description:
 #        create a summary of tests in a sub-directory
-def runSummary(subdir, local=True):
+def runSummary(subdir, local=True, genhtml=False):
         inpath = os.path.abspath(subdir)
         outpath = inpath
         if local:
                 outpath = os.path.abspath('.')
         print('Generating a summary of folder "%s"' % inpath)
+        if genhtml:
+                for dirname, dirnames, filenames in os.walk(subdir):
+                        sysvals.dmesgfile = sysvals.ftracefile = sysvals.htmlfile = ''
+                        for filename in filenames:
+                                if(re.match('.*_dmesg.txt', filename)):
+                                        sysvals.dmesgfile = os.path.join(dirname, filename)
+                                elif(re.match('.*_ftrace.txt', filename)):
+                                        sysvals.ftracefile = os.path.join(dirname, filename)
+                        sysvals.setOutputFile()
+                        if sysvals.ftracefile and sysvals.htmlfile and \
+                                not os.path.exists(sysvals.htmlfile):
+                                print('FTRACE: %s' % sysvals.ftracefile)
+                                if sysvals.dmesgfile:
+                                        print('DMESG : %s' % sysvals.dmesgfile)
+                                rerunTest()
         testruns = []
         for dirname, dirnames, filenames in os.walk(subdir):
                 for filename in filenames:
                         if(not re.match('.*.html', filename)):
                                 continue
                         file = os.path.join(dirname, filename)
-                        html = open(file, 'r').read(10000)
-                        suspend = find_in_html(html,
-                                ['Kernel Suspend: ', 'Kernel Suspend Time: '])
-                        resume = find_in_html(html,
-                                ['Kernel Resume: ', 'Kernel Resume Time: '])
-                        line = find_in_html(html, ['<div class="stamp">'], True)
+                        html = open(file, 'r').read()
+                        suspend = find_in_html(html, 'Kernel Suspend', 'ms')
+                        resume = find_in_html(html, 'Kernel Resume', 'ms')
+                        line = find_in_html(html, '<div class="stamp">', '</div>')
                         stmp = line.split()
-                        if not suspend or not resume or len(stmp) < 4:
+                        if not suspend or not resume or len(stmp) != 8:
                                 continue
+                        try:
+                                dt = datetime.strptime(' '.join(stmp[3:]), '%B %d %Y, %I:%M:%S %p')
+                        except:
+                                continue
+                        tstr = dt.strftime('%Y/%m/%d %H:%M:%S')
+                        error = find_in_html(html, '<table class="testfail"><tr><td>', '</td>')
+                        result = 'fail' if error else 'pass'
+                        ilist = []
+                        e = find_in_html(html, 'class="err"[\w=":;\.%\- ]*>', '&rarr;</div>', False)
+                        for i in list(set(e)):
+                                ilist.append('%sx%d' % (i, e.count(i)) if e.count(i) > 1 else i)
                         data = {
+                                'mode': stmp[2],
                                 'host': stmp[0],
                                 'kernel': stmp[1],
-                                'mode': stmp[2],
-                                'time': string.join(stmp[3:], ' '),
+                                'time': tstr,
+                                'result': result,
+                                'issues': ','.join(ilist),
                                 'suspend': suspend,
                                 'resume': resume,
                                 'url': os.path.relpath(file, outpath),
                         }
-                        if len(stmp) == 7:
-                                data['kernel'] = 'unknown'
-                                data['mode'] = stmp[1]
-                                data['time'] = string.join(stmp[2:], ' ')
                         testruns.append(data)
         outfile = os.path.join(outpath, 'summary.html')
         print('Summary file: %s' % outfile)
@@ -5609,11 +5728,12 @@ def printHelp():
         print('   -modes       List available suspend modes')
         print('   -status      Test to see if the system is enabled to run this tool')
         print('   -fpdt        Print out the contents of the ACPI Firmware Performance Data Table')
+        print('   -battery     Print out battery info (if available)')
         print('   -sysinfo     Print out system info extracted from BIOS')
         print('   -devinfo     Print out the pm settings of all devices which support runtime suspend')
         print('   -flist       Print the list of functions currently being captured in ftrace')
         print('   -flistall    Print all functions capable of being captured in ftrace')
-        print('   -summary directory  Create a summary of all test in this dir')
+        print('   -summary dir Create a summary of tests in this dir [-genhtml builds missing html]')
         print('  [redo]')
         print('   -ftrace ftracefile  Create HTML output using ftrace input (used with -dmesg)')
         print('   -dmesg dmesgfile    Create HTML output using dmesg (used with -ftrace)')
@@ -5623,8 +5743,9 @@ def printHelp():
 # ----------------- MAIN --------------------
 # exec start (skipped if script is loaded as library)
 if __name__ == '__main__':
+        genhtml = False
         cmd = ''
-        simplecmds = ['-sysinfo', '-modes', '-fpdt', '-flist', '-flistall', '-devinfo', '-status']
+        simplecmds = ['-sysinfo', '-modes', '-fpdt', '-flist', '-flistall', '-devinfo', '-status', '-battery']
         if '-f' in sys.argv:
                 sysvals.cgskip = sysvals.configFile('cgskip.txt')
         # loop through the command line arguments
@@ -5660,6 +5781,8 @@ if __name__ == '__main__':
                         sysvals.skiphtml = True
                 elif(arg == '-cgdump'):
                         sysvals.cgdump = True
+                elif(arg == '-genhtml'):
+                        genhtml = True
                 elif(arg == '-addlogs'):
                         sysvals.dmesglog = sysvals.ftracelog = True
                 elif(arg == '-verbose'):
@@ -5856,6 +5979,8 @@ if __name__ == '__main__':
                         statusCheck(True)
                 elif(cmd == 'fpdt'):
                         getFPDT(True)
+                elif(cmd == 'battery'):
+                        print 'AC Connect: %s\nCharge: %d' % getBattery()
                 elif(cmd == 'sysinfo'):
                         sysvals.printSystemInfo(True)
                 elif(cmd == 'devinfo'):
@@ -5867,7 +5992,7 @@ if __name__ == '__main__':
                 elif(cmd == 'flistall'):
                         sysvals.getFtraceFilterFunctions(False)
                 elif(cmd == 'summary'):
-                        runSummary(sysvals.outdir, True)
+                        runSummary(sysvals.outdir, True, genhtml)
                 sys.exit()
 
         # if instructed, re-analyze existing data files
@@ -5920,7 +6045,7 @@ if __name__ == '__main__':
                         print('TEST (%d/%d) COMPLETE' % (i+1, sysvals.multitest['count']))
                         sysvals.logmsg = ''
                 if not sysvals.skiphtml:
-                        runSummary(sysvals.outdir, False)
+                        runSummary(sysvals.outdir, False, False)
                 sysvals.sudouser(sysvals.outdir)
         else:
                 if sysvals.outdir:
diff --git a/tools/power/x86/intel_pstate_tracer/intel_pstate_tracer.py b/tools/power/x86/intel_pstate_tracer/intel_pstate_tracer.py
index 29f50d4cfea0..84e2b648e622 100755
--- a/tools/power/x86/intel_pstate_tracer/intel_pstate_tracer.py
+++ b/tools/power/x86/intel_pstate_tracer/intel_pstate_tracer.py
@@ -28,6 +28,7 @@ import subprocess
 import os
 import time
 import re
+import signal
 import sys
 import getopt
 import Gnuplot
@@ -78,11 +79,12 @@ def print_help():
     print('    Or')
     print('      ./intel_pstate_tracer.py [--cpu cpus] ---trace_file <trace_file> --name <test_name>')
     print('    To generate trace file, parse and plot, use (sudo required):')
-    print('      sudo ./intel_pstate_tracer.py [-c cpus] -i <interval> -n <test_name>')
+    print('      sudo ./intel_pstate_tracer.py [-c cpus] -i <interval> -n <test_name> -m <kbytes>')
     print('    Or')
-    print('      sudo ./intel_pstate_tracer.py [--cpu cpus] --interval <interval> --name <test_name>')
+    print('      sudo ./intel_pstate_tracer.py [--cpu cpus] --interval <interval> --name <test_name> --memory <kbytes>')
     print('    Optional argument:')
-    print('      cpus:  comma separated list of CPUs')
+    print('      cpus:   comma separated list of CPUs')
+    print('      kbytes: Kilo bytes of memory per CPU to allocate to the trace buffer. Default: 10240')
     print('  Output:')
     print('    If not already present, creates a "results/test_name" folder in the current working directory with:')
     print('      cpu.csv - comma seperated values file with trace contents and some additional calculations.')
@@ -379,7 +381,7 @@ def clear_trace_file():
         f_handle.close()
     except:
         print('IO error clearing trace file ')
-        quit()
+        sys.exit(2)
 
 def enable_trace():
     """ Enable trace """
@@ -389,7 +391,7 @@ def enable_trace():
                  , 'w').write("1")
     except:
         print('IO error enabling trace ')
-        quit()
+        sys.exit(2)
 
 def disable_trace():
     """ Disable trace """
@@ -399,17 +401,17 @@ def disable_trace():
                  , 'w').write("0")
     except:
         print('IO error disabling trace ')
-        quit()
+        sys.exit(2)
 
 def set_trace_buffer_size():
     """ Set trace buffer size """
 
     try:
-       open('/sys/kernel/debug/tracing/buffer_size_kb'
-                 , 'w').write("10240")
+       with open('/sys/kernel/debug/tracing/buffer_size_kb', 'w') as fp:
+          fp.write(memory)
     except:
-        print('IO error setting trace buffer size ')
-        quit()
+       print('IO error setting trace buffer size ')
+       sys.exit(2)
 
 def free_trace_buffer():
     """ Free the trace buffer memory """
@@ -418,8 +420,8 @@ def free_trace_buffer():
        open('/sys/kernel/debug/tracing/buffer_size_kb'
                  , 'w').write("1")
     except:
-        print('IO error setting trace buffer size ')
-        quit()
+        print('IO error freeing trace buffer ')
+        sys.exit(2)
 
 def read_trace_data(filename):
     """ Read and parse trace data """
@@ -431,7 +433,7 @@ def read_trace_data(filename):
         data = open(filename, 'r').read()
     except:
         print('Error opening ', filename)
-        quit()
+        sys.exit(2)
 
     for line in data.splitlines():
         search_obj = \
@@ -489,10 +491,22 @@ def read_trace_data(filename):
 # Now seperate the main overall csv file into per CPU csv files.
     split_csv()
 
+def signal_handler(signal, frame):
+    print(' SIGINT: Forcing cleanup before exit.')
+    if interval:
+        disable_trace()
+        clear_trace_file()
+        # Free the memory
+        free_trace_buffer()
+        sys.exit(0)
+
+signal.signal(signal.SIGINT, signal_handler)
+
 interval = ""
 filename = ""
 cpu_list = ""
 testname = ""
+memory = "10240"
 graph_data_present = False;
 
 valid1 = False
@@ -501,7 +515,7 @@ valid2 = False
 cpu_mask = zeros((MAX_CPUS,), dtype=int)
 
 try:
-    opts, args = getopt.getopt(sys.argv[1:],"ht:i:c:n:",["help","trace_file=","interval=","cpu=","name="])
+    opts, args = getopt.getopt(sys.argv[1:],"ht:i:c:n:m:",["help","trace_file=","interval=","cpu=","name=","memory="])
 except getopt.GetoptError:
     print_help()
     sys.exit(2)
@@ -521,6 +535,8 @@ for opt, arg in opts:
     elif opt in ("-n", "--name"):
         valid2 = True
         testname = arg
+    elif opt in ("-m", "--memory"):
+        memory = arg
 
 if not (valid1 and valid2):
     print_help()
@@ -569,6 +585,11 @@ current_max_cpu = 0
 
 read_trace_data(filename)
 
+clear_trace_file()
+# Free the memory
+if interval:
+    free_trace_buffer()
+
 if graph_data_present == False:
     print('No valid data to plot')
     sys.exit(2)
@@ -593,9 +614,4 @@ for root, dirs, files in os.walk('.'):
     for f in files:
         fix_ownership(f)
 
-clear_trace_file()
-# Free the memory
-if interval:
-    free_trace_buffer()
-
 os.chdir('../../')
diff --git a/tools/power/x86/turbostat/Makefile b/tools/power/x86/turbostat/Makefile
index a9bc914a8fe8..2ab25aa38263 100644
--- a/tools/power/x86/turbostat/Makefile
+++ b/tools/power/x86/turbostat/Makefile
@@ -25,4 +25,4 @@ install : turbostat
         install -d  $(DESTDIR)$(PREFIX)/bin
         install $(BUILD_OUTPUT)/turbostat $(DESTDIR)$(PREFIX)/bin/turbostat
         install -d  $(DESTDIR)$(PREFIX)/share/man/man8
-        install turbostat.8 $(DESTDIR)$(PREFIX)/share/man/man8
+        install -m 644 turbostat.8 $(DESTDIR)$(PREFIX)/share/man/man8
diff --git a/tools/power/x86/turbostat/turbostat.8 b/tools/power/x86/turbostat/turbostat.8
index ccf2a69365cc..ca9ef7017624 100644
--- a/tools/power/x86/turbostat/turbostat.8
+++ b/tools/power/x86/turbostat/turbostat.8
@@ -54,9 +54,12 @@ name as necessary to disambiguate it from others is necessary.  Note that option
 .PP
 \fB--cpu cpu-set\fP limit output to system summary plus the specified cpu-set.  If cpu-set is the string "core", then the system summary plus the first CPU in each core are printed -- eg. subsequent HT siblings are not printed.  Or if cpu-set is the string "package", then the system summary plus the first CPU in each package is printed.  Otherwise, the system summary plus the specified set of CPUs are printed.  The cpu-set is ordered from low to high, comma delimited with ".." and "-" permitted to denote a range. eg. 1,2,8,14..17,21-44
 .PP
-\fB--hide column\fP do not show the specified columns.  May be invoked multiple times, or with a comma-separated list of column names.  Use "--hide sysfs" to hide the sysfs statistics columns as a group.
+\fB--hide column\fP do not show the specified built-in columns.  May be invoked multiple times, or with a comma-separated list of column names.  Use "--hide sysfs" to hide the sysfs statistics columns as a group.
 .PP
-\fB--show column\fP show only the specified columns.  May be invoked multiple times, or with a comma-separated list of column names.  Use "--show sysfs" to show the sysfs statistics columns as a group.
+\fB--enable column\fP show the specified built-in columns, which are otherwise disabled, by default.  Currently the only built-in counters disabled by default are "usec" and "Time_Of_Day_Seconds".
+The column name "all" can be used to enable all disabled-by-default built-in counters.
+.PP
+\fB--show column\fP show only the specified built-in columns.  May be invoked multiple times, or with a comma-separated list of column names.  Use "--show sysfs" to show the sysfs statistics columns as a group.
 .PP
 \fB--Dump\fP displays the raw counter values.
 .PP
@@ -64,6 +67,8 @@ name as necessary to disambiguate it from others is necessary.  Note that option
 .PP
 \fB--interval seconds\fP overrides the default 5.0 second measurement interval.
 .PP
+\fB--num_iterations num\fP number of the measurement iterations.
+.PP
 \fB--out output_file\fP turbostat output is written to the specified output_file.
 The file is truncated if it already exists, and it is created if it does not exist.
 .PP
@@ -86,6 +91,8 @@ displays the statistics gathered since it was forked.
 The system configuration dump (if --quiet is not used) is followed by statistics.  The first row of the statistics labels the content of each column (below).  The second row of statistics is the system summary line.  The system summary line has a '-' in the columns for the Package, Core, and CPU.  The contents of the system summary line depends on the type of column.  Columns that count items (eg. IRQ) show the sum across all CPUs in the system.  Columns that show a percentage show the average across all CPUs in the system.  Columns that dump raw MSR values simply show 0 in the summary.  After the system summary row, each row describes a specific Package/Core/CPU.  Note that if the --cpu parameter is used to limit which specific CPUs are displayed, turbostat will still collect statistics for all CPUs in the system and will still show the system summary for all CPUs in the system.
 .SH COLUMN DESCRIPTIONS
 .nf
+\fBusec\fP For each CPU, the number of microseconds elapsed during counter collection, including thread migration -- if any.  This counter is disabled by default, and is enabled with "--enable usec", or --debug.  On the summary row, usec refers to the total elapsed time to collect the counters on all cpus.
+\fBTime_Of_Day_Seconds\fP For each CPU, the gettimeofday(2) value (seconds.subsec since Epoch) when the counters ending the measurement interval were collected.  This column is disabled by default, and can be enabled with "--enable Time_Of_Day_Seconds" or "--debug".  On the summary row, Time_Of_Day_Seconds refers to the timestamp following collection of counters on the last CPU.
 \fBCore\fP processor core number.  Note that multiple CPUs per core indicate support for Intel(R) Hyper-Threading Technology (HT).
 \fBCPU\fP Linux CPU (logical processor) number.  Yes, it is okay that on many systems the CPUs are not listed in numerical order -- for efficiency reasons, turbostat runs in topology order, so HT siblings appear together.
 \fBPackage\fP processor package number -- not present on systems with a single processor package.
@@ -262,6 +269,21 @@ CPU	  PRF_CTRL
 
 .fi
 
+.SH INPUT
+
+For interval-mode, turbostat will immediately end the current interval
+when it sees a newline on standard input.
+turbostat will then start the next interval.
+Control-C will be send a SIGINT to turbostat,
+which will immediately abort the program with no further processing.
+.SH SIGNALS
+
+SIGINT will interrupt interval-mode.
+The end-of-interval data will be collected and displayed before turbostat exits.
+
+SIGUSR1 will end current interval,
+end-of-interval data will be collected and displayed before turbostat
+starts a new interval.
 .SH NOTES
 
 .B "turbostat "
diff --git a/tools/power/x86/turbostat/turbostat.c b/tools/power/x86/turbostat/turbostat.c
index bd9c6b31a504..d6cff3070ebd 100644
--- a/tools/power/x86/turbostat/turbostat.c
+++ b/tools/power/x86/turbostat/turbostat.c
@@ -29,6 +29,7 @@
 #include <sys/types.h>
 #include <sys/wait.h>
 #include <sys/stat.h>
+#include <sys/select.h>
 #include <sys/resource.h>
 #include <fcntl.h>
 #include <signal.h>
@@ -47,9 +48,13 @@
 char *proc_stat = "/proc/stat";
 FILE *outf;
 int *fd_percpu;
+struct timeval interval_tv = {5, 0};
 struct timespec interval_ts = {5, 0};
+struct timespec one_msec = {0, 1000000};
+unsigned int num_iterations;
 unsigned int debug;
 unsigned int quiet;
+unsigned int shown;
 unsigned int sums_need_wide_columns;
 unsigned int rapl_joules;
 unsigned int summary_only;
@@ -58,6 +63,7 @@ unsigned int dump_only;
 unsigned int do_snb_cstates;
 unsigned int do_knl_cstates;
 unsigned int do_slm_cstates;
+unsigned int do_cnl_cstates;
 unsigned int use_c1_residency_msr;
 unsigned int has_aperf;
 unsigned int has_epb;
@@ -80,6 +86,8 @@ unsigned int do_rapl;
 unsigned int do_dts;
 unsigned int do_ptm;
 unsigned long long  gfx_cur_rc6_ms;
+unsigned long long cpuidle_cur_cpu_lpi_us;
+unsigned long long cpuidle_cur_sys_lpi_us;
 unsigned int gfx_cur_mhz;
 unsigned int tcc_activation_temp;
 unsigned int tcc_activation_temp_override;
@@ -87,6 +95,7 @@ double rapl_power_units, rapl_time_units;
 double rapl_dram_energy_units, rapl_energy_units;
 double rapl_joule_counter_range;
 unsigned int do_core_perf_limit_reasons;
+unsigned int has_automatic_cstate_conversion;
 unsigned int do_gfx_perf_limit_reasons;
 unsigned int do_ring_perf_limit_reasons;
 unsigned int crystal_hz;
@@ -147,7 +156,9 @@ char *progname;
 #define CPU_SUBSET_MAXCPUS      1024    /* need to use before probe... */
 cpu_set_t *cpu_present_set, *cpu_affinity_set, *cpu_subset;
 size_t cpu_present_setsize, cpu_affinity_setsize, cpu_subset_size;
-#define MAX_ADDED_COUNTERS 16
+#define MAX_ADDED_COUNTERS 8
+#define MAX_ADDED_THREAD_COUNTERS 24
+#define BITMASK_SIZE 32
 
 struct thread_data {
         struct timeval tv_begin;
@@ -162,7 +173,7 @@ struct thread_data {
         unsigned int flags;
 #define CPU_IS_FIRST_THREAD_IN_CORE     0x2
 #define CPU_IS_FIRST_CORE_IN_PACKAGE    0x4
-        unsigned long long counter[MAX_ADDED_COUNTERS];
+        unsigned long long counter[MAX_ADDED_THREAD_COUNTERS];
 } *thread_even, *thread_odd;
 
 struct core_data {
@@ -183,6 +194,8 @@ struct pkg_data {
         unsigned long long pc8;
         unsigned long long pc9;
         unsigned long long pc10;
+        unsigned long long cpu_lpi;
+        unsigned long long sys_lpi;
         unsigned long long pkg_wtd_core_c0;
         unsigned long long pkg_any_core_c0;
         unsigned long long pkg_any_gfxe_c0;
@@ -203,12 +216,21 @@ struct pkg_data {
 #define ODD_COUNTERS thread_odd, core_odd, package_odd
 #define EVEN_COUNTERS thread_even, core_even, package_even
 
-#define GET_THREAD(thread_base, thread_no, core_no, pkg_no) \
-        (thread_base + (pkg_no) * topo.num_cores_per_pkg * \
-                topo.num_threads_per_core + \
-                (core_no) * topo.num_threads_per_core + (thread_no))
-#define GET_CORE(core_base, core_no, pkg_no) \
-        (core_base + (pkg_no) * topo.num_cores_per_pkg + (core_no))
+#define GET_THREAD(thread_base, thread_no, core_no, node_no, pkg_no)          \
+        ((thread_base) +                                                      \
+         ((pkg_no) *                                                          \
+          topo.nodes_per_pkg * topo.cores_per_node * topo.threads_per_core) + \
+         ((node_no) * topo.cores_per_node * topo.threads_per_core) +          \
+         ((core_no) * topo.threads_per_core) +                                \
+         (thread_no))
+
+#define GET_CORE(core_base, core_no, node_no, pkg_no)                   \
+        ((core_base) +                                                  \
+         ((pkg_no) *  topo.nodes_per_pkg * topo.cores_per_node) +       \
+         ((node_no) * topo.cores_per_node) +                            \
+         (core_no))
+
+
 #define GET_PKG(pkg_base, pkg_no) (pkg_base + pkg_no)
 
 enum counter_scope {SCOPE_CPU, SCOPE_CORE, SCOPE_PACKAGE};
@@ -244,14 +266,25 @@ struct system_summary {
         struct pkg_data packages;
 } average;
 
+struct cpu_topology {
+        int physical_package_id;
+        int logical_cpu_id;
+        int physical_node_id;
+        int logical_node_id;    /* 0-based count within the package */
+        int physical_core_id;
+        int thread_id;
+        cpu_set_t *put_ids; /* Processing Unit/Thread IDs */
+} *cpus;
 
 struct topo_params {
         int num_packages;
         int num_cpus;
         int num_cores;
         int max_cpu_num;
-        int num_cores_per_pkg;
-        int num_threads_per_core;
+        int max_node_num;
+        int nodes_per_pkg;
+        int cores_per_node;
+        int threads_per_core;
 } topo;
 
 struct timeval tv_even, tv_odd, tv_delta;
@@ -273,27 +306,33 @@ int cpu_is_not_present(int cpu)
 int for_all_cpus(int (func)(struct thread_data *, struct core_data *, struct pkg_data *),
         struct thread_data *thread_base, struct core_data *core_base, struct pkg_data *pkg_base)
 {
-        int retval, pkg_no, core_no, thread_no;
+        int retval, pkg_no, core_no, thread_no, node_no;
 
         for (pkg_no = 0; pkg_no < topo.num_packages; ++pkg_no) {
-                for (core_no = 0; core_no < topo.num_cores_per_pkg; ++core_no) {
-                        for (thread_no = 0; thread_no <
-                                topo.num_threads_per_core; ++thread_no) {
-                                struct thread_data *t;
-                                struct core_data *c;
-                                struct pkg_data *p;
-
-                                t = GET_THREAD(thread_base, thread_no, core_no, pkg_no);
-
-                                if (cpu_is_not_present(t->cpu_id))
-                                        continue;
-
-                                c = GET_CORE(core_base, core_no, pkg_no);
-                                p = GET_PKG(pkg_base, pkg_no);
-
-                                retval = func(t, c, p);
-                                if (retval)
-                                        return retval;
+                for (core_no = 0; core_no < topo.cores_per_node; ++core_no) {
+                        for (node_no = 0; node_no < topo.nodes_per_pkg;
+                             node_no++) {
+                                for (thread_no = 0; thread_no <
+                                        topo.threads_per_core; ++thread_no) {
+                                        struct thread_data *t;
+                                        struct core_data *c;
+                                        struct pkg_data *p;
+
+                                        t = GET_THREAD(thread_base, thread_no,
+                                                       core_no, node_no,
+                                                       pkg_no);
+
+                                        if (cpu_is_not_present(t->cpu_id))
+                                                continue;
+
+                                        c = GET_CORE(core_base, core_no,
+                                                     node_no, pkg_no);
+                                        p = GET_PKG(pkg_base, pkg_no);
+
+                                        retval = func(t, c, p);
+                                        if (retval)
+                                                return retval;
+                                }
                         }
                 }
         }
@@ -346,6 +385,8 @@ int get_msr(int cpu, off_t offset, unsigned long long *msr)
  * Thus, strings that are proper sub-sets must follow their more specific peers.
  */
 struct msr_counter bic[] = {
+        { 0x0, "usec" },
+        { 0x0, "Time_Of_Day_Seconds" },
         { 0x0, "Package" },
         { 0x0, "Avg_MHz" },
         { 0x0, "Bzy_MHz" },
@@ -369,7 +410,9 @@ struct msr_counter bic[] = {
         { 0x0, "Pkg%pc7" },
         { 0x0, "Pkg%pc8" },
         { 0x0, "Pkg%pc9" },
-        { 0x0, "Pkg%pc10" },
+        { 0x0, "Pk%pc10" },
+        { 0x0, "CPU%LPI" },
+        { 0x0, "SYS%LPI" },
         { 0x0, "PkgWatt" },
         { 0x0, "CorWatt" },
         { 0x0, "GFXWatt" },
@@ -389,62 +432,72 @@ struct msr_counter bic[] = {
         { 0x0, "Any%C0" },
         { 0x0, "GFX%C0" },
         { 0x0, "CPUGFX%" },
+        { 0x0, "Node%" },
 };
 
 
 
 #define MAX_BIC (sizeof(bic) / sizeof(struct msr_counter))
-#define BIC_Package     (1ULL << 0)
-#define BIC_Avg_MHz     (1ULL << 1)
-#define BIC_Bzy_MHz     (1ULL << 2)
-#define BIC_TSC_MHz     (1ULL << 3)
-#define BIC_IRQ         (1ULL << 4)
-#define BIC_SMI         (1ULL << 5)
-#define BIC_Busy        (1ULL << 6)
-#define BIC_CPU_c1      (1ULL << 7)
-#define BIC_CPU_c3      (1ULL << 8)
-#define BIC_CPU_c6      (1ULL << 9)
-#define BIC_CPU_c7      (1ULL << 10)
-#define BIC_ThreadC     (1ULL << 11)
-#define BIC_CoreTmp     (1ULL << 12)
-#define BIC_CoreCnt     (1ULL << 13)
-#define BIC_PkgTmp      (1ULL << 14)
-#define BIC_GFX_rc6     (1ULL << 15)
-#define BIC_GFXMHz      (1ULL << 16)
-#define BIC_Pkgpc2      (1ULL << 17)
-#define BIC_Pkgpc3      (1ULL << 18)
-#define BIC_Pkgpc6      (1ULL << 19)
-#define BIC_Pkgpc7      (1ULL << 20)
-#define BIC_Pkgpc8      (1ULL << 21)
-#define BIC_Pkgpc9      (1ULL << 22)
-#define BIC_Pkgpc10     (1ULL << 23)
-#define BIC_PkgWatt     (1ULL << 24)
-#define BIC_CorWatt     (1ULL << 25)
-#define BIC_GFXWatt     (1ULL << 26)
-#define BIC_PkgCnt      (1ULL << 27)
-#define BIC_RAMWatt     (1ULL << 28)
-#define BIC_PKG__       (1ULL << 29)
-#define BIC_RAM__       (1ULL << 30)
-#define BIC_Pkg_J       (1ULL << 31)
-#define BIC_Cor_J       (1ULL << 32)
-#define BIC_GFX_J       (1ULL << 33)
-#define BIC_RAM_J       (1ULL << 34)
-#define BIC_Core        (1ULL << 35)
-#define BIC_CPU         (1ULL << 36)
-#define BIC_Mod_c6      (1ULL << 37)
-#define BIC_sysfs       (1ULL << 38)
-#define BIC_Totl_c0     (1ULL << 39)
-#define BIC_Any_c0      (1ULL << 40)
-#define BIC_GFX_c0      (1ULL << 41)
-#define BIC_CPUGFX      (1ULL << 42)
-
-unsigned long long bic_enabled = 0xFFFFFFFFFFFFFFFFULL;
-unsigned long long bic_present = BIC_sysfs;
+#define BIC_USEC        (1ULL << 0)
+#define BIC_TOD         (1ULL << 1)
+#define BIC_Package     (1ULL << 2)
+#define BIC_Avg_MHz     (1ULL << 3)
+#define BIC_Bzy_MHz     (1ULL << 4)
+#define BIC_TSC_MHz     (1ULL << 5)
+#define BIC_IRQ         (1ULL << 6)
+#define BIC_SMI         (1ULL << 7)
+#define BIC_Busy        (1ULL << 8)
+#define BIC_CPU_c1      (1ULL << 9)
+#define BIC_CPU_c3      (1ULL << 10)
+#define BIC_CPU_c6      (1ULL << 11)
+#define BIC_CPU_c7      (1ULL << 12)
+#define BIC_ThreadC     (1ULL << 13)
+#define BIC_CoreTmp     (1ULL << 14)
+#define BIC_CoreCnt     (1ULL << 15)
+#define BIC_PkgTmp      (1ULL << 16)
+#define BIC_GFX_rc6     (1ULL << 17)
+#define BIC_GFXMHz      (1ULL << 18)
+#define BIC_Pkgpc2      (1ULL << 19)
+#define BIC_Pkgpc3      (1ULL << 20)
+#define BIC_Pkgpc6      (1ULL << 21)
+#define BIC_Pkgpc7      (1ULL << 22)
+#define BIC_Pkgpc8      (1ULL << 23)
+#define BIC_Pkgpc9      (1ULL << 24)
+#define BIC_Pkgpc10     (1ULL << 25)
+#define BIC_CPU_LPI     (1ULL << 26)
+#define BIC_SYS_LPI     (1ULL << 27)
+#define BIC_PkgWatt     (1ULL << 26)
+#define BIC_CorWatt     (1ULL << 27)
+#define BIC_GFXWatt     (1ULL << 28)
+#define BIC_PkgCnt      (1ULL << 29)
+#define BIC_RAMWatt     (1ULL << 30)
+#define BIC_PKG__       (1ULL << 31)
+#define BIC_RAM__       (1ULL << 32)
+#define BIC_Pkg_J       (1ULL << 33)
+#define BIC_Cor_J       (1ULL << 34)
+#define BIC_GFX_J       (1ULL << 35)
+#define BIC_RAM_J       (1ULL << 36)
+#define BIC_Core        (1ULL << 37)
+#define BIC_CPU         (1ULL << 38)
+#define BIC_Mod_c6      (1ULL << 39)
+#define BIC_sysfs       (1ULL << 40)
+#define BIC_Totl_c0     (1ULL << 41)
+#define BIC_Any_c0      (1ULL << 42)
+#define BIC_GFX_c0      (1ULL << 43)
+#define BIC_CPUGFX      (1ULL << 44)
+#define BIC_Node        (1ULL << 45)
+
+#define BIC_DISABLED_BY_DEFAULT (BIC_USEC | BIC_TOD)
+
+unsigned long long bic_enabled = (0xFFFFFFFFFFFFFFFFULL & ~BIC_DISABLED_BY_DEFAULT);
+unsigned long long bic_present = BIC_USEC | BIC_TOD | BIC_sysfs;
 
 #define DO_BIC(COUNTER_NAME) (bic_enabled & bic_present & COUNTER_NAME)
+#define ENABLE_BIC(COUNTER_NAME) (bic_enabled |= COUNTER_NAME)
 #define BIC_PRESENT(COUNTER_BIT) (bic_present |= COUNTER_BIT)
 #define BIC_NOT_PRESENT(COUNTER_BIT) (bic_present &= ~COUNTER_BIT)
 
+
 #define MAX_DEFERRED 16
 char *deferred_skip_names[MAX_DEFERRED];
 int deferred_skip_index;
@@ -469,9 +522,10 @@ void help(void)
         "--cpu  cpu-set limit output to summary plus cpu-set:\n"
         "               {core | package | j,k,l..m,n-p }\n"
         "--quiet        skip decoding system configuration header\n"
-        "--interval sec Override default 5-second measurement interval\n"
+        "--interval sec.subsec  Override default 5-second measurement interval\n"
         "--help         print this help message\n"
         "--list         list column headers only\n"
+        "--num_iterations num   number of the measurement iterations\n"
         "--out file     create or truncate \"file\" for all output\n"
         "--version      print version information\n"
         "\n"
@@ -496,6 +550,9 @@ unsigned long long bic_lookup(char *name_list, enum show_hide_mode mode)
                 if (comma)
                         *comma = '\0';
 
+                if (!strcmp(name_list, "all"))
+                        return ~0;
+
                 for (i = 0; i < MAX_BIC; ++i) {
                         if (!strcmp(name_list, bic[i].name)) {
                                 retval |= (1ULL << i);
@@ -532,10 +589,14 @@ void print_header(char *delim)
         struct msr_counter *mp;
         int printed = 0;
 
-        if (debug)
-                outp += sprintf(outp, "usec %s", delim);
+        if (DO_BIC(BIC_USEC))
+                outp += sprintf(outp, "%susec", (printed++ ? delim : ""));
+        if (DO_BIC(BIC_TOD))
+                outp += sprintf(outp, "%sTime_Of_Day_Seconds", (printed++ ? delim : ""));
         if (DO_BIC(BIC_Package))
                 outp += sprintf(outp, "%sPackage", (printed++ ? delim : ""));
+        if (DO_BIC(BIC_Node))
+                outp += sprintf(outp, "%sNode", (printed++ ? delim : ""));
         if (DO_BIC(BIC_Core))
                 outp += sprintf(outp, "%sCore", (printed++ ? delim : ""));
         if (DO_BIC(BIC_CPU))
@@ -576,7 +637,7 @@ void print_header(char *delim)
 
         if (DO_BIC(BIC_CPU_c1))
                 outp += sprintf(outp, "%sCPU%%c1", (printed++ ? delim : ""));
-        if (DO_BIC(BIC_CPU_c3) && !do_slm_cstates && !do_knl_cstates)
+        if (DO_BIC(BIC_CPU_c3) && !do_slm_cstates && !do_knl_cstates && !do_cnl_cstates)
                 outp += sprintf(outp, "%sCPU%%c3", (printed++ ? delim : ""));
         if (DO_BIC(BIC_CPU_c6))
                 outp += sprintf(outp, "%sCPU%%c6", (printed++ ? delim : ""));
@@ -635,6 +696,10 @@ void print_header(char *delim)
                 outp += sprintf(outp, "%sPkg%%pc9", (printed++ ? delim : ""));
         if (DO_BIC(BIC_Pkgpc10))
                 outp += sprintf(outp, "%sPk%%pc10", (printed++ ? delim : ""));
+        if (DO_BIC(BIC_CPU_LPI))
+                outp += sprintf(outp, "%sCPU%%LPI", (printed++ ? delim : ""));
+        if (DO_BIC(BIC_SYS_LPI))
+                outp += sprintf(outp, "%sSYS%%LPI", (printed++ ? delim : ""));
 
         if (do_rapl && !rapl_joules) {
                 if (DO_BIC(BIC_PkgWatt))
@@ -739,6 +804,9 @@ int dump_counters(struct thread_data *t, struct core_data *c,
                 outp += sprintf(outp, "pc8: %016llX\n", p->pc8);
                 outp += sprintf(outp, "pc9: %016llX\n", p->pc9);
                 outp += sprintf(outp, "pc10: %016llX\n", p->pc10);
+                outp += sprintf(outp, "pc10: %016llX\n", p->pc10);
+                outp += sprintf(outp, "cpu_lpi: %016llX\n", p->cpu_lpi);
+                outp += sprintf(outp, "sys_lpi: %016llX\n", p->sys_lpi);
                 outp += sprintf(outp, "Joules PKG: %0X\n", p->energy_pkg);
                 outp += sprintf(outp, "Joules COR: %0X\n", p->energy_cores);
                 outp += sprintf(outp, "Joules GFX: %0X\n", p->energy_gfx);
@@ -786,7 +854,7 @@ int format_counters(struct thread_data *t, struct core_data *c,
                 (cpu_subset && !CPU_ISSET_S(t->cpu_id, cpu_subset_size, cpu_subset)))
                 return 0;
 
-        if (debug) {
+        if (DO_BIC(BIC_USEC)) {
                 /* on each row, print how many usec each timestamp took to gather */
                 struct timeval tv;
 
@@ -794,6 +862,10 @@ int format_counters(struct thread_data *t, struct core_data *c,
                 outp += sprintf(outp, "%5ld\t", tv.tv_sec * 1000000 + tv.tv_usec);
         }
 
+        /* Time_Of_Day_Seconds: on each row, print sec.usec last timestamp taken */
+        if (DO_BIC(BIC_TOD))
+                outp += sprintf(outp, "%10ld.%06ld\t", t->tv_end.tv_sec, t->tv_end.tv_usec);
+
         interval_float = tv_delta.tv_sec + tv_delta.tv_usec/1000000.0;
 
         tsc = t->tsc * tsc_tweak;
@@ -802,6 +874,8 @@ int format_counters(struct thread_data *t, struct core_data *c,
         if (t == &average.threads) {
                 if (DO_BIC(BIC_Package))
                         outp += sprintf(outp, "%s-", (printed++ ? delim : ""));
+                if (DO_BIC(BIC_Node))
+                        outp += sprintf(outp, "%s-", (printed++ ? delim : ""));
                 if (DO_BIC(BIC_Core))
                         outp += sprintf(outp, "%s-", (printed++ ? delim : ""));
                 if (DO_BIC(BIC_CPU))
@@ -813,6 +887,15 @@ int format_counters(struct thread_data *t, struct core_data *c,
                         else
                                 outp += sprintf(outp, "%s-", (printed++ ? delim : ""));
                 }
+                if (DO_BIC(BIC_Node)) {
+                        if (t)
+                                outp += sprintf(outp, "%s%d",
+                                                (printed++ ? delim : ""),
+                                              cpus[t->cpu_id].physical_node_id);
+                        else
+                                outp += sprintf(outp, "%s-",
+                                                (printed++ ? delim : ""));
+                }
                 if (DO_BIC(BIC_Core)) {
                         if (c)
                                 outp += sprintf(outp, "%s%d", (printed++ ? delim : ""), c->core_id);
@@ -882,7 +965,7 @@ int format_counters(struct thread_data *t, struct core_data *c,
         if (!(t->flags & CPU_IS_FIRST_THREAD_IN_CORE))
                 goto done;
 
-        if (DO_BIC(BIC_CPU_c3) && !do_slm_cstates && !do_knl_cstates)
+        if (DO_BIC(BIC_CPU_c3) && !do_slm_cstates && !do_knl_cstates && !do_cnl_cstates)
                 outp += sprintf(outp, "%s%.2f", (printed++ ? delim : ""), 100.0 * c->c3/tsc);
         if (DO_BIC(BIC_CPU_c6))
                 outp += sprintf(outp, "%s%.2f", (printed++ ? delim : ""), 100.0 * c->c6/tsc);
@@ -959,6 +1042,11 @@ int format_counters(struct thread_data *t, struct core_data *c,
         if (DO_BIC(BIC_Pkgpc10))
                 outp += sprintf(outp, "%s%.2f", (printed++ ? delim : ""), 100.0 * p->pc10/tsc);
 
+        if (DO_BIC(BIC_CPU_LPI))
+                outp += sprintf(outp, "%s%.2f", (printed++ ? delim : ""), 100.0 * p->cpu_lpi / 1000000.0 / interval_float);
+        if (DO_BIC(BIC_SYS_LPI))
+                outp += sprintf(outp, "%s%.2f", (printed++ ? delim : ""), 100.0 * p->sys_lpi / 1000000.0 / interval_float);
+
         /*
          * If measurement interval exceeds minimum RAPL Joule Counter range,
          * indicate that results are suspect by printing "**" in fraction place.
@@ -1006,7 +1094,8 @@ int format_counters(struct thread_data *t, struct core_data *c,
         }
 
 done:
-        outp += sprintf(outp, "\n");
+        if (*(outp - 1) != '\n')
+                outp += sprintf(outp, "\n");
 
         return 0;
 }
@@ -1083,6 +1172,8 @@ delta_package(struct pkg_data *new, struct pkg_data *old)
         old->pc8 = new->pc8 - old->pc8;
         old->pc9 = new->pc9 - old->pc9;
         old->pc10 = new->pc10 - old->pc10;
+        old->cpu_lpi = new->cpu_lpi - old->cpu_lpi;
+        old->sys_lpi = new->sys_lpi - old->sys_lpi;
         old->pkg_temp_c = new->pkg_temp_c;
 
         /* flag an error when rc6 counter resets/wraps */
@@ -1140,6 +1231,15 @@ delta_thread(struct thread_data *new, struct thread_data *old,
         int i;
         struct msr_counter *mp;
 
+        /*
+         * the timestamps from start of measurement interval are in "old"
+         * the timestamp from end of measurement interval are in "new"
+         * over-write old w/ new so we can print end of interval values
+         */
+
+        old->tv_begin = new->tv_begin;
+        old->tv_end = new->tv_end;
+
         old->tsc = new->tsc - old->tsc;
 
         /* check for TSC < 1 Mcycles over interval */
@@ -1228,6 +1328,11 @@ void clear_counters(struct thread_data *t, struct core_data *c, struct pkg_data
         int i;
         struct msr_counter  *mp;
 
+        t->tv_begin.tv_sec = 0;
+        t->tv_begin.tv_usec = 0;
+        t->tv_end.tv_sec = 0;
+        t->tv_end.tv_usec = 0;
+
         t->tsc = 0;
         t->aperf = 0;
         t->mperf = 0;
@@ -1260,6 +1365,8 @@ void clear_counters(struct thread_data *t, struct core_data *c, struct pkg_data
         p->pc8 = 0;
         p->pc9 = 0;
         p->pc10 = 0;
+        p->cpu_lpi = 0;
+        p->sys_lpi = 0;
 
         p->energy_pkg = 0;
         p->energy_dram = 0;
@@ -1286,6 +1393,13 @@ int sum_counters(struct thread_data *t, struct core_data *c,
         int i;
         struct msr_counter *mp;
 
+        /* remember first tv_begin */
+        if (average.threads.tv_begin.tv_sec == 0)
+                average.threads.tv_begin = t->tv_begin;
+
+        /* remember last tv_end */
+        average.threads.tv_end = t->tv_end;
+
         average.threads.tsc += t->tsc;
         average.threads.aperf += t->aperf;
         average.threads.mperf += t->mperf;
@@ -1341,6 +1455,9 @@ int sum_counters(struct thread_data *t, struct core_data *c,
         average.packages.pc9 += p->pc9;
         average.packages.pc10 += p->pc10;
 
+        average.packages.cpu_lpi = p->cpu_lpi;
+        average.packages.sys_lpi = p->sys_lpi;
+
         average.packages.energy_pkg += p->energy_pkg;
         average.packages.energy_dram += p->energy_dram;
         average.packages.energy_cores += p->energy_cores;
@@ -1487,7 +1604,7 @@ int get_mp(int cpu, struct msr_counter *mp, unsigned long long *counterp)
                 if (get_msr(cpu, mp->msr_num, counterp))
                         return -1;
         } else {
-                char path[128];
+                char path[128 + PATH_BYTES];
 
                 if (mp->flags & SYSFS_PERCPU) {
                         sprintf(path, "/sys/devices/system/cpu/cpu%d/%s",
@@ -1603,7 +1720,7 @@ retry:
         if (!(t->flags & CPU_IS_FIRST_THREAD_IN_CORE))
                 goto done;
 
-        if (DO_BIC(BIC_CPU_c3) && !do_slm_cstates && !do_knl_cstates) {
+        if (DO_BIC(BIC_CPU_c3) && !do_slm_cstates && !do_knl_cstates && !do_cnl_cstates) {
                 if (get_msr(cpu, MSR_CORE_C3_RESIDENCY, &c->c3))
                         return -6;
         }
@@ -1684,6 +1801,11 @@ retry:
                 if (get_msr(cpu, MSR_PKG_C10_RESIDENCY, &p->pc10))
                         return -13;
 
+        if (DO_BIC(BIC_CPU_LPI))
+                p->cpu_lpi = cpuidle_cur_cpu_lpi_us;
+        if (DO_BIC(BIC_SYS_LPI))
+                p->sys_lpi = cpuidle_cur_sys_lpi_us;
+
         if (do_rapl & RAPL_PKG) {
                 if (get_msr(cpu, MSR_PKG_ENERGY_STATUS, &msr))
                         return -13;
@@ -1769,7 +1891,7 @@ int slv_pkg_cstate_limits[16] = {PCL__0, PCL__1, PCLRSV, PCLRSV, PCL__4, PCLRSV,
 int amt_pkg_cstate_limits[16] = {PCLUNL, PCL__1, PCL__2, PCLRSV, PCLRSV, PCLRSV, PCL__6, PCL__7, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV};
 int phi_pkg_cstate_limits[16] = {PCL__0, PCL__2, PCL_6N, PCL_6R, PCLRSV, PCLRSV, PCLRSV, PCLUNL, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV};
 int bxt_pkg_cstate_limits[16] = {PCL__0, PCL__2, PCLUNL, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV};
-int skx_pkg_cstate_limits[16] = {PCL__0, PCL__2, PCL_6N, PCL_6R, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLUNL, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV};
+int skx_pkg_cstate_limits[16] = {PCL__0, PCL__2, PCL_6N, PCL_6R, PCLRSV, PCLRSV, PCLRSV, PCLUNL, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV, PCLRSV};
 
 
 static void
@@ -2071,12 +2193,9 @@ dump_nhm_cst_cfg(void)
 
         get_msr(base_cpu, MSR_PKG_CST_CONFIG_CONTROL, &msr);
 
-#define SNB_C1_AUTO_UNDEMOTE              (1UL << 27)
-#define SNB_C3_AUTO_UNDEMOTE              (1UL << 28)
-
         fprintf(outf, "cpu%d: MSR_PKG_CST_CONFIG_CONTROL: 0x%08llx", base_cpu, msr);
 
-        fprintf(outf, " (%s%s%s%s%slocked: pkg-cstate-limit=%d: %s)\n",
+        fprintf(outf, " (%s%s%s%s%slocked, pkg-cstate-limit=%d (%s)",
                 (msr & SNB_C3_AUTO_UNDEMOTE) ? "UNdemote-C3, " : "",
                 (msr & SNB_C1_AUTO_UNDEMOTE) ? "UNdemote-C1, " : "",
                 (msr & NHM_C3_AUTO_DEMOTE) ? "demote-C3, " : "",
@@ -2084,6 +2203,15 @@ dump_nhm_cst_cfg(void)
                 (msr & (1 << 15)) ? "" : "UN",
                 (unsigned int)msr & 0xF,
                 pkg_cstate_limit_strings[pkg_cstate_limit]);
+
+#define AUTOMATIC_CSTATE_CONVERSION             (1UL << 16)
+        if (has_automatic_cstate_conversion) {
+                fprintf(outf, ", automatic c-state conversion=%s",
+                        (msr & AUTOMATIC_CSTATE_CONVERSION) ? "on" : "off");
+        }
+
+        fprintf(outf, ")\n");
+
         return;
 }
 
@@ -2184,6 +2312,8 @@ void free_fd_percpu(void)
 
 void free_all_buffers(void)
 {
+        int i;
+
         CPU_FREE(cpu_present_set);
         cpu_present_set = NULL;
         cpu_present_setsize = 0;
@@ -2216,6 +2346,12 @@ void free_all_buffers(void)
 
         free(irq_column_2_cpu);
         free(irqs_per_cpu);
+
+        for (i = 0; i <= topo.max_cpu_num; ++i) {
+                if (cpus[i].put_ids)
+                        CPU_FREE(cpus[i].put_ids);
+        }
+        free(cpus);
 }
 
 
@@ -2240,44 +2376,6 @@ int parse_int_file(const char *fmt, ...)
 }
 
 /*
- * get_cpu_position_in_core(cpu)
- * return the position of the CPU among its HT siblings in the core
- * return -1 if the sibling is not in list
- */
-int get_cpu_position_in_core(int cpu)
-{
-        char path[64];
-        FILE *filep;
-        int this_cpu;
-        char character;
-        int i;
-
-        sprintf(path,
-                "/sys/devices/system/cpu/cpu%d/topology/thread_siblings_list",
-                cpu);
-        filep = fopen(path, "r");
-        if (filep == NULL) {
-                perror(path);
-                exit(1);
-        }
-
-        for (i = 0; i < topo.num_threads_per_core; i++) {
-                fscanf(filep, "%d", &this_cpu);
-                if (this_cpu == cpu) {
-                        fclose(filep);
-                        return i;
-                }
-
-                /* Account for no separator after last thread*/
-                if (i != (topo.num_threads_per_core - 1))
-                        fscanf(filep, "%c", &character);
-        }
-
-        fclose(filep);
-        return -1;
-}
-
-/*
  * cpu_is_first_core_in_package(cpu)
  * return 1 if given CPU is 1st core in package
  */
@@ -2296,35 +2394,115 @@ int get_core_id(int cpu)
         return parse_int_file("/sys/devices/system/cpu/cpu%d/topology/core_id", cpu);
 }
 
-int get_num_ht_siblings(int cpu)
+void set_node_data(void)
 {
         char path[80];
         FILE *filep;
-        int sib1;
-        int matches = 0;
-        char character;
-        char str[100];
-        char *ch;
+        int pkg, node, cpu;
 
-        sprintf(path, "/sys/devices/system/cpu/cpu%d/topology/thread_siblings_list", cpu);
-        filep = fopen_or_die(path, "r");
+        struct pkg_node_info {
+                int count;
+                int min;
+        } *pni;
 
-        /*
-         * file format:
-         * A ',' separated or '-' separated set of numbers
-         * (eg 1-2 or 1,3,4,5)
-         */
-        fscanf(filep, "%d%c\n", &sib1, &character);
-        fseek(filep, 0, SEEK_SET);
-        fgets(str, 100, filep);
-        ch = strchr(str, character);
-        while (ch != NULL) {
-                matches++;
-                ch = strchr(ch+1, character);
+        pni = calloc(topo.num_packages, sizeof(struct pkg_node_info));
+        if (!pni)
+                err(1, "calloc pkg_node_count");
+
+        for (pkg = 0; pkg < topo.num_packages; pkg++)
+                pni[pkg].min = topo.num_cpus;
+
+        for (node = 0; node <= topo.max_node_num; node++) {
+                /* find the "first" cpu in the node */
+                sprintf(path, "/sys/bus/node/devices/node%d/cpulist", node);
+                filep = fopen(path, "r");
+                if (!filep)
+                        continue;
+                fscanf(filep, "%d", &cpu);
+                fclose(filep);
+
+                pkg = cpus[cpu].physical_package_id;
+                pni[pkg].count++;
+
+                if (node < pni[pkg].min)
+                        pni[pkg].min = node;
         }
 
+        for (pkg = 0; pkg < topo.num_packages; pkg++)
+                if (pni[pkg].count > topo.nodes_per_pkg)
+                        topo.nodes_per_pkg = pni[0].count;
+
+        for (cpu = 0; cpu < topo.num_cpus; cpu++) {
+                pkg = cpus[cpu].physical_package_id;
+                node = cpus[cpu].physical_node_id;
+                cpus[cpu].logical_node_id = node - pni[pkg].min;
+        }
+        free(pni);
+
+}
+
+int get_physical_node_id(struct cpu_topology *thiscpu)
+{
+        char path[80];
+        FILE *filep;
+        int i;
+        int cpu = thiscpu->logical_cpu_id;
+
+        for (i = 0; i <= topo.max_cpu_num; i++) {
+                sprintf(path, "/sys/devices/system/cpu/cpu%d/node%i/cpulist",
+                        cpu, i);
+                filep = fopen(path, "r");
+                if (!filep)
+                        continue;
+                fclose(filep);
+                return i;
+        }
+        return -1;
+}
+
+int get_thread_siblings(struct cpu_topology *thiscpu)
+{
+        char path[80], character;
+        FILE *filep;
+        unsigned long map;
+        int so, shift, sib_core;
+        int cpu = thiscpu->logical_cpu_id;
+        int offset = topo.max_cpu_num + 1;
+        size_t size;
+        int thread_id = 0;
+
+        thiscpu->put_ids = CPU_ALLOC((topo.max_cpu_num + 1));
+        if (thiscpu->thread_id < 0)
+                thiscpu->thread_id = thread_id++;
+        if (!thiscpu->put_ids)
+                return -1;
+
+        size = CPU_ALLOC_SIZE((topo.max_cpu_num + 1));
+        CPU_ZERO_S(size, thiscpu->put_ids);
+
+        sprintf(path,
+                "/sys/devices/system/cpu/cpu%d/topology/thread_siblings", cpu);
+        filep = fopen_or_die(path, "r");
+        do {
+                offset -= BITMASK_SIZE;
+                fscanf(filep, "%lx%c", &map, &character);
+                for (shift = 0; shift < BITMASK_SIZE; shift++) {
+                        if ((map >> shift) & 0x1) {
+                                so = shift + offset;
+                                sib_core = get_core_id(so);
+                                if (sib_core == thiscpu->physical_core_id) {
+                                        CPU_SET_S(so, size, thiscpu->put_ids);
+                                        if ((so != cpu) &&
+                                            (cpus[so].thread_id < 0))
+                                                cpus[so].thread_id =
+                                                                    thread_id++;
+                                }
+                        }
+                }
+        } while (!strncmp(&character, ",", 1));
         fclose(filep);
-        return matches+1;
+
+        return CPU_COUNT_S(size, thiscpu->put_ids);
 }
 
 /*
@@ -2339,32 +2517,42 @@ int for_all_cpus_2(int (func)(struct thread_data *, struct core_data *,
         struct thread_data *thread_base2, struct core_data *core_base2,
         struct pkg_data *pkg_base2)
 {
-        int retval, pkg_no, core_no, thread_no;
+        int retval, pkg_no, node_no, core_no, thread_no;
 
         for (pkg_no = 0; pkg_no < topo.num_packages; ++pkg_no) {
-                for (core_no = 0; core_no < topo.num_cores_per_pkg; ++core_no) {
-                        for (thread_no = 0; thread_no <
-                                topo.num_threads_per_core; ++thread_no) {
-                                struct thread_data *t, *t2;
-                                struct core_data *c, *c2;
-                                struct pkg_data *p, *p2;
-
-                                t = GET_THREAD(thread_base, thread_no, core_no, pkg_no);
-
-                                if (cpu_is_not_present(t->cpu_id))
-                                        continue;
-
-                                t2 = GET_THREAD(thread_base2, thread_no, core_no, pkg_no);
-
-                                c = GET_CORE(core_base, core_no, pkg_no);
-                                c2 = GET_CORE(core_base2, core_no, pkg_no);
-
-                                p = GET_PKG(pkg_base, pkg_no);
-                                p2 = GET_PKG(pkg_base2, pkg_no);
-
-                                retval = func(t, c, p, t2, c2, p2);
-                                if (retval)
-                                        return retval;
+                for (node_no = 0; node_no < topo.nodes_per_pkg; ++node_no) {
+                        for (core_no = 0; core_no < topo.cores_per_node;
+                             ++core_no) {
+                                for (thread_no = 0; thread_no <
+                                        topo.threads_per_core; ++thread_no) {
+                                        struct thread_data *t, *t2;
+                                        struct core_data *c, *c2;
+                                        struct pkg_data *p, *p2;
+
+                                        t = GET_THREAD(thread_base, thread_no,
+                                                       core_no, node_no,
+                                                       pkg_no);
+
+                                        if (cpu_is_not_present(t->cpu_id))
+                                                continue;
+
+                                        t2 = GET_THREAD(thread_base2, thread_no,
+                                                        core_no, node_no,
+                                                        pkg_no);
+
+                                        c = GET_CORE(core_base, core_no,
+                                                     node_no, pkg_no);
+                                        c2 = GET_CORE(core_base2, core_no,
+                                                      node_no,
+                                                      pkg_no);
+
+                                        p = GET_PKG(pkg_base, pkg_no);
+                                        p2 = GET_PKG(pkg_base2, pkg_no);
+
+                                        retval = func(t, c, p, t2, c2, p2);
+                                        if (retval)
+                                                return retval;
+                                }
                         }
                 }
         }
@@ -2409,6 +2597,20 @@ void re_initialize(void)
         printf("turbostat: re-initialized with num_cpus %d\n", topo.num_cpus);
 }
 
+void set_max_cpu_num(void)
+{
+        FILE *filep;
+        unsigned long dummy;
+
+        topo.max_cpu_num = 0;
+        filep = fopen_or_die(
+                        "/sys/devices/system/cpu/cpu0/topology/thread_siblings",
+                        "r");
+        while (fscanf(filep, "%lx,", &dummy) == 1)
+                topo.max_cpu_num += BITMASK_SIZE;
+        fclose(filep);
+        topo.max_cpu_num--; /* 0 based */
+}
 
 /*
  * count_cpus()
@@ -2416,10 +2618,7 @@ void re_initialize(void)
  */
 int count_cpus(int cpu)
 {
-        if (topo.max_cpu_num < cpu)
-                topo.max_cpu_num = cpu;
-
-        topo.num_cpus += 1;
+        topo.num_cpus++;
         return 0;
 }
 int mark_cpu_present(int cpu)
@@ -2428,6 +2627,12 @@ int mark_cpu_present(int cpu)
         return 0;
 }
 
+int init_thread_id(int cpu)
+{
+        cpus[cpu].thread_id = -1;
+        return 0;
+}
+
 /*
  * snapshot_proc_interrupts()
  *
@@ -2542,6 +2747,52 @@ int snapshot_gfx_mhz(void)
 }
 
 /*
+ * snapshot_cpu_lpi()
+ *
+ * record snapshot of
+ * /sys/devices/system/cpu/cpuidle/low_power_idle_cpu_residency_us
+ *
+ * return 1 if config change requires a restart, else return 0
+ */
+int snapshot_cpu_lpi_us(void)
+{
+        FILE *fp;
+        int retval;
+
+        fp = fopen_or_die("/sys/devices/system/cpu/cpuidle/low_power_idle_cpu_residency_us", "r");
+
+        retval = fscanf(fp, "%lld", &cpuidle_cur_cpu_lpi_us);
+        if (retval != 1)
+                err(1, "CPU LPI");
+
+        fclose(fp);
+
+        return 0;
+}
+/*
+ * snapshot_sys_lpi()
+ *
+ * record snapshot of
+ * /sys/devices/system/cpu/cpuidle/low_power_idle_system_residency_us
+ *
+ * return 1 if config change requires a restart, else return 0
+ */
+int snapshot_sys_lpi_us(void)
+{
+        FILE *fp;
+        int retval;
+
+        fp = fopen_or_die("/sys/devices/system/cpu/cpuidle/low_power_idle_system_residency_us", "r");
+
+        retval = fscanf(fp, "%lld", &cpuidle_cur_sys_lpi_us);
+        if (retval != 1)
+                err(1, "SYS LPI");
+
+        fclose(fp);
+
+        return 0;
+}
+/*
  * snapshot /proc and /sys files
  *
  * return 1 if configuration restart needed, else return 0
@@ -2558,13 +2809,83 @@ int snapshot_proc_sysfs_files(void)
         if (DO_BIC(BIC_GFXMHz))
                 snapshot_gfx_mhz();
 
+        if (DO_BIC(BIC_CPU_LPI))
+                snapshot_cpu_lpi_us();
+
+        if (DO_BIC(BIC_SYS_LPI))
+                snapshot_sys_lpi_us();
+
         return 0;
 }
 
+int exit_requested;
+
+static void signal_handler (int signal)
+{
+        switch (signal) {
+        case SIGINT:
+                exit_requested = 1;
+                if (debug)
+                        fprintf(stderr, " SIGINT\n");
+                break;
+        case SIGUSR1:
+                if (debug > 1)
+                        fprintf(stderr, "SIGUSR1\n");
+                break;
+        }
+        /* make sure this manually-invoked interval is at least 1ms long */
+        nanosleep(&one_msec, NULL);
+}
+
+void setup_signal_handler(void)
+{
+        struct sigaction sa;
+
+        memset(&sa, 0, sizeof(sa));
+
+        sa.sa_handler = &signal_handler;
+
+        if (sigaction(SIGINT, &sa, NULL) < 0)
+                err(1, "sigaction SIGINT");
+        if (sigaction(SIGUSR1, &sa, NULL) < 0)
+                err(1, "sigaction SIGUSR1");
+}
+
+void do_sleep(void)
+{
+        struct timeval select_timeout;
+        fd_set readfds;
+        int retval;
+
+        FD_ZERO(&readfds);
+        FD_SET(0, &readfds);
+
+        if (!isatty(fileno(stdin))) {
+                nanosleep(&interval_ts, NULL);
+                return;
+        }
+
+        select_timeout = interval_tv;
+        retval = select(1, &readfds, NULL, NULL, &select_timeout);
+
+        if (retval == 1) {
+                switch (getc(stdin)) {
+                case 'q':
+                        exit_requested = 1;
+                        break;
+                }
+                /* make sure this manually-invoked interval is at least 1ms long */
+                nanosleep(&one_msec, NULL);
+        }
+}
+
 void turbostat_loop()
 {
         int retval;
         int restarted = 0;
+        int done_iters = 0;
+
+        setup_signal_handler();
 
 restart:
         restarted++;
@@ -2581,6 +2902,7 @@ restart:
                 goto restart;
         }
         restarted = 0;
+        done_iters = 0;
         gettimeofday(&tv_even, (struct timezone *)NULL);
 
         while (1) {
@@ -2588,7 +2910,7 @@ restart:
                         re_initialize();
                         goto restart;
                 }
-                nanosleep(&interval_ts, NULL);
+                do_sleep();
                 if (snapshot_proc_sysfs_files())
                         goto restart;
                 retval = for_all_cpus(get_counters, ODD_COUNTERS);
@@ -2607,7 +2929,11 @@ restart:
                 compute_average(EVEN_COUNTERS);
                 format_all_counters(EVEN_COUNTERS);
                 flush_output_stdout();
-                nanosleep(&interval_ts, NULL);
+                if (exit_requested)
+                        break;
+                if (num_iterations && ++done_iters >= num_iterations)
+                        break;
+                do_sleep();
                 if (snapshot_proc_sysfs_files())
                         goto restart;
                 retval = for_all_cpus(get_counters, EVEN_COUNTERS);
@@ -2626,6 +2952,10 @@ restart:
                 compute_average(ODD_COUNTERS);
                 format_all_counters(ODD_COUNTERS);
                 flush_output_stdout();
+                if (exit_requested)
+                        break;
+                if (num_iterations && ++done_iters >= num_iterations)
+                        break;
         }
 }
 
@@ -2740,6 +3070,7 @@ int probe_nhm_msrs(unsigned int family, unsigned int model)
         case INTEL_FAM6_SKYLAKE_DESKTOP:        /* SKL */
         case INTEL_FAM6_KABYLAKE_MOBILE:        /* KBL */
         case INTEL_FAM6_KABYLAKE_DESKTOP:       /* KBL */
+        case INTEL_FAM6_CANNONLAKE_MOBILE:      /* CNL */
                 pkg_cstate_limits = hsw_pkg_cstate_limits;
                 has_misc_feature_control = 1;
                 break;
@@ -2945,6 +3276,7 @@ int has_config_tdp(unsigned int family, unsigned int model)
         case INTEL_FAM6_SKYLAKE_DESKTOP:        /* SKL */
         case INTEL_FAM6_KABYLAKE_MOBILE:        /* KBL */
         case INTEL_FAM6_KABYLAKE_DESKTOP:       /* KBL */
+        case INTEL_FAM6_CANNONLAKE_MOBILE:      /* CNL */
         case INTEL_FAM6_SKYLAKE_X:      /* SKX */
 
         case INTEL_FAM6_XEON_PHI_KNL:   /* Knights Landing */
@@ -3399,6 +3731,7 @@ void rapl_probe(unsigned int family, unsigned int model)
         case INTEL_FAM6_SKYLAKE_DESKTOP:        /* SKL */
         case INTEL_FAM6_KABYLAKE_MOBILE:        /* KBL */
         case INTEL_FAM6_KABYLAKE_DESKTOP:       /* KBL */
+        case INTEL_FAM6_CANNONLAKE_MOBILE:      /* CNL */
                 do_rapl = RAPL_PKG | RAPL_CORES | RAPL_CORE_POLICY | RAPL_DRAM | RAPL_DRAM_PERF_STATUS | RAPL_PKG_PERF_STATUS | RAPL_GFX | RAPL_PKG_POWER_INFO;
                 BIC_PRESENT(BIC_PKG__);
                 BIC_PRESENT(BIC_RAM__);
@@ -3523,6 +3856,12 @@ void perf_limit_reasons_probe(unsigned int family, unsigned int model)
         }
 }
 
+void automatic_cstate_conversion_probe(unsigned int family, unsigned int model)
+{
+        if (is_skx(family, model) || is_bdx(family, model))
+                has_automatic_cstate_conversion = 1;
+}
+
 int print_thermal(struct thread_data *t, struct core_data *c, struct pkg_data *p)
 {
         unsigned long long msr;
@@ -3728,6 +4067,7 @@ int has_snb_msrs(unsigned int family, unsigned int model)
         case INTEL_FAM6_SKYLAKE_DESKTOP:        /* SKL */
         case INTEL_FAM6_KABYLAKE_MOBILE:        /* KBL */
         case INTEL_FAM6_KABYLAKE_DESKTOP:       /* KBL */
+        case INTEL_FAM6_CANNONLAKE_MOBILE:      /* CNL */
         case INTEL_FAM6_SKYLAKE_X:      /* SKX */
         case INTEL_FAM6_ATOM_GOLDMONT:  /* BXT */
         case INTEL_FAM6_ATOM_GEMINI_LAKE:
@@ -3761,6 +4101,7 @@ int has_hsw_msrs(unsigned int family, unsigned int model)
         case INTEL_FAM6_SKYLAKE_DESKTOP:        /* SKL */
         case INTEL_FAM6_KABYLAKE_MOBILE:        /* KBL */
         case INTEL_FAM6_KABYLAKE_DESKTOP:       /* KBL */
+        case INTEL_FAM6_CANNONLAKE_MOBILE:      /* CNL */
         case INTEL_FAM6_ATOM_GOLDMONT:  /* BXT */
         case INTEL_FAM6_ATOM_GEMINI_LAKE:
                 return 1;
@@ -3786,6 +4127,7 @@ int has_skl_msrs(unsigned int family, unsigned int model)
         case INTEL_FAM6_SKYLAKE_DESKTOP:        /* SKL */
         case INTEL_FAM6_KABYLAKE_MOBILE:        /* KBL */
         case INTEL_FAM6_KABYLAKE_DESKTOP:       /* KBL */
+        case INTEL_FAM6_CANNONLAKE_MOBILE:      /* CNL */
                 return 1;
         }
         return 0;
@@ -3815,6 +4157,19 @@ int is_knl(unsigned int family, unsigned int model)
         return 0;
 }
 
+int is_cnl(unsigned int family, unsigned int model)
+{
+        if (!genuine_intel)
+                return 0;
+
+        switch (model) {
+        case INTEL_FAM6_CANNONLAKE_MOBILE: /* CNL */
+                return 1;
+        }
+
+        return 0;
+}
+
 unsigned int get_aperf_mperf_multiplier(unsigned int family, unsigned int model)
 {
         if (is_knl(family, model))
@@ -3947,7 +4302,7 @@ void decode_misc_enable_msr(void)
                         base_cpu, msr,
                         msr & MSR_IA32_MISC_ENABLE_TM1 ? "" : "No-",
                         msr & MSR_IA32_MISC_ENABLE_ENHANCED_SPEEDSTEP ? "" : "No-",
-                        msr & MSR_IA32_MISC_ENABLE_MWAIT ? "No-" : "",
+                        msr & MSR_IA32_MISC_ENABLE_MWAIT ? "" : "No-",
                         msr & MSR_IA32_MISC_ENABLE_PREFETCH_DISABLE ? "No-" : "",
                         msr & MSR_IA32_MISC_ENABLE_TURBO_DISABLE ? "No-" : "");
 }
@@ -4152,7 +4507,6 @@ void process_cpuid()
                                 case INTEL_FAM6_KABYLAKE_DESKTOP:       /* KBL */
                                         crystal_hz = 24000000;  /* 24.0 MHz */
                                         break;
-                                case INTEL_FAM6_SKYLAKE_X:      /* SKX */
                                 case INTEL_FAM6_ATOM_DENVERTON: /* DNV */
                                         crystal_hz = 25000000;  /* 25.0 MHz */
                                         break;
@@ -4253,6 +4607,7 @@ void process_cpuid()
         }
         do_slm_cstates = is_slm(family, model);
         do_knl_cstates  = is_knl(family, model);
+        do_cnl_cstates = is_cnl(family, model);
 
         if (!quiet)
                 decode_misc_pwr_mgmt_msr();
@@ -4262,6 +4617,7 @@ void process_cpuid()
 
         rapl_probe(family, model);
         perf_limit_reasons_probe(family, model);
+        automatic_cstate_conversion_probe(family, model);
 
         if (!quiet)
                 dump_cstate_pstate_config_info(family, model);
@@ -4280,6 +4636,16 @@ void process_cpuid()
         if (!access("/sys/class/graphics/fb0/device/drm/card0/gt_cur_freq_mhz", R_OK))
                 BIC_PRESENT(BIC_GFXMHz);
 
+        if (!access("/sys/devices/system/cpu/cpuidle/low_power_idle_cpu_residency_us", R_OK))
+                BIC_PRESENT(BIC_CPU_LPI);
+        else
+                BIC_NOT_PRESENT(BIC_CPU_LPI);
+
+        if (!access("/sys/devices/system/cpu/cpuidle/low_power_idle_system_residency_us", R_OK))
+                BIC_PRESENT(BIC_SYS_LPI);
+        else
+                BIC_NOT_PRESENT(BIC_SYS_LPI);
+
         if (!quiet)
                 decode_misc_feature_control();
 
@@ -4310,14 +4676,10 @@ void topology_probe()
         int max_core_id = 0;
         int max_package_id = 0;
         int max_siblings = 0;
-        struct cpu_topology {
-                int core_id;
-                int physical_package_id;
-        } *cpus;
 
         /* Initialize num_cpus, max_cpu_num */
+        set_max_cpu_num();
         topo.num_cpus = 0;
-        topo.max_cpu_num = 0;
         for_all_proc_cpus(count_cpus);
         if (!summary_only && topo.num_cpus > 1)
                 BIC_PRESENT(BIC_CPU);
@@ -4357,6 +4719,7 @@ void topology_probe()
         cpu_affinity_setsize = CPU_ALLOC_SIZE((topo.max_cpu_num + 1));
         CPU_ZERO_S(cpu_affinity_setsize, cpu_affinity_set);
 
+        for_all_proc_cpus(init_thread_id);
 
         /*
          * For online cpus
@@ -4370,26 +4733,45 @@ void topology_probe()
                                 fprintf(outf, "cpu%d NOT PRESENT\n", i);
                         continue;
                 }
-                cpus[i].core_id = get_core_id(i);
-                if (cpus[i].core_id > max_core_id)
-                        max_core_id = cpus[i].core_id;
 
+                cpus[i].logical_cpu_id = i;
+
+                /* get package information */
                 cpus[i].physical_package_id = get_physical_package_id(i);
                 if (cpus[i].physical_package_id > max_package_id)
                         max_package_id = cpus[i].physical_package_id;
 
-                siblings = get_num_ht_siblings(i);
+                /* get numa node information */
+                cpus[i].physical_node_id = get_physical_node_id(&cpus[i]);
+                if (cpus[i].physical_node_id > topo.max_node_num)
+                        topo.max_node_num = cpus[i].physical_node_id;
+
+                /* get core information */
+                cpus[i].physical_core_id = get_core_id(i);
+                if (cpus[i].physical_core_id > max_core_id)
+                        max_core_id = cpus[i].physical_core_id;
+
+                /* get thread information */
+                siblings = get_thread_siblings(&cpus[i]);
                 if (siblings > max_siblings)
                         max_siblings = siblings;
+                if (cpus[i].thread_id != -1)
+                        topo.num_cores++;
+
                 if (debug > 1)
-                        fprintf(outf, "cpu %d pkg %d core %d\n",
-                                i, cpus[i].physical_package_id, cpus[i].core_id);
+                        fprintf(outf,
+                                "cpu %d pkg %d node %d core %d thread %d\n",
+                                i, cpus[i].physical_package_id,
+                                cpus[i].physical_node_id,
+                                cpus[i].physical_core_id,
+                                cpus[i].thread_id);
         }
-        topo.num_cores_per_pkg = max_core_id + 1;
+
+        topo.cores_per_node = max_core_id + 1;
         if (debug > 1)
                 fprintf(outf, "max_core_id %d, sizing for %d cores per package\n",
-                        max_core_id, topo.num_cores_per_pkg);
-        if (!summary_only && topo.num_cores_per_pkg > 1)
+                        max_core_id, topo.cores_per_node);
+        if (!summary_only && topo.cores_per_node > 1)
                 BIC_PRESENT(BIC_Core);
 
         topo.num_packages = max_package_id + 1;
@@ -4399,33 +4781,38 @@ void topology_probe()
         if (!summary_only && topo.num_packages > 1)
                 BIC_PRESENT(BIC_Package);
 
-        topo.num_threads_per_core = max_siblings;
+        set_node_data();
         if (debug > 1)
-                fprintf(outf, "max_siblings %d\n", max_siblings);
+                fprintf(outf, "nodes_per_pkg %d\n", topo.nodes_per_pkg);
+        if (!summary_only && topo.nodes_per_pkg > 1)
+                BIC_PRESENT(BIC_Node);
 
-        free(cpus);
+        topo.threads_per_core = max_siblings;
+        if (debug > 1)
+                fprintf(outf, "max_siblings %d\n", max_siblings);
 }
 
 void
-allocate_counters(struct thread_data **t, struct core_data **c, struct pkg_data **p)
+allocate_counters(struct thread_data **t, struct core_data **c,
+                  struct pkg_data **p)
 {
         int i;
+        int num_cores = topo.cores_per_node * topo.nodes_per_pkg *
+                        topo.num_packages;
+        int num_threads = topo.threads_per_core * num_cores;
 
-        *t = calloc(topo.num_threads_per_core * topo.num_cores_per_pkg *
-                topo.num_packages, sizeof(struct thread_data));
+        *t = calloc(num_threads, sizeof(struct thread_data));
         if (*t == NULL)
                 goto error;
 
-        for (i = 0; i < topo.num_threads_per_core *
-                topo.num_cores_per_pkg * topo.num_packages; i++)
+        for (i = 0; i < num_threads; i++)
                 (*t)[i].cpu_id = -1;
 
-        *c = calloc(topo.num_cores_per_pkg * topo.num_packages,
-                sizeof(struct core_data));
+        *c = calloc(num_cores, sizeof(struct core_data));
         if (*c == NULL)
                 goto error;
 
-        for (i = 0; i < topo.num_cores_per_pkg * topo.num_packages; i++)
+        for (i = 0; i < num_cores; i++)
                 (*c)[i].core_id = -1;
 
         *p = calloc(topo.num_packages, sizeof(struct pkg_data));
@@ -4442,47 +4829,39 @@ error:
 /*
  * init_counter()
  *
- * set cpu_id, core_num, pkg_num
  * set FIRST_THREAD_IN_CORE and FIRST_CORE_IN_PACKAGE
- *
- * increment topo.num_cores when 1st core in pkg seen
  */
 void init_counter(struct thread_data *thread_base, struct core_data *core_base,
-        struct pkg_data *pkg_base, int thread_num, int core_num,
-        int pkg_num, int cpu_id)
+        struct pkg_data *pkg_base, int cpu_id)
 {
+        int pkg_id = cpus[cpu_id].physical_package_id;
+        int node_id = cpus[cpu_id].logical_node_id;
+        int core_id = cpus[cpu_id].physical_core_id;
+        int thread_id = cpus[cpu_id].thread_id;
         struct thread_data *t;
         struct core_data *c;
         struct pkg_data *p;
 
-        t = GET_THREAD(thread_base, thread_num, core_num, pkg_num);
-        c = GET_CORE(core_base, core_num, pkg_num);
-        p = GET_PKG(pkg_base, pkg_num);
+        t = GET_THREAD(thread_base, thread_id, core_id, node_id, pkg_id);
+        c = GET_CORE(core_base, core_id, node_id, pkg_id);
+        p = GET_PKG(pkg_base, pkg_id);
 
         t->cpu_id = cpu_id;
-        if (thread_num == 0) {
+        if (thread_id == 0) {
                 t->flags |= CPU_IS_FIRST_THREAD_IN_CORE;
                 if (cpu_is_first_core_in_package(cpu_id))
                         t->flags |= CPU_IS_FIRST_CORE_IN_PACKAGE;
         }
 
-        c->core_id = core_num;
-        p->package_id = pkg_num;
+        c->core_id = core_id;
+        p->package_id = pkg_id;
 }
 
 
 int initialize_counters(int cpu_id)
 {
-        int my_thread_id, my_core_id, my_package_id;
-
-        my_package_id = get_physical_package_id(cpu_id);
-        my_core_id = get_core_id(cpu_id);
-        my_thread_id = get_cpu_position_in_core(cpu_id);
-        if (!my_thread_id)
-                topo.num_cores++;
-
-        init_counter(EVEN_COUNTERS, my_thread_id, my_core_id, my_package_id, cpu_id);
-        init_counter(ODD_COUNTERS, my_thread_id, my_core_id, my_package_id, cpu_id);
+        init_counter(EVEN_COUNTERS, cpu_id);
+        init_counter(ODD_COUNTERS, cpu_id);
         return 0;
 }
 
@@ -4630,7 +5009,7 @@ int get_and_dump_counters(void)
 }
 
 void print_version() {
-        fprintf(outf, "turbostat version 17.06.23"
+        fprintf(outf, "turbostat version 18.06.01"
                 " - Len Brown <lenb@kernel.org>\n");
 }
 
@@ -4661,7 +5040,7 @@ int add_counter(unsigned int msr_num, char *path, char *name,
                 msrp->next = sys.tp;
                 sys.tp = msrp;
                 sys.added_thread_counters++;
-                if (sys.added_thread_counters > MAX_ADDED_COUNTERS) {
+                if (sys.added_thread_counters > MAX_ADDED_THREAD_COUNTERS) {
                         fprintf(stderr, "exceeded max %d added thread counters\n",
                                 MAX_ADDED_COUNTERS);
                         exit(-1);
@@ -4820,7 +5199,7 @@ void probe_sysfs(void)
         if (!DO_BIC(BIC_sysfs))
                 return;
 
-        for (state = 10; state > 0; --state) {
+        for (state = 10; state >= 0; --state) {
 
                 sprintf(path, "/sys/devices/system/cpu/cpu%d/cpuidle/state%d/name",
                         base_cpu, state);
@@ -4847,7 +5226,7 @@ void probe_sysfs(void)
                                 FORMAT_PERCENT, SYSFS_PERCPU);
         }
 
-        for (state = 10; state > 0; --state) {
+        for (state = 10; state >= 0; --state) {
 
                 sprintf(path, "/sys/devices/system/cpu/cpu%d/cpuidle/state%d/name",
                         base_cpu, state);
@@ -4960,34 +5339,6 @@ error:
         exit(-1);
 }
 
-int shown;
-/*
- * parse_show_hide() - process cmdline to set default counter action
- */
-void parse_show_hide(char *optarg, enum show_hide_mode new_mode)
-{
-        /*
-         * --show: show only those specified
-         *  The 1st invocation will clear and replace the enabled mask
-         *  subsequent invocations can add to it.
-         */
-        if (new_mode == SHOW_LIST) {
-                if (shown == 0)
-                        bic_enabled = bic_lookup(optarg, new_mode);
-                else
-                        bic_enabled |= bic_lookup(optarg, new_mode);
-                shown = 1;
-
-                return;
-        }
-
-        /*
-         * --hide: do not show those specified
-         *  multiple invocations simply clear more bits in enabled mask
-         */
-        bic_enabled &= ~bic_lookup(optarg, new_mode);
-
-}
 
 void cmdline(int argc, char **argv)
 {
@@ -4998,7 +5349,9 @@ void cmdline(int argc, char **argv)
                 {"cpu",         required_argument,      0, 'c'},
                 {"Dump",        no_argument,            0, 'D'},
                 {"debug",       no_argument,            0, 'd'},        /* internal, not documented */
+                {"enable",      required_argument,      0, 'e'},
                 {"interval",    required_argument,      0, 'i'},
+                {"num_iterations",      required_argument,      0, 'n'},
                 {"help",        no_argument,            0, 'h'},
                 {"hide",        required_argument,      0, 'H'},        // meh, -h taken by --help
                 {"Joules",      no_argument,            0, 'J'},
@@ -5014,7 +5367,7 @@ void cmdline(int argc, char **argv)
 
         progname = argv[0];
 
-        while ((opt = getopt_long_only(argc, argv, "+C:c:Ddhi:JM:m:o:qST:v",
+        while ((opt = getopt_long_only(argc, argv, "+C:c:Dde:hi:Jn:o:qST:v",
                                 long_options, &option_index)) != -1) {
                 switch (opt) {
                 case 'a':
@@ -5026,11 +5379,20 @@ void cmdline(int argc, char **argv)
                 case 'D':
                         dump_only++;
                         break;
+                case 'e':
+                        /* --enable specified counter */
+                        bic_enabled |= bic_lookup(optarg, SHOW_LIST);
+                        break;
                 case 'd':
                         debug++;
+                        ENABLE_BIC(BIC_DISABLED_BY_DEFAULT);
                         break;
                 case 'H':
-                        parse_show_hide(optarg, HIDE_LIST);
+                        /*
+                         * --hide: do not show those specified
+                         *  multiple invocations simply clear more bits in enabled mask
+                         */
+                        bic_enabled &= ~bic_lookup(optarg, HIDE_LIST);
                         break;
                 case 'h':
                 default:
@@ -5046,7 +5408,8 @@ void cmdline(int argc, char **argv)
                                         exit(2);
                                 }
 
-                                interval_ts.tv_sec = interval;
+                                interval_tv.tv_sec = interval_ts.tv_sec = interval;
+                                interval_tv.tv_usec = (interval - interval_tv.tv_sec) * 1000000;
                                 interval_ts.tv_nsec = (interval - interval_ts.tv_sec) * 1000000000;
                         }
                         break;
@@ -5054,6 +5417,7 @@ void cmdline(int argc, char **argv)
                         rapl_joules++;
                         break;
                 case 'l':
+                        ENABLE_BIC(BIC_DISABLED_BY_DEFAULT);
                         list_header_only++;
                         quiet++;
                         break;
@@ -5063,8 +5427,26 @@ void cmdline(int argc, char **argv)
                 case 'q':
                         quiet = 1;
                         break;
+                case 'n':
+                        num_iterations = strtod(optarg, NULL);
+
+                        if (num_iterations <= 0) {
+                                fprintf(outf, "iterations %d should be positive number\n",
+                                        num_iterations);
+                                exit(2);
+                        }
+                        break;
                 case 's':
-                        parse_show_hide(optarg, SHOW_LIST);
+                        /*
+                         * --show: show only those specified
+                         *  The 1st invocation will clear and replace the enabled mask
+                         *  subsequent invocations can add to it.
+                         */
+                        if (shown == 0)
+                                bic_enabled = bic_lookup(optarg, SHOW_LIST);
+                        else
+                                bic_enabled |= bic_lookup(optarg, SHOW_LIST);
+                        shown = 1;
                         break;
                 case 'S':
                         summary_only++;
diff --git a/tools/power/x86/x86_energy_perf_policy/Makefile b/tools/power/x86/x86_energy_perf_policy/Makefile
index 2447b1bbaacf..f4534fb8b951 100644
--- a/tools/power/x86/x86_energy_perf_policy/Makefile
+++ b/tools/power/x86/x86_energy_perf_policy/Makefile
@@ -24,5 +24,5 @@ install : x86_energy_perf_policy
         install -d  $(DESTDIR)$(PREFIX)/bin
         install $(BUILD_OUTPUT)/x86_energy_perf_policy $(DESTDIR)$(PREFIX)/bin/x86_energy_perf_policy
         install -d  $(DESTDIR)$(PREFIX)/share/man/man8
-        install x86_energy_perf_policy.8 $(DESTDIR)$(PREFIX)/share/man/man8
+        install -m 644 x86_energy_perf_policy.8 $(DESTDIR)$(PREFIX)/share/man/man8