1

Hardware:

  • 4 Intel HNS2600TPR in one chassis on 2 power cords,
  • Each with 1 Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz,
  • 128 GB RAM.
[root@sigma02 Linux_X64]# dmidecode
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.
80 structures occupying 4366 bytes.
Table at 0x7A4EC000.

Handle 0x0000, DMI type 133, 12 bytes
OEM-specific Type
        Header and Data:
                85 0C 00 00 00 B0 67 7B 00 40 00 00

Handle 0x0001, DMI type 0, 24 bytes
BIOS Information
        Vendor: Intel Corporation
        Version: SE5C610.86B.01.01.2024.041020181059
        Release Date: 04/10/2018
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 16 MB
        Characteristics:
                PCI is supported
                PNP is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                EDD is supported
                5.25"/1.2 MB floppy services are supported (int 13h)
                3.5"/720 kB floppy services are supported (int 13h)
                3.5"/2.88 MB floppy services are supported (int 13h)
                Print screen service is supported (int 5h)
                8042 keyboard services are supported (int 9h)
                Serial services are supported (int 14h)
                Printer services are supported (int 17h)
                CGA/mono video services are supported (int 10h)
                ACPI is supported
                USB legacy is supported
                LS-120 boot is supported
                ATAPI Zip drive boot is supported
                BIOS boot specification is supported
                Function key-initiated network boot is supported
                Targeted content distribution is supported
                UEFI is supported
        BIOS Revision: 0.0
        Firmware Revision: 0.0

Handle 0x0002, DMI type 1, 27 bytes
System Information
        Manufacturer: Intel Corporation
        Product Name: S2600TPR
        Version: HNS2600TPR
        Serial Number: BQTP94490080
        UUID: 80c1fd42-f1ec-e811-906e-0017a4403562
        Wake-up Type: Power Switch
        SKU Number: SKU Number
        Family: Family

Handle 0x0003, DMI type 2, 17 bytes
Base Board Information
        Manufacturer: Intel Corporation
        Product Name: S2600TPR
        Version: H26989-274
        Serial Number: BQTP84500150
        Asset Tag: Base Board Asset Tag
        Features:
                Board is a hosting board
                Board is replaceable
        Location In Chassis: Part Component
        Chassis Handle: 0x0000
        Type: Motherboard
        Contained Object Handles: 0

Handle 0x0004, DMI type 3, 24 bytes
Chassis Information
        Manufacturer: ...............................
        Type: Rack Mount Chassis
        Lock: Not Present
        Version: ..................
        Serial Number: ..................
        Asset Tag: ....................
        Boot-up State: Safe
        Power Supply State: Safe
        Thermal State: Safe
        Security Status: None
        OEM Information: 0x00000000
        Height: Unspecified
        Number Of Power Cords: Unspecified
        Contained Elements: 0
        SKU Number: Not Specified

Handle 0x000A, DMI type 11, 5 bytes
OEM Strings
        String 1: To Be Filled By O.E.M.

Handle 0x000C, DMI type 13, 22 bytes
BIOS Language Information
        Language Description Format: Abbreviated
        Installable Languages: 1
                enUS
        Currently Installed Language: enUS

Handle 0x000D, DMI type 27, 15 bytes
Cooling Device
        Temperature Probe Handle: 0x000B
        Type: Fan
        Status: OK
        Cooling Unit Group: 1
        OEM-specific Information: 0x00000000
        Nominal Speed: Unknown Or Non-rotating
        Description: Not Specified

Handle 0x000E, DMI type 28, 22 bytes
Temperature Probe
        Description: LM78A
        Location: System Management Module
        Status: <OUT OF SPEC>
        Maximum Value: Unknown
        Minimum Value: Unknown
        Resolution: Unknown
        Tolerance: Unknown
        Accuracy: Unknown
        OEM-specific Information: 0x00000000
        Nominal Value: Unknown

Handle 0x000F, DMI type 32, 11 bytes
System Boot Information
        Status: No errors detected

Handle 0x0010, DMI type 34, 11 bytes
Management Device
        Description: UNKNOWN
        Type: Unknown
        Address: 0x00000000
        Address Type: Unknown

Handle 0x0011, DMI type 35, 11 bytes
Management Device Component
        Description: To Be Filled By O.E.M.
        Management Device Handle: 0x000D
        Component Handle: 0x000A
        Threshold Handle: 0x000F

Handle 0x0012, DMI type 36, 16 bytes
Management Device Threshold Data

Handle 0x0014, DMI type 24, 5 bytes
Hardware Security
        Power-On Password Status: Not Implemented
        Keyboard Password Status: Not Implemented
        Administrator Password Status: Disabled
        Front Panel Reset Status: Disabled

Handle 0x0018, DMI type 39, 22 bytes
System Power Supply
        Power Unit Group: 1
        Location: To Be Filled By O.E.M.
        Name: To Be Filled By O.E.M.
        Manufacturer: To Be Filled By O.E.M.
        Serial Number: To Be Filled By O.E.M.
        Asset Tag: To Be Filled By O.E.M.
        Model Part Number: To Be Filled By O.E.M.
        Revision: To Be Filled By O.E.M.
        Max Power Capacity: Unknown
        Status: Present, Unknown
        Type: Unknown
        Input Voltage Range Switching: Unknown
        Plugged: Yes
        Hot Replaceable: No
        Input Voltage Probe Handle: 0x0000
        Cooling Device Handle: 0x000A
        Input Current Probe Handle: 0x0000

Handle 0x0019, DMI type 7, 19 bytes
Cache Information
        Socket Designation: L1-Cache
        Configuration: Enabled, Not Socketed, Level 1
        Operational Mode: Write Back
        Location: Internal
        Installed Size: 768 kB
        Maximum Size: 768 kB
        Supported SRAM Types:
                Synchronous
        Installed SRAM Type: Synchronous
        Speed: Unknown
        Error Correction Type: Single-bit ECC
        System Type: Instruction
        Associativity: 8-way Set-associative

Handle 0x001A, DMI type 7, 19 bytes
Cache Information
        Socket Designation: L2-Cache
        Configuration: Enabled, Not Socketed, Level 2
        Operational Mode: Varies With Memory Address
        Location: Internal
        Installed Size: 3072 kB
        Maximum Size: 3072 kB
        Supported SRAM Types:
                Synchronous
        Installed SRAM Type: Synchronous
        Speed: Unknown
        Error Correction Type: Single-bit ECC
        System Type: Unified
        Associativity: 8-way Set-associative

Handle 0x001B, DMI type 7, 19 bytes
Cache Information
        Socket Designation: L3-Cache
        Configuration: Enabled, Not Socketed, Level 3
        Operational Mode: Varies With Memory Address
        Location: Internal
        Installed Size: 30720 kB
        Maximum Size: 30720 kB
        Supported SRAM Types:
                Synchronous
        Installed SRAM Type: Synchronous
        Speed: Unknown
        Error Correction Type: Single-bit ECC
        System Type: Unified
        Associativity: 20-way Set-associative

Handle 0x001C, DMI type 4, 48 bytes
Processor Information
        Socket Designation: CPU1
        Type: Central Processor
        Family: Xeon
        Manufacturer: Intel(R) Corporation
        ID: F1 06 04 00 FF FB EB BF
        Signature: Type 0, Family 6, Model 79, Stepping 1
        Flags:
                FPU (Floating-point unit on-chip)
                VME (Virtual mode extension)
                DE (Debugging extension)
                PSE (Page size extension)
                TSC (Time stamp counter)
                MSR (Model specific registers)
                PAE (Physical address extension)
                MCE (Machine check exception)
                CX8 (CMPXCHG8 instruction supported)
                APIC (On-chip APIC hardware supported)
                SEP (Fast system call)
                MTRR (Memory type range registers)
                PGE (Page global enable)
                MCA (Machine check architecture)
                CMOV (Conditional move instruction supported)
                PAT (Page attribute table)
                PSE-36 (36-bit page size extension)
                CLFSH (CLFLUSH instruction supported)
                DS (Debug store)
                ACPI (ACPI supported)
                MMX (MMX technology supported)
                FXSR (FXSAVE and FXSTOR instructions supported)
                SSE (Streaming SIMD extensions)
                SSE2 (Streaming SIMD extensions 2)
                SS (Self-snoop)
                HTT (Multi-threading)
                TM (Thermal monitor supported)
                PBE (Pending break enabled)
        Version: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
        Voltage: 1.8 V
        External Clock: 100 MHz
        Max Speed: 4000 MHz
        Current Speed: 2200 MHz
        Status: Populated, Enabled
        Upgrade: Socket LGA2011-3
        L1 Cache Handle: 0x0019
        L2 Cache Handle: 0x001A
        L3 Cache Handle: 0x001B
        Serial Number:
        Asset Tag:
        Part Number:
        Core Count: 12
        Core Enabled: 12
        Thread Count: 24
        Characteristics:
                64-bit capable
                Multi-Core
                Hardware Thread
                Execute Protection
                Enhanced Virtualization
                Power/Performance Control

Handle 0x001D, DMI type 4, 48 bytes
Processor Information
        Socket Designation: CPU2
        Type: Central Processor
        Family: <OUT OF SPEC>
        Manufacturer: Not Specified
        ID: 00 00 00 00 00 00 00 00
        Version: Not Specified
        Voltage: Unknown
        External Clock: Unknown
        Max Speed: 4000 MHz
        Current Speed: Unknown
        Status: Unpopulated
        Upgrade: Socket LGA2011-3
        L1 Cache Handle: Not Provided
        L2 Cache Handle: Not Provided
        L3 Cache Handle: Not Provided
        Serial Number: Not Specified
        Asset Tag: Not Specified
        Part Number: Not Specified
        Characteristics: None

Handle 0x001E, DMI type 16, 23 bytes
Physical Memory Array
        Location: System Board Or Motherboard
        Use: System Memory
        Error Correction Type: Single-bit ECC
        Maximum Capacity: 128 GB
        Error Information Handle: Not Provided
        Number Of Devices: 4

Handle 0x001F, DMI type 19, 31 bytes
Memory Array Mapped Address
        Starting Address: 0x00000000000
        Ending Address: 0x00FFFFFFFFF
        Range Size: 64 GB
        Physical Array Handle: 0x001E
        Partition Width: 4

Handle 0x0020, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x001E
        Error Information Handle: Not Provided
        Total Width: 72 bits
        Data Width: 64 bits
        Size: 32 GB
        Form Factor: DIMM
        Set: None
        Locator: DIMM_A1
        Bank Locator: NODE 1
        Type: DDR4
        Type Detail: Synchronous
        Speed: 2400 MT/s
        Manufacturer: Kingston
        Serial Number: 042424BA
        Asset Tag:
        Part Number: 9965640-035.C00G
        Rank: 2
        Configured Memory Speed: 2400 MT/s
        Minimum Voltage: Unknown
        Maximum Voltage: Unknown
        Configured Voltage: Unknown

Handle 0x0021, DMI type 20, 35 bytes
Memory Device Mapped Address
        Starting Address: 0x00000000000
        Ending Address: 0x007FFFFFFFF
        Range Size: 32 GB
        Physical Device Handle: 0x0020
        Memory Array Mapped Address Handle: 0x001F
        Partition Row Position: 1

Handle 0x0022, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x001E
        Error Information Handle: Not Provided
        Total Width: Unknown
        Data Width: Unknown
        Size: No Module Installed
        Form Factor: DIMM
        Set: None
        Locator: DIMM_A2
        Bank Locator: NODE 1
        Type: DDR4
        Type Detail: Synchronous
        Speed: Unknown
        Manufacturer: NO DIMM
        Serial Number: NO DIMM
        Asset Tag:
        Part Number: NO DIMM
        Rank: Unknown
        Configured Memory Speed: Unknown
        Minimum Voltage: Unknown
        Maximum Voltage: Unknown
        Configured Voltage: Unknown

Handle 0x0023, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x001E
        Error Information Handle: Not Provided
        Total Width: 72 bits
        Data Width: 64 bits
        Size: 32 GB
        Form Factor: DIMM
        Set: None
        Locator: DIMM_B1
        Bank Locator: NODE 1
        Type: DDR4
        Type Detail: Synchronous
        Speed: 2400 MT/s
        Manufacturer: Kingston
        Serial Number: 042418D8
        Asset Tag:
        Part Number: 9965640-035.C00G
        Rank: 2
        Configured Memory Speed: 2400 MT/s
        Minimum Voltage: Unknown
        Maximum Voltage: Unknown
        Configured Voltage: Unknown

Handle 0x0024, DMI type 20, 35 bytes
Memory Device Mapped Address
        Starting Address: 0x00800000000
        Ending Address: 0x00FFFFFFFFF
        Range Size: 32 GB
        Physical Device Handle: 0x0023
        Memory Array Mapped Address Handle: 0x001F
        Partition Row Position: 2

Handle 0x0025, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x001E
        Error Information Handle: Not Provided
        Total Width: Unknown
        Data Width: Unknown
        Size: No Module Installed
        Form Factor: DIMM
        Set: None
        Locator: DIMM_B2
        Bank Locator: NODE 1
        Type: DDR4
        Type Detail: Synchronous
        Speed: Unknown
        Manufacturer: NO DIMM
        Serial Number: NO DIMM
        Asset Tag:
        Part Number: NO DIMM
        Rank: Unknown
        Configured Memory Speed: Unknown
        Minimum Voltage: Unknown
        Maximum Voltage: Unknown
        Configured Voltage: Unknown

[...]

Handle 0x004A, DMI type 9, 17 bytes
System Slot Information
        Designation: Riser 4, slot 1
        Type: x16 PCI Express 3 x16
        Current Usage: Available
        Length: Long
        ID: 0
        Characteristics:
                3.3 V is provided
                PME signal is supported
        Bus Address: 0000:80:02.0

Handle 0x004B, DMI type 10, 14 bytes
On Board Device 1 Information
        Type: Video
        Status: Enabled
        Description: ServerEngines Pilot III
On Board Device 2 Information
        Type: Ethernet
        Status: Enabled
        Description: Intel I350
On Board Device 3 Information
        Type: SATA Controller
        Status: Enabled
        Description: PCH Integrated SATA Controller
On Board Device 4 Information
        Type: SATA Controller
        Status: Enabled
        Description: PCH Integrated sSATA Controller
On Board Device 5 Information
        Type: Ethernet
        Status: Enabled
        Description: Intel I350

Handle 0x000B, DMI type 12, 5 bytes
System Configuration Options
        Option 1: J7C2: Close to boot with MFG mode
        Option 2: J7B2 2-3: Close for BIOS Image swap
        Option 3: J5D2 2-3: ME force update
        Option 4: J7A7 2-3: Recovery BIOS
        Option 5: J7A6 2-3: Clear password
        Option 6: J7A3 2-3: Clear CMOS register
        Option 7: J7A2 2-3: Force BMC update
        Option 8: J7B3 1-2: PLD program enable
        Option 9: J6C2: RAID key

Handle 0x0015, DMI type 41, 11 bytes
Onboard Device
        Reference Designation: ServerEngines Pilot III
        Type: Video
        Status: Enabled
        Type Instance: 1
        Bus Address: 0000:05:00.0

Handle 0x0016, DMI type 41, 11 bytes
Onboard Device
        Reference Designation: Intel I350
        Type: Ethernet
        Status: Enabled
        Type Instance: 1
        Bus Address: 0000:06:00.0

Handle 0x0017, DMI type 41, 11 bytes
Onboard Device
        Reference Designation: PCH Integrated SATA Controller
        Type: SATA Controller
        Status: Enabled
        Type Instance: 1
        Bus Address: 0000:00:1f.2

Handle 0x004C, DMI type 41, 11 bytes
Onboard Device
        Reference Designation: PCH Integrated sSATA Controller
        Type: SATA Controller
        Status: Enabled
        Type Instance: 2
        Bus Address: 0000:00:11.4

Handle 0x004D, DMI type 41, 11 bytes
Onboard Device
        Reference Designation: Intel I350
        Type: Ethernet
        Status: Enabled
        Type Instance: 2
        Bus Address: 0000:06:00.1

Handle 0x004E, DMI type 148, 48 bytes
OEM-specific Type
        Header and Data:
                94 30 4E 00 02 01 02 03 04 05 06 07 08 09 0A 0B
                0C 0D 0E 0F 10 11 12 13 14 15 16 17 18 19 1A 1B
                1C 1D 1E 1F 20 21 22 23 00 00 00 00 00 00 00 00
        Strings:
                SE5C610.86B.01.01.2024.041020181059
                 1.81.11142
                3.1.3.43
                SDR Package 1.17
                FRU Ver 1.00
                SDR File 1.17
                M6F306F2_0000003C
                MEF406F1_0B00002A
                N/A
                BF176FCB
                19A708C8
                4.3.0
                4.3.0
                0.9.77
                N/A
                N/A
                N/A
                N/A
                N/A
                72DACB44
                8E0AAD10
                2F3F5F52
                80C5846D
                6AF7E746
                N/A
                3E8576C0
                N/A
                291CD502
                04BD7E53
                46D789BD
                AA029E92
                N/A
                N/A
                N/A
                N/A

Handle 0xFEFF, DMI type 127, 4 bytes
End Of Table

Software:

  • Running CoreOS with different releases:

2512.2.0 Release Date: May 19, 2020 kernel: 4.19.123 rkt: 1.30.0 docker: 18.06.3 etcd: 3.3.20 systemd: 241 Ignition: 0.34.0

2345.3.0 Release Date: March 2, 2020 kernel: 4.19.106 rkt: 1.30.0 docker: 18.06.3 etcd: 3.3.18 systemd: 241 Ignition: 0.33.0

  • Kubernetes 1.17.0

From time to time, some nodes go down to 400 MHz on all CPU cores as follows:

sigma01 sigma # cat /proc/cpuinfo
processor       : 23
vendor_id       : GenuineIntel
cpu family      : 6
model           : 79
model name      : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
stepping        : 1
microcode       : 0xb000038
cpu MHz         : 412.535
cache size      : 30720 KB
physical id     : 0
siblings        : 24
core id         : 13
cpu cores       : 12
apicid          : 27
initial apicid  : 27
fpu             : yes
fpu_exception   : yes
cpuid level     : 20
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
bogomips        : 4389.81
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:
Every 2.0s: cat /proc/cpuinfo | grep MHz                                                                                  sigma01: Fri May 22 13:44:33 2020

cpu MHz         : 422.084
cpu MHz         : 413.291
cpu MHz         : 420.521
cpu MHz         : 421.059
cpu MHz         : 417.286
cpu MHz         : 417.869
cpu MHz         : 419.568
cpu MHz         : 413.913
cpu MHz         : 416.606
cpu MHz         : 416.767
cpu MHz         : 418.188
cpu MHz         : 422.938
cpu MHz         : 413.258
cpu MHz         : 414.553
cpu MHz         : 409.921
cpu MHz         : 407.358
cpu MHz         : 410.833
cpu MHz         : 413.726
cpu MHz         : 417.325
cpu MHz         : 414.957
cpu MHz         : 411.737
cpu MHz         : 415.100
cpu MHz         : 413.458
cpu MHz         : 411.024
sigma03 sigma # ls /sys/devices/system/cpu/cpufreq/policy0/
affected_cpus  cpuinfo_max_freq  cpuinfo_min_freq  cpuinfo_transition_latency  related_cpus  scaling_available_governors  scaling_cur_freq  scaling_driver  scaling_governor  scaling_max_freq  scaling_min_freq  scaling_setspeed

sigma03 sigma # cat /sys/devices/system/cpu/cpufreq/policy0/scaling_driver
intel_pstate

sigma03 sigma # cat /sys/devices/system/cpu/cpufreq/policy0/scaling_setspeed
<unsupported>
[root@sigma01 ~]# cpupower frequency-info
sh: modprobe: command not found
analyzing CPU 0:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 1.20 GHz - 2.90 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 1.20 GHz and 2.90 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 426 MHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes

In BIOS, the HNS2600TPR is set to Performance mode as for power management, and as for Fans, they are set to Performance as well. SpeedStep is enabled.

Running the following commands for a while does not solve the problem, but the CPU MHz values will be 411 without much fluctuation.

echo $(seq 1 24) | xargs -P 24 -n 1 sh -c 'while :;do :; done' or stress --cpu 24

Without load, the CPU MHz values fluctuate between 400 and 430.

Then I disabled intel_pstate and intel_idle driver using:

set linux_append="$linux_append intel_idle.max_cstate=0 processor.max_cstate=0 intel_pstate=passive"

The current driver is now intel_cpufreq.

[root@sigma02 ~]# cpupower frequency-info
sh: modprobe: command not found
analyzing CPU 0:
  driver: intel_cpufreq
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency: 20.0 us
  hardware limits: 1.20 GHz - 2.90 GHz
  available cpufreq governors: performance
  current policy: frequency should be within 1.20 GHz and 2.00 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 1.20 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes

The C-state times is now missing from /sys/devices/cpu/..., which counts how many times each sleep state is used. This did not improve in any way.

intel_pstate is correctly set:

sigma04 sigma # cat /sys/devices/system/cpu/intel_pstate/max_perf_pct
100

When kernel parameters are changed and servers are rebooted, for a short period of time, they use correct CPU frequencies, around 2500 MHz, but in an unpredictable way.

I'm not sure that this information could be enough for anyone to help to fix the problem, but some hints would be useful in how to systematically drill down and look for cues.

Currently 3 nodes out of 4 has normal CPU clock around 2.5 GHz, 1 node is stuck at 400 Mhz. Some other day, other nodes are stuck at 400 MHz.

While CPUs are at 400 MHz, their temperature is around 25-30 C.

Update

Found something, this just happened: - When all 4 nodes are throttled to 400 MHz. - Two nodes are issued sudo reboot at the same time. - Two remaining nodes go up to max speed as follows:

Every 2.0s: cat /proc/cpuinfo | grep MHz                                                                                                                                                                    sigma02: Fri May 22 20:16:53 2020

cpu MHz         : 2494.244
cpu MHz         : 2507.850
cpu MHz         : 2502.095
cpu MHz         : 2494.222
cpu MHz         : 2501.193
cpu MHz         : 2494.445
[...]
  • When the two nodes come back online, all nodes are throttled to 400 Mhz.

As mentioned, these nodes are "in one chassis on 2 power cords". Could this mean that one power cord is malfunctioning? Why does this happen once every 1-2 months? Why is it so hectic? It certainly affected by whether other nodes are online or not, but not clear. For example, now, 2 nodes are online, 2 on the maximum speed of 2500 MHz, while 1 is booting up and 1 is half-throttled:

Every 2.0s: cat /proc/cpuinfo | grep MHz                                                                                  sigma03: Fri May 22 20:22:11 2020

cpu MHz         : 1197.514
cpu MHz         : 1197.706
cpu MHz         : 1197.370
cpu MHz         : 1197.358

Moreover, irrespective or node's current load, state or uptime, when all nodes go to 400 MHz at the same time.

Could there be any other issue? Malfunctioning network?

Help is much appreciated!

11
  • Hi! I added more information on your request to the question itself as well. Here are the results inline: Running the following commands for a while does not solve the problem, but the CPU MHz values will be 411 without much fluctuation. echo $(seq 1 24) | xargs -P 24 -n 1 sh -c 'while :;do :; done' or stress -n 24 Without load, the CPU MHz values fluctuate between 400 and 430. – Dyin May 22 '20 at 14:41
  • Ok, I was trying to rule out the obvious, and indeed this ruled out the obvious: the limit doesn't appear to be caused by systems being idle. I don't have an idea thus on how to solve your actual problem. – A.B May 22 '20 at 14:43
  • Thanks, if you have any more ideas, help is much appreciated! I have no clue on how to drill down more into the problem, or whether it is a CPU scaling governor problem. If I could answer the following questions, it would be helpful: Do I have to fix something in the BIOS? In the kernel? In the CoreOS distribution? – Dyin May 22 '20 at 14:52
  • If you can afford to remove a node from production (drain) and boot it with various Live CD linux distributions using various kernels, it would help rule out kernel and distribution .. – A.B May 22 '20 at 14:56
  • Check out what is the setting in /sys/devices/system/cpu/intel_pstate/max_perf_pct. If you set it to 0, your CPU will be stuck on lowest frequency. – konradmb May 22 '20 at 16:34
0

When you mess with intel_pstate, intel_idle, BIOS settings and it still does not work, there is probably something deeper.

You may want to check your power supply. Most chassis provide a backup power supply, and even with that going in standby, the primary may fail to provide the sufficient power for the nodes.

Pulling out the primary power supply on the fixed the problem. All nodes are operational at 2500 MHz. Plugging the power supply back and now it blinks in yellow light. It is malfunctioning most probably, but the secondary did not take over during all of this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.