Drastic Disk Performance Drop Caused by NVMe Power Management in Proxmox VE
Drastic Disk Performance Drop Caused by NVMe Power Management in Proxmox VE
A few days ago I deployed a two-node proxmox ve cluster. It worked fine at first, but after a day, when I logged in to the dell r720 host over ssh, I found the shell responding very slowly, and going into the linux container on it was even worse — every time I typed a command, zsh would freeze for quite a while.
I thought it was a problem with zsh, so I switched to bash and fish, but they were only slightly better. Opening a local file with vim was also very slow — could the disk be the problem? The system disk on this dell host is a West Digital SN750 1T, which by all accounts shouldn’t be slow, so I ran a test:
|
|
I had no idea until I tested it, and the result gave me a shock:
|
|
That was abnormally slow. Neither sudo dmesg | grep -i error nor sudo grep -i 'error' /var/log/syslog revealed any problem. It is the system disk and hadn’t been touched since installation.
I tried the good old reboot, and found that booting into the system took a very long time. Checking with smarttools showed no bad sectors and no errors at all. I was frustrated. After searching on google and bing for a long time without much to show, one answer mentioned in passing that it might be an apm power management issue and that hdparm should be used to inspect the settings. I looked for a long time, but it had no effect on my disk and just threw errors.
After thinking for a while, I realized I had been searching for ssd, so I switched the keyword to nvme and searched again.
I found a new tool: nvme-ctl. Someone said to use it to check the nvme power management configuration. I checked its manpage and saw that nvme get-feature is used to inspect power management, but it didn’t say what the --feature-id parameter should be, so I kept looking.
On nvme’s official site I found one: feature-id=0x02 refers to power management. I tried running it:
|
|
And got the result:
|
|
There was a result, but what exactly is 0x03? I remembered that when I checked earlier with smarttools there seemed to be a list of power states, so I tried it, and sure enough there was one:
|
|
As you can see, 0x03 is drastically different from the previous ones — it’s basically asleep. I set it to 0x02 to try:
|
|
The result was satisfying:
|
|
222 MB/s — finally it felt like real read/write speed. I changed it to 0x01:
|
|
Result:
|
|
After struggling with this all evening and about to reinstall the system, I found the answer by talking step by step with new bing’s chat gpt. As capable as chat gpt is, asking the right question matters too — it doesn’t have the human knack for guessing.
At last, speed freedom achieved.