At the AMD Tech Day I had the opportunity to get some hands-on time with AMD Ryzen and a few different systems. Thanks to a wonderful Linux-Bench version on a USB 3.0 stick, I was able to get numbers to publish. It was a painful process and required several reboots until I got it all working. After hammering the four test configurations we have, I came to the conclusion that the actual issue was not a pre-retail chip. Instead, it is just running AMD Ryzen using the current version of Ubuntu. We had issues with Ubuntu 14.04.5 LTS (the current latest) that is the STH standard. We also had issues with Ubuntu 16.04.2 LTS. These were present in both Desktop and Server versions of Ubuntu. That is enough different configurations where we are now satisfied that it is not a fluke, but instead an issue with AMD Ryzen and older kernels.
Update: We published Stop constant CentOS 7.3 crashes with AMD Ryzen using Kernel 4.10 as a piece for those looking to fix issues on CentOS 7.
As you can see, we even built two test platforms with different CPUs just to ensure it was not a CPU model-specific issue we were seeing. One had the Ryzen 7 1700
AMD Ryzen with Ubuntu – Why your system is crashing
There have been numerous reports of SMT issues and it is a known fix that got rolled out in the Linux 4.10 and 4.11 kernels as we mentioned in our release piece. We have four systems up already with different CPUs and motherboards that are exhibiting this problem. We have tried simultaneous wall clock timers to see when the systems fail but it does not seem to be a specific time trigger. Here is an example of what you will see at some point running AMD Ryzen with Ubuntu 14.04.5 AMD64 Server:
At this point using reboot methods such as ctrl + alt + del will likely not work and will require a hard reboot.
How to fix AMD Ryzen with Ubuntu
Since as of release day (2 March 2017) the latest stable kernel posted on kernel.org is 4.10.1, which was released on 26 February 2017, we can use that to upgrade our Ubuntu installations and get a working, stable, system. Here is what you can do:
nano kernelupdate.sh
In that paste the following text:
#!/bin/bash
cd /tmp
wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10.1/linux-headers-4.10.1-041001_4.10.1-041001.201702260735_all.deb
wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10.1/linux-headers-4.10.1-041001-generic_4.10.1-041001.201702260735_amd64.deb
wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10.1/linux-image-4.10.1-041001-generic_4.10.1-041001.201702260735_amd64.deb
echo Everything is downloaded. Time to install.
sudo dpkg -i linux-headers-4.10*.deb linux-image-4.10*.deb
echo Type sudo reboot to restart your system with the new kernel.
Now hit control x to exit and then y to save changes.
bash kernelupdate.sh
Let that download. It will ask you for sudo credentials. After it is complete, do sudo reboot as the screen will say to reboot the system and use your new Linux 4.10.1 kernel. You can verify using uname -r
Final Words
One plus side to this is that the newer kernel version also is giving us significantly more performance. Our advice, use whatever Linux flavor you want, but upgrade it to the latest stable kernel immediately if you want to use AMD Ryzen. Since we will undoubtedly get the question, this does not mean that the chips are not x86 and AMD64 compatible. It is simply an example of where development has been focused on how Intel does things. With AMD’s implementation, we simply need to apply an existing update.
It would be really great if you can file a bug about it here, so the fix can be landed into 14.04 / 16.04 / 16.10 as necessary – best if you can file so that you can potentially test the fix:
https://bugs.launchpad.net/ubuntu/+source/linux/+filebug
Patrick, curious if there was a reason why you didn’t add the ppa to apt-sources and then pin the version with apt hold? Has the advantage of notifying you when newer versions are available (I’m sure there will more Ryzen fixes coming down the pipe). Either way thanks for the update.
Thanks for this info servethehome. I am planning to build a home server/workstation with Ryzen running Ubuntu so I am glad you published this. I would like to have build with ECC memory, since Ryzen supports it and one of my use cases is having ZFS raid in this home server. It would be great, if you were able to put some built tohgether and test what components works seamlessly together in regard of ECC memory, motherboard and pherhaps NVMe disk running Ubuntu as main OS on Ryzen? Just a suggestion, but I would very appreciate it, and probably other readers as well. Thank you very much for your site.
Indeed. as Trent mentioned, please keep this central and file the bug at https://bugs.launchpad.net/ubuntu/+source/linux/+filebug
If everyone would post this on their own blog, it doesn’t help the community to get issues solved. All of us have to search over the internet if someone fixed it and mentioned it on his or her blog.
I happened across this article today as I was setting up lubuntu-16.04.1-desktop-amd64 on my new Ryzen 7 1800X box, and ran the script to update the kernel. It showed the correct 4.10.1-041001-generic kernel after rebooting. After this, I left it alone for a while, running HandBrake, and came back to find the machine completely unresponsive and not outputting a video signal. I power cycled it, and found in /var/log/syslog.1 a steady stream of hours of those errors as described above. Despite having updated, this error still seems to be there.
Anyone have any suggestions? I realize now I didn’t install the very latest Ubuntu available (I had the image lying around from last month). I’m really hoping I didn’t build a flakey system.
In my experience, it still happens. With 4.10.1 i had crashed for me every 2-3 hours, with 4.11-rc1 it crashed after 5 or 6 hours, referncing nouveau; i blacklisted it, running another uptime test (at 2 hours exactly right now). It will take some time for the platform to mature enough for 24/7 Ryzen functionality.
Thank you. Makes me feel better knowing it isn’t just me at least.
I didn’t want to install Windows on this machine (forced reboots and CPU/Memory-hogging built-in processes). But if that is what it takes to be stable today, I just have to bite the bullet.
Windows 10 made it stable. At least for the last 12 hours of continuous HandBrake encodes. It seems to be performing roughly 10% better, too. I was getting about 22.5 FPS with 1080p H.264 -> H.265 Medium q23 on Linux. 25.5 FPS on Windows. I’m sure things will even out in the upcoming months, but for anyone pondering a Ryzen Linux build today, I must advise against it.
Brian is that lununtu or Ubuntu?
We have 6x Ryzen systems setup (3 for our lab and 3 in DemoEval) and both with Ubuntu 16.04.2 and CentOS 7.3 with 4.10.1 they have been very stable. We have been running different workloads at 100% utilization and have not seen crashes.
I used lubuntu 16.04.1 as specified in my earlier post (I try pretty hard not to typo important details — a virtue more people should share). Something about that may have been what made it unstable — either the fact it was lubuntu or the fact it was a 16.04.1 disk image (not 16.04.2) that I used before upgrading the kernel.
Either way, Windows 10 has been perfectly stable since I installed it two days ago. As much as I wanted to reduce my dependence on Microsoft’s OS, installing it solved a number of other problems I was having (Teamviewer unable to connect to its servers, unstable x11vnc every other reboot, SMB permission problems), so I’m just going to stick with it. There is no substitute for experience, and I have vastly more experience bending Windows to my will than I do with Linux.
I’ve been trying to run Budgie using kernel 4.10.0-15-generic and I am getting the constant crashing and have been since I built this thing. It’s a Ryzen 7 1700 with ASRock X370 Fatal1ty Gaming K4 and G.Skill Trident Z 3000 running at 2133 and most recent UEFI/bios w/ AGESA update. I have NO idea what else to do and I’m seriously considering RMAing it all before time is up… SO frustrated!
Try using the Ubuntu 17.04 base. The release is scheduled for April 13, 2017 IIRC
Hi Patrick, I am using Ubuntu kernel 17.04 with Budgie, and I just upgraded the kernel to 4.11.0-041100rc5-generic and it is STILL crashing. Maybe an RMA is the only option left?
not Ubuntu kernel, Ubuntu base…
As a test, I installed openSUSE with Gnome and got the same behavior (intermittent crashing), but when I switched to openSUSE with KDE, inexplicably, the system became completely stable. I couldn’t find anything in the logs that might indicate why…
Unfortunately, that is the nature of Linux + Ryzen at the moment.
Another tip is using Ubuntu 17.04 that will be released this week, or Using 16.04 + Upgrading the kernel above will work.
We tried using updated kernels on Ubuntu 17.04, including 4.10.9 and 4.11-rc5, but it did not stop the weird input issues, constant reboots, or hard system freezes. (We have Asrock x370s and the system will either reboot, or more frequently, just shut off with a “00” flashing on the Dr. Debug display, indicating a CPU error.)
We are currently using openSUSE Tumbleweed. The Gnome desktop exhibited instability almost immediately after booting, but using KDE works pretty well, only crashing once in the past several days. I really wish I could figure out what’s causing it. Since it crashes so fast, it doesn’t write anything to any log files, so even tail -f is useless here. Interestingly, the only crash on KDE happened while running Chrome, which uses gtk, but I’m not positive if that’s related.
I’ve just ordered a Ryzen CPU for my 14.04.5 based Mythtv system.
I’ll try swapping the new MB and CPU in, but if that fails, I’ll do a fresh install. It’s using an old core i7 at the moment.
I’ve done the kernel upgrade in prep for the swap. Only hiccup is that Unity desktop wouldn’t load and kept kicking me back to the login prompt. I’m using the proprietary Nvidia driver for VDPAU video decoding.
I did an apt-get purge nvidia* which got my desktop back with the xorg driver. Then reinstalled the nvidia driver via the optional driver panel.
Actually, the reinstall of the Nvidia driver failed with the new kernel (from the Ubuntu repository).
I also tried via the .run file from the Nvidia website and it terminated with a failure message. I’ll just go with a fresh 17.04 install when my cpu arrives.
Any news on this?
I’ve just ordered a Ryzen 5 1400 based machine (Mobo Gigabyte MB350 Gaming 3) and I wonder if I’ll use the 17.04 with 4.10 kernel will I be fine or should I panic right about now
Go with 17.04 release and that works decently well.
I upgraded the kenel on 16.04 after a few days the system hung again.
I tried to install 17.04. I use and NVidia card, so the install does not work.
I run kubuntu.
I also swapped the MB ASUS-MSI, and replaced the memory and video card.
Still no joy.
I have syslog of the failure. Something to do with interrupts.
How do I report this to the kernel guys?
Any suggestions?
Still happening :(. I have my Ryzen since a week ago. Config is a Ryzen 1700 on a M350B Tomahawk motherboard, GT710 graphics card and 16GB Corsair compatible 2x modules.
I installed 16.04.2 Ubuntu with 4.8 kernel, it freezed with the tainted log. Upgraded to 4.11 and still happened, changed two times RAM modules (different brand) and happened again.
Then I discovered this post and downgraded to 4.10.1, then ran for a day and a half without problems… then I installed Virtualbox5 and hang up again (dunno if it had something to do with it but I suppose it’s worth mentioning) – removed Virtualbox and every component associated, reinstalled the kernel with your script in case it had changed anything… and still happened T_T. I’m getting desperate here.
Hi, I updated Ubuntu 16.04 with your kernel update script, but now my ZFS is nor working. When I boot up, my zfs filesystems are not there and any zfs commands in terminal say “The ZFS modules are not loaded.”
I suppose somehow the zfs version in ubuntu 16.04 is not compatible with the newer kernel? What can I do?
If you are using Ubuntu + ZFS you may want to try the newest 17.04 kernel for ZFS support.
I put a new box last week using a ASUS Prime x370 pro, AMD Ryzen 7 1700X, 2x8Gb Ballistix 2400 , running Ubuntu 17.04.
I started having several mouse/keyboard freezing events. I tried slowing down the memory speed to 2133 with no success. Then I upgrade the motherboard BIOS (originally 604) to 803 and the freezes just stopped.
Today we learned from Jorge that a BIOS update is the first thing that we do.
Are people finding 4.12 and 4.13 stable enough? I just got a Ryzen for a server at my workplace and want to know whether updating to one of those will make it stop hard locking up or whether I should look into getting an older AMD and/or go to Intel.
Well, kind of answered my own question and the answer seems to be no. Updated to 4.13 and had another hard lock up just now.
I have one running 4.10 in the data center that has uptime of over 4 months now at 70-75% load.
For servers, you want EPYC or Intel.
And another 6 hours later another hard lockup…mine’s going back to Microcenter.
I have debugged this issues with my limited knowledge. Post about my information collection is here: https://andresnamm.github.io/parallel/hadoop-materials/2017-12-13-OS-problems/ but for me things are happy in server when im using linux-generic-hwe-16.04-edge. (4.13 KERNEL) In desktop im also using Nvidia proprietary driver. linux-generic-hwe-16.04 (4.10.0 KERNEL) Freezes after 1 hour.
Not happy after 5 days. 2 out of 4 machines are down
Just bought a 1700 system with 3200 Ryzen optimized memory and an expensive motherboard. Record atm is 10 hours. Sometimes it freezes within an hour. Trying to run ubuntu 16.04. Uptime on my two intel systems with the same image is in moths.
Try a newer Ubuntu version such as 17.10 or use the HWE kernel for Ubuntu 16.04.3. That should help quite a bit.
Unfortunately, for some unknown reasons performance for my tasks are worse with 17 than 16. So I would like to run 16.04. Ofc, I cannot custom build a 4.x kernel kernel myself since the system hangs before done. I seem to experience several types of crashes. Sometimes there is nothing in kern.log other times I see the message that the system hanged for 22 seconds. I suspect some are related to the C-state problem while others are related to the CONFIG_RCU_NOCB_CPU setting.
I might just install windows 10 and give the machine to my kid 🙁
Try using the HWE kernel.
This is ridiculous. How long have these chips been on the market? My server crashes anywhere from a few hours to 3 days. Using 16.04 with whatever kernel came with it. I really don’t want to install Windows… I’m going to try a BIOS update and maybe 17.10 this weekend and see what happens. If it still crashes then it’s going to be Windows until 18.04 is officially released.
Josh – try 16.04.3 HWE kernel or 17.10.
18.04 didn’t fix this for me on 2700x.
Tried disabling c6 state, disabling all but c1 state, upping vcore voltage .1v, kernel flags, mainline kernel 4.18.20, nothing works.
18.04 doesn’t even install any more.
Horrendous.
Sadly this is STILL an issue as of July 2019 and really I would advise users to keep away from AMD for Linux until it is properly fixed (if ever?). See launchpad bug #1810546 and similar.
Running 18.04 Ubuntu with latest kernel, etc, and BIOS update for MSI motherboard and the GPU still crashes a couple of times per week. Seems the only solution is to go for a 3rd party graphics card. This is over two years of trouble, I mean WTF!
same issue with ryzen 2700 @ubuntu 19.04 with 5.0 linux kernel
I have no idea what to change in the bios, toying with C6 state did not work it still crashes while using, forcing a full reboot. Very infuriating.
Same issue with Ryzen 5800U on kubuntu 20.04 with all 5.8, 5.11, 5.13 kernels. The machine is lenovo ideapad 5 pro 14acn16. Crashes happen once every two weeks or suchs. Absolutely nothing in logs. I am going to try to upgrade to 21.10 and latest kernel
@Danila I’m not sure if the same issue applies to my MinisForum MC 560 with AMD® Ryzen 5 5625u or my previous Asus Mini PN50-1 with AMD Ryzen 7 4700-U CPU. Both run/ran Ubuntu 22.04 desktop. I have posted my issue as a bug at https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/2022826