supermicro / x9drw / quest for kvm

Written by
Walter Doekes
Published on 2022-09-09

I'm connected to an “ancient” Supermicro machine — according to today's standards — that saw the light somewhere around 2013. I'm looking for a way to access the KVM module (Keyboard, Video, Mouse) so I can update it safely. You know, to be able to fix boot issues if they arise. Unfortunately, the firmware is rather old and I cannot get the iKVM application to run, like I'm used to.

Using ipmikvm

Normally, for these Supermicro servers, I use the ipmikvm wrapper. It's a script that (a) logs into the web interface, (b) downloads launch.jnlp, (c) parses it and fetches the prerequisite jar files and lastly (d) fires up Java to start the iKVM application.

You type this:

$ ipmikvm -u $USER -p $PASS $IP

The script then fires up the appropriate Java application for that particular host, generally a particular generation of the iKVM application that comes with the firmware.

For this oldie from 2013 however, that did not work:

The web interface is different, and ipmikvm cannot find launch.jnlp;
downloading launch.jnlp manually is troublesome, because the web server writes invalid Content-Length headers;
running the provided JViewer did not appear to work: the user interface started, but showed “0 fps” in the title bar and no video was shown.

We can go into the browser network inspector and use Copy as cURL to download the launch properties file:

curl -vv 'http://IP/Java/jviewer.jnlp?EXTRNIP=IP&JNLPSTR=JViewer' \
  -H $'Cookie: test=1; ...' \
  -H 'Referer: http://IP/page/jviewer_launch.html?JNLPSTR=JViewer&JNLPNAME=/Java/jviewer.jnlp' \
  --compressed \
  --insecure \
  >launch.jnlp

...

< HTTP/1.0 200 OK
< Server: GoAhead-Webs
< Expires: 0
< Content-length: 4134
< Content-type: application/x-java-jnlp-file
< Set-Cookie: test=1;path=/
<
{ [1216 bytes data]
* transfer closed with 1070 bytes remaining to read
* Closing connection 0
curl: (18) transfer closed with 1070 bytes remaining to read

A valid launch.jnlp XML file of 3065 bytes is fetched, even though the server promised me 4134 bytes. An annoying bug, because newer browsers will refuse to store this file.

Does it run?

Yes! But only after fixing ipmikvm so it also downloads and unzips Linux_x86_64.jar for old JViewer:

$ ipmikvm ./launch.jnlp
...
+ exec java -Djava.library.path=/home/walter/.local/lib/ipmikvm/JViewer/release \
  -cp /home/walter/.local/lib/ipmikvm/JViewer/release/JViewer.jar \
  com.ami.kvm.jviewer.JViewer -apptype JViewer -hostname IP \
  -kvmtoken gWQwGxvrstzd -kvmsecure 0 -kvmport 7578 -vmsecure 0 \
  -cdstate 1 -fdstate 1 -hdstate 1 -cdport 5120 -fdport 5122 \
  -hdport 5123 -cdnum 1 -fdnum 1 -hdnum 1 -userpriv 255 -lang EN \
  -webcookie 76741dOicDdU4NSgnzyIIVTv

Access to libjava*.so was a needed for the application to work properly.

Upgrading firmware to SMM_X9_2_35

Before I found and fixed the ipmikvm problem, I tried to tackle this by upgrading the firmware. That seemed like the best way forward but boy, was I wrong.

To find the right firmware, one needs to know the board version:

# journalctl -b0 -ocat --grep DMI:
DMI: Supermicro X9DRW/X9DRW, BIOS 3.0a 08/08/2013

# ipmitool mc info
Device ID                 : 32
Device Revision           : 1
Firmware Revision         : 2.19
IPMI Version              : 2.0
Manufacturer ID           : 10876
Manufacturer Name         : Supermicro
Product ID                : 43707 (0xaabb)
Product Name              : Unknown (0xAABB)
Device Available          : yes
Provides Device SDRs      : no

(There is also dmidecode, but it tells us nothing more than we already know.)

So, it's a Supermicro X9DRW with BMC Firmware version 2.19 installed. Unfortunately, finding and installing appropriate firmware is easier said than done.

WARNING/CAVEATS: Before you go any further: if you're doing this on a remote machine, make sure you have /dev/ipmi0 access to it, so you can reconfigure the LAN address and/or authentication after resetting/flashing the firmware.

On the web interface of this BMC, there was an BMC Firmware Update button, but nowhere to upload any firmware. Pressing the update button made it go into “update mode” but that did nothing more than stall the BMC for at least 15 minutes.

Attempts to access the web interface during that time, showed:

Access Error: Target device firmware is
being upgraded and not currently available.
Please try again after update completes.

Card in flash mode !

Luckily, it snapped out of that after a while (15 minutes?). Another thing to keep in mind: every time the BMC does a cold reset, it takes about 85 seconds for it to come back.

We needed something better...

After several attempts with firmwares from Drunkencat Supermicro BIOS, I did get it to install version 2.35, like this:

$ unzip ../SMM_X9_2_35.zip

$ unzip RLinFlsh2.9.zip

$ cd Linux_x86_64/

$ LD_LIBRARY_PATH=. ./RLin64Flsh -nw -ip $IP -u $USER -p $PASS -i ../SMM_X9_2_35.ima
./RLin64Flsh: error while loading shared libraries: libipmi.so.1: cannot open shared object file: No such file or directory

(I should write something about using confinement (firejail, docker, apparmor) when running untrusted binaries here. But that is for another day.)

$ ln -s libipmi.so.1.0 libipmi.so.1

$ LD_LIBRARY_PATH=. ./RLin64Flsh -nw -ip $IP -u $USER -p $PASS -i ../SMM_X9_2_35.ima
-------------------------------------------------
YAFUFlash - Firmware Upgrade Utility (Version 2.9)
-------------------------------------------------
(C)Copyright 2008, American Megatrends Inc.

Creating IPMI session via network with address IP...Done
===============================================================================
                             Firmware Details
===============================================================================
    RomImage                   ExistingImage from Flash

    ModuleName   Description   Version     ModuleName   Description   Version
1.  boot         BootLoader    0.2         boot          BootLoader    0.1
2.  pcie                       0.1         pcie                        0.1
3.  conf         ConfigParams  0.1         conf          ConfigParams  0.1
4.  bkupconf                   1.2         bkupconf                    1.2
5.  root         Root          0.1         root          Root          0.1
6.  osimage      Linux OS      0.6         osimage       Linux OS      0.6
7.  www          Web Pages     0.6         www           Web Pages     0.6
8.  rainier                    2.35         rainier                     2.19

For Full Firmware upgrade,Please  type (0)  alone
For Module Upgrade enter the total no. of Modules to Upgrade
Enter your choice : 1
Enter the Module Name to Update : rainier

****************************************************************************
 WARNING!
        FIRMWARE UPGRADE MUST NOT BE INTERRUPTED ONCE IT IS STARTED.
        PLEASE DO NOT USE THIS FLASH TOOL FROM THE REDIRECTION CONSOLE.
****************************************************************************
Updating the module rainier ..... Done
Resetting the firmware..........

The above output was from one of the lucky tries. During other attempts I got Error in activate flash mode or some other error. It appeared that updating firmware using RLin64Flsh was not very deterministic with seemingly random failures.

Failing upgrades to SMT_X9_315

Trying to update to 3.15 or 3.53 did not pan out well either. The zip files contain an lUpdate tool that did not work remotely (-i lan). It did operate on /dev/ipmi0, albeit very slowly.

Using the local lUpdate started promising, with three upload phases. However, all attempts with ended in:

Please wait....If the FW update fails. PLEASE WAIT 5 MINS AND REMOVE THE AC...


Update progress:193 %

193%!? Hopeless...

Other attempts

I also tried the fwum option in ipmitool:

# ipmitool fwum info
FWUM extension Version 1.3

IPMC Info
=========
Manufacturer Id           : 10876
Board Id                  : 43707
Firmware Revision         : 2.35
FWUM Firmware Get Info returned c1

But "downloading" firmware onto the BMC only got me segmentation faults, i.e. bugs in ipmitool itself.

I've come to believe that not being able to upgrade to 3.x is probably a Renesas (X9, *.ima) versus Harmon (X9, *.bin) thing. Likely this machine has Renesas BMC hardware and will not update to any newer firmware.

After some more failed attempts at upgrading to firmware versions in the 3.x range, I did manage to update it to 2.60 through sheer luck. In the mean time, I also tried a factory reset. For that it is important that you have /dev/ipmi0 access, so you can fix (static) IPs:

ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr 1.2.3.4
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 1.2.3.1
ipmitool lan set 1 ipsrc static

If you forget to set it to static first, you get the following error, along with a timeout:

# ipmitool lan set 1 defgw ipaddr 1.2.3.1
Setting LAN Default Gateway IP to 1.2.3.1
LAN Parameter Data does not match!  Write may have failed.

But alas, no working JViewer after the factory reset either.

Upgrading firmware to SMM_X9_2_77

I managed upgrading to 2.77 — the latest 2.x I could find — using SMCIPMITool, which you can fetch from the Supermicro downloads if you agree to Supermicro's terms.

File Name:SMM_X9_2_77.zip
Revision:2.77
Size(KB):20,461
This zip file contains BIOS ROM, Flash utility, and Readme
instructions. You may download the free WINZIP utility to
extract the ontents of this file.
MD5:dd3c8aee63b60f3fc2485caed05f669a
SHA1:3906d2367ca22bdb54d69e8723bad34f0cd032e1
SHA256:eba03b51c5e8ebc561481cd136c305ef9495ff197b3ec56817dc3646190aef97

$ md5sum ../SMCIPMITool_2.26.0_build.220209_bundleJRE_Linux_x64.tar.gz
6a50f5246e0e472d1c80bd1fb6378bba  SMCIPMITool_2.26.0_build.220209_bundleJRE_Linux_x64.tar.gz

$ ./SMCIPMITool $IP $USER $PASS ipmi flashr ../../2.77/SMM_X9_2_77.ima
**************************************************************
WARNING!
Firmware upgrade must not be interrupted once it is started.
Once you get error after Upgrading, please use local KCS tool
for recovery.(DOS:RKCSFlsh.exe, Linux:RLin32Flsh or
Windows:RWin32Flsh.exe )
**************************************************************
Check firmware file... Done (ver:2.77.0)
Check BMC status... Done (ver:2.60.0)
Enter to Flash Mode

Uploading >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>100%
Upgrading >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>100%
Verifying >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>100%
Resetting BMC
Success
Total Elapse Time: 20 min 26 sec(s)

This felt a lot less flaky. Finally a tool that is less vague about what it's doing and whether it's doing anything at all.

Using SMCIPMITool

The mentioned SMCIPMITool appears to be better than many of the other tools out there.

$ ./SMCIPMITool $IP $USER $PASS ipmi ver
Firmware Revision  = 02.77.00
IPMI Version       = 2.0
Manufacturer ID    = 7C 2A 00
product ID         = BB AA 00

$ ./SMCIPMITool $IP $USER $PASS nm ver
Node Manager Version = 2.0
Firmware Version     = 2.17

$ ./SMCIPMITool $IP $USER $PASS nm20 summary
...
             Power Usage
+------------------------------------+
|Domain                  |  Usage (W)|
+====================================+
|Entire platform         |        153|
+------------------------------------+
|CPU subsystem           |         38|
+------------------------------------+
|Memory subsystem        |          6|
+------------------------------------+

Using SMCIPMITool for KVM

In fact, you can also use SMCIPMITool for KVM purposes:

$ ./SMCIPMITool $IP $USER $PASS ukvm
Starting JViewer (X9) Process...Done
Please wait for JViewer (X9) window

This opened the JViewer application. And this was the first time I got video instead of the dreaded “0 fps”.

Under the hood, SMCIPMITool looks up which firmware it speaks to, and then invokes a bundled JViewer.jar, like this:

$ java -jar JViewerX9.jar $IP $USER $PASS

(For newer firmwares it will invoke the bundled iKVM application instead.)

So, it might be a good idea to get that SMCIPMITool_2.26.0_build.220209_bundleJRE_Linux_x64.tar.gz and its contained JViewerX9.jar today and store it somewhere safe.

supermicro / x9drw / quest for kvm