In my living room there’s a server rack holding a Dell R710. This server is nice and tidy, has good airflow available, and has an acceptable amount of redundancy built into various aspects. It spends it’s life powered down. In my closet there’s a whitebox jammed in the corner. It literally can’t contain the disks it has so the SSD’s just limply hang out the side, and there’s a material bag resting on the top of the otherwise open case to help the dubious airflow. I can only pray that the RAID doesn’t fail completely. It runs 24/7.

If only the Dell R710 wasn’t as loud, then I could migrate everything and decomission the shambling Tangela hiding in my closet that I pretend to ignore.

Hushing Techniques #

Below are some ways that I seriously considered in order to bring the noise levels down.

Move the rack somewhere else. #

Considering my current premise, this wasn’t practical. This would be one of the easiest ways, just wheel it to another room. I did consider mounting it vertically in some small dead space, but this never became realistic..

Replace the stock fans with quieter ones. #

There’s a bunch of people that have tried and succeeded in replacing the stock fans. However, this involed making permanent modifications like soldering in the new fans due to the proprietary connector of the default fans. Some alternatives included taking out some of the stock fans, and soldering in resistors to the stock fans.

Apply noise deadening material around the rack. #

The rack holding the server is an open air rack, so there was potential for adding some kind of sound deadening material around it. Logistically this was a pain and would require a reasonable amount of effort to do.

Make the stock fans spin slower. #

Simply make the fans spin slower. This is what I ended up doing.

There’s also this neat post by Michael SanAngelo called R710 Be Quiet! where they hook up an Arduino to control the fan speed, which is really nifty.

Hushing Implementation #

Overview #

You’ll create a virtual machine which will communicate to the iDRAC (which is an implementation of IPMI), telling it to set the fans to a certain RPM. Then there’s some fail safes, alerting, and automation applied. For some extra detail, the Dell R710 is running ESXi, with a Ubuntu 18.04 LTS guest which communicates via RCMP+ to the iDRAC6, which will control the fan speed.

Someone already went and did all the hard work. There’s a reddit thread which points to the NoLooseEnds - R710-IPMI-TEMP Github repository. There’s some stuff we need to do first, but the actual hard part is already done.

iDRAC Considerations #

Considering we’ll be talking to the iDRAC over a LAN, and access to an iDRAC is a pretty big deal, there’s a few things we need to do and think about. We’ll first create an account just for this task. Then we’ll make sure the IMPI Over LAN (IOL) is actually enabled and set an encryption key. After that, we should consider segmenting the management interface off from the riff raff.

iDRAC User #

The iDRAC user will need to have the Administrator role for the “IPMI User Privileges” which appears under the column “LAN”. Keep in mind we’re dealing with an iDRAC 6, specifically version 2.91.

Go to iDRAC Settings -> Network/Security -> Users then click on a free “User ID” number. Select the “Configure User” radio button, and click “Next”.

The “Enable User” check box should be ticked.
Enter a username (eg “ipmi-admin”).
Enter a password (eg “D2j9C63nj8DXrUmHP87p” - lanplus won’t let you have one longer than 20 bytes, and special characters might ruin your day).
Set “Maximum LAN User Privilege Granted” to “Administrator”.
The “Enable Serial Over LAN” check box should be ticked.
Everything else should be default, which will effectively be no permissions.

Click “Apply” to create our new IPMI Administrator user.

IPMI Over LAN #

Now we can enable IPMI Over LAN by navigating to iDRAC Settings -> Network/Security -> IPMI Settings then ticking the “Enable IPMI Over LAN” check box. Ensure the “Channel Privilege Level Limit” is set to “Administrator”.

It’s also a good idea to configure the “Encryption Key”, so we can add a little bit more goodness. The Encryption Key is a hex value that needs to be supplied to the iDRAC along with a valid username and password in order for the communication to work. So… not really an encryption key and more like password number two but anyway, add in a 40 character hex value, something like “3E140EC4DAE3339BEC3BCA83EDBDB022FDAB7AFB”. As a quick aside, I couldn’t find too much information on the “Encryption Key” and how it works, so it just seems to be an extra piece of information provided to validate the communication, but I might be wrong.

Network Segmentation #

The fun part is that some implementations of IPMI are just broken because the (brace for impact) RMCP+ Authenticated Key-Exchange Protocol (RAKP) authentication process will just tell you the HMAC prior to actually authenticating you. Then you can go and crack the hash offline, and come back conveniently knowing all the required details.

So, segment the management interface somewhere most people can’t get to, and only allow specific trusted machines (and people) access to it. Or just keep IOL disabled.

Guest Virtual Machine Configuration #

Install Utilities #

Assuming you already have a Ubuntu 18.04 LTS virtual machine, we’ll want to install ipmitool so we can actually communicate with the iDRAC.

sudo apt update
sudo apt install ipmitool

You’ll get an error saying that the ipmievd.service failed with the error: Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory

This error is benign for our use case, the ipmievd daemon doesn’t need to run. I spent way too long trying to fix this, when we just need the utility installed. As it turns out, the daemon would listen for events from the iDRAC, however we’re sending requests to the iDRAC and then receiving the response to our request. So we don’t care about this error.

Test Connections #

Everything we need to configure is good to go, so let’s test our connection. From our virtual machine, we can attempt to connect to the iDRAC and request temperature information. In my infinite wisdom, I tried to connect from the virtual machine to the virtual machine. Naturally it didn’t work, I don’t recommend it.

Some quick explanation to our command below: -I is the interface, in our case it’s “lanplus”. -H is the host or address, so the iDRAC IP address 192.168.0.120. -U is our username, so “ipmi-admin”. -P is our password, so “D2j9C63nj8DXrUmHP87p”. -y is our hexadecimal encryption key, so “3E140EC4DAE3339BEC3BCA83EDBDB022FDAB7AFB”. sdr stands for Sensor Data Repository, so the sdr type temperature says to search the Sensor Data Repository for the type of temperature. If you forgo the type, then you’ll just see all records from the SDR.

ipmitool -I lanplus -H 192.168.0.120 -U ipmi-admin -P D2j9C63nj8DXrUmHP87p -y 3E140EC4DAE3339BEC3BCA83EDBDB022FDAB7AFB sdr type temperature

Hopefully, you’ll get a bunch of temperature related information. If not, you’ll probably get something saying “Error: Unable to establish IPMI v2 / RMCP+ session”. If you get the error, double check you’re communicating to the correct IP, and your credentials are correct. If you’re still getting an error, make sure the iDRAC account has the correct permissions, and that IOL is enabled.

Hushing Execution #

At this point, you’re basically set as we have the means to slow the fan speed. Now we’ll use the NoLooseEnds - R710-IPMI-TEMP scripts to do the hard yards for us. You have the Static script, which you run once and it will set the fan speed to manual control, then drop the fan speed to the configured RPM. Then there’s the Temp script which you set to run periodically, which will set the fan speed to automatic control once the server breaches a certain temperature threshold.

Clone the two .sh scripts onto your machine from the repository - you can also get a zip which you’ll need to extract from here: https://github.com/NoLooseEnds/Scripts/archive/master.zip

Then edit both the IPMITemp and IMPIStatic scripts and plug your IPMIHOST, IPMIUSER, IPMIPW, and IPMIEK details into them. If you’d like, you can also take the time to set up slacktee which is included in the IPMI scripts. Make sure the scripts are executable with chmod +x ./R710-IPMIStatic.sh && chmod +x ./R710-IPMITemp.sh. Nothing left to do now except execute the Static script ./R710-IPMIStatic.sh and listen to the hush happen.

The quiet is deafening.

You’ll want to set up your cron jobs as noted, so that if things start getting toasty then automatic control can kick back in.

*/5 * * * * /bin/bash /path/to/script/R710-IPMITemp.sh > /dev/null 2>&1

Hushing the Dell R710 Fans