Friday, August 28, 2020

How to install Mellanox OFED IB Drivers in CentOS Linux 7 (unattended)

Title: Unattended Mellanox Infiniband OFED Installation - Tutorial

OS: CentOS Linux release 7.8.2003 (Core)

Kernel Version: 3.10.0-1127.13.1.el7.x86_64

Compiler: gcc v4.8.5

Software: Mellanox OFED

Software Version: 5.0-

User Account: Use ‘sudo’ (recommended)

Reboot Required: Yes (recommended)

·        Create a directory ‘/tools/apps/sources/mellanox/ofed/5.0-’ (Please create directory according to your requirements)

o   mkdir -p /tools/apps/sources/mellanox/ofed/5.0-

o   export MLNX_ROOT_DIR=/tools/apps/sources/mellanox/ofed/5.0- 

·        Download the latest Mellanox OFED .tgz file for your OS distribution and architecture from Mellanox website ( and place it into the directory created in the previous step. In my case the downloaded file is ‘MLNX_OFED_LINUX-5.0-’.

·        Navigate into the directory and extract the tarball.

o   cd ${MLNX_ROOT_DIR}

o   tar xvfz MLNX_OFED_LINUX-5.0- 

·        Navigate into ‘MLNX_OFED_LINUX-5.0-’ directory



(If your kernel version is not listed in ‘.supported_kernels’ file then you will have to add support for your kernel. To accomplish this, please follow the steps under ‘Troubleshooting’ section below)

·        Mellanox package comes with some sample configuration files which can be used to install mellanox OFED unattended. Configuration samples can be found in ‘${MLNX_ROOT_DIR}/${MLNX_SOURCE}/docs/conf’ directory i.e ofed-basic.conf, ofed-hpc.conf and ofed-all.conf. 

·        Install the necessary packages required for Mellanox OFED installations

o   yum install tcl tk 

·        To install Mellanox OFED stack run the following script with the necessary flags for unattended installation. You can use configuration file according to your requirements.

o   ./mlnxofedinstall –c docs/conf/ofed-hpc.conf --force

·        After the above command, the installation will be completed. Restart the system so that it may load the device drivers on startup.

How to start infiniband related services?

Once the installation has been performed, it installs two main services:

·        Openibd: Infiniband related drivers

·        Opensmd: Subnet manager

An instance of a ‘Subnet Manager’ is required on an infiniband fabric (network) for machines to communicate. This service can either be started on Infiniband switches (subject to having the ‘Subnet Manager’ support on the switches) or on servers connected to the same infiniband fabric.

openibd’ service needs to be running on all servers whereas ‘opensmd’ service can only be started on a single server/switch or on multiple core servers/switches.

·        Start and enable ‘openibd’ service

o   systemctl start openibd

o   systemctl enable openibd 

·        Start and enable ‘opensmd’ service

o   systemctl start opensmd

o   systemctl enable opensmd


Adding Additional Kernel Support

Sometimes a downloaded .tgz file doesn’t come with the RPMs which support the latest kernel. To add support for a different kernel follow these steps:

o   Navigate into the Mellanox OFED source directory and run these commands

      •   cd MLNX_OFED_LINUX-5.0-
      •   yum install python-devel
      •   ./ -m . -make-tgz

The above command will re-compile Mellanox OFED stack and will create a ‘tgz’ file in ‘/tmp’ directory. This will be the tarball which can be used to install the OFED stack on machines with a specific kernel.


No comments:

Post a Comment

How to install and configure xCAT (for beginners)

Are you fed-up of manual server installation? Are you not a fan of managing your cluster using DVDs and USB sticks anymore? Are you afra...