Sign In
Upload
Manuals
Brands
Qlogic Manuals
Software
OFED+ Host 1.5.4
User Manuals: Qlogic OFED+ Host 1.5.4 Software Release
Manuals and User Guides for Qlogic OFED+ Host 1.5.4 Software Release. We have
1
Qlogic OFED+ Host 1.5.4 Software Release manual available for free PDF download: Software User's Manual
Qlogic OFED+ Host 1.5.4 Software User's Manual (284 pages)
Brand:
Qlogic
| Category:
Software
| Size: 2.24 MB
Table of Contents
Document Revision History
2
Table of Contents
3
Preface
15
Related Materials
15
License Agreements
16
Technical Support
17
Training
17
Contact Information
17
Knowledge Database
18
How this Guide Is Organized
19
Intended Audience
15
Documentation Conventions
15
Introduction
19
Overview
20
Interoperability
21
Step-By-Step Cluster Setup and MPI Usage Checklists
23
Intel
24
Cluster Setup
25
Infiniband
25
Introduction
25
Qlogic OFED+ Software Structure
25
Installed Layout
26
IB and Openfabrics Driver Overview
27
Ipoib Network Interface Configuration
27
Ipoib Administration
29
Administering Ipoib
29
Stopping, Starting and Restarting the Ipoib Driver
29
Configuring Ipoib
29
Editing the Ipoib Configuration File
29
IB Bonding
30
Interface Configuration Scripts
30
Red hat EL5 and EL6
31
Suse Linux Enterprise Server (SLES) 10 and 11
32
Verify IB Bonding Is Configured
33
Subnet Manager Configuration
34
Qlogic Distributed Subnet Administration
36
Applications that Use Distributed SA
36
Virtual Fabrics and the Distributed SA
37
Configuring the Distributed SA
37
Default Configuration
37
Distributed SA Default Configuration
37
Multiple Virtual Fabrics Example
38
Distributed SA Multiple Virtual Fabrics Example
38
Virtual Fabrics with Overlapping Definitions
39
Distributed SA Multiple Virtual Fabrics Configured Example
39
Virtual Fabrics with All Sids Assigned to PSM_MPI Virtual Fabric
40
Distributed SA Configuration File
41
Virtual Fabrics with PSM_MPI Virtual Fabric Enabled
40
Virtual Fabrics with Unique Numeric Indexes
41
Sid
42
Scanfrequency
42
Logfile
42
Dbg
43
Other Settings
43
Changing the MTU Size
44
Ibmtu Values
44
Managing the Ib_Qib Driver
45
Configure the Ib_Qib Driver State
46
Start, Stop, or Restart Ib_Qib Driver
46
Unload the Driver/Modules Manually
47
Ib_Qib Driver Filesystem
47
More Information on Configuring and Loading Drivers
48
Performance Settings and Management Tips
48
Performance Tuning
49
Systems in General (with Either Intel or AMD Cpus)
49
Krcvqs Parameter Settings
51
AMD CPU Systems
52
AMD Interlagos CPU Systems
52
Intel CPU Systems
52
High Risk Tuning for Intel Harpertown Cpus
54
Additional Driver Module Parameter Tunings Available
55
Performance Tuning Using Ipath_Perf_Tuning Tool
58
Checks Preformed by Ipath_Perf_Tuning Tool
58
Options
59
Ipath_Perf_Tuning Tool Options
59
AUTOMATIC Vs. INTERACTIVE MODE
60
Test Execution Modes
60
Affected Files
61
Homogeneous Nodes
61
Adapter and Other Settings
62
Remove Unneeded Services
63
Host Environment Setup for MPI
64
Configuring for Ssh
64
Configuring Ssh and Sshd Using Shosts.equiv
64
Configuring for Ssh Using Ssh-Agent
65
Checking Cluster and Software Status
68
Ipath_Control
68
Iba_Opp_Query
69
Ibstatus
70
Ibv_Devinfo
71
Ipath_Checkout
71
Introduction
73
Mpis Packaged with Qlogic OFED
73
Open MPI
73
Installation
74
Setup
74
Compiling Open MPI Applications
74
Running MPI on Qlogic Adapters
73
Open MPI Wrapper Scripts
74
Create the Mpihosts File
75
Running Open MPI Applications
75
Command Line Options for Scripts
75
Further Information on Open MPI
76
Configuring MPI Programs for Open MPI
77
To Use Another Compiler
77
Portland Group
78
Compiler and Linker Variables
79
Process Allocation
79
IB Hardware Contexts on the QDR IB Adapters
80
Available Hardware and Software Contexts
80
Enabling and Disabling Software Context Sharing
81
Restricting IB Hardware Contexts in a Batch Environment
82
Context Sharing Error Messages
83
Running in Shared Memory Mode
83
Mpihosts File Details
84
Using Open Mpi's Mpirun
85
Console I/O in Open MPI Programs
86
Environment for Node Programs
87
Remote Execution
87
Exported Environment Variables
88
Setting MCA Parameters
89
Environment Variables
90
Environment Variables Relevant for any PSM
90
Job Blocking in Case of Temporary IB Link Failures
92
Environment Variables Relevant for Open MPI
92
Open MPI and Hybrid Mpi/Openmp Applications
93
Debugging MPI Programs
94
MPI Errors
94
Using Debuggers
94
Introduction
97
Other Supported MPI Implementations
97
Using Other Mpis
97
Installed Layout
98
Mvapich
99
Compiling MVAPICH Applications
99
Running MVAPICH Applications
99
MVAPICH Wrapper Scripts
99
Further Information on MVAPICH
100
Open MPI
99
MVAPICH Wrapper Scripts
100
Running MVAPICH2 Applications
101
Further Information on MVAPICH2
101
Mvapich2
100
Compiling MVAPICH2 Applications
100
Managing MVAPICH, and MVAPICH2 with the Mpi-Selector Utility
101
With the Mpi-Selector Utility
101
Platform MPI 8
102
Installation
102
Setup
102
Compiling Platform MPI 8 Applications
103
Running Platform MPI 8 Applications
103
More Information on Platform MPI 8
103
Intel MPI
103
Platform MPI 8 Wrapper Scripts
103
Installation
104
Setup
104
Compiling Intel MPI Applications
106
Running Intel MPI Applications
106
Intel MPI Wrapper Scripts
106
Further Information on Intel MPI
107
Improving Performance of Other Mpis over IB Verbs
108
Installation
109
Interoperability
109
Overview
109
SHMEM Description and Configuration
109
SHMEM Programs
111
Basic SHMEM Program
111
Compiling SHMEM Programs
112
Running SHMEM Programs
113
Using Shmemrun
113
Running Programs Without Using Shmemrun
114
Qlogic SHMEM Relationship with MPI
115
Slurm Integration
116
Full Integration
116
Two-Step Integration
116
No Integration
117
Sizing Global Shared Memory
117
Progress Model
119
Active Progress
120
Passive Progress
120
Active Versus Passive Progress
121
Environment Variables
121
SHMEM Run Time Library Environment Variables
121
Implementation Behavior
123
Shmemrun Environment Variables
123
Application Programming Interface
125
SHMEM Application Programming Interface Calls
126
Qlogic SHMEM Micro-Benchmarks Options
135
SHMEM Benchmark Programs
135
Qlogic SHMEM Random Access Benchmark Options
136
Qlogic SHMEM All-To-All Benchmark Options
137
Qlogic SHMEM Barrier Benchmark Options
138
Qlogic SHMEM Reduce Benchmark Options
139
Introduction
141
Virtual Fabric Support in PSM
141
Using SL and Pkeys
142
Virtual Fabric Support
142
SL2VL Mapping from the Fabric Manager
143
Using Service ID
143
Verifying SL2VL Tables on Qlogic 7300 Series Adapters
144
Dispersive Routing
145
Gpxe Setup
149
Required Steps
150
Preparing the DHCP Server in Linux
150
Installing DHCP
151
Configuring DHCP
152
Netbooting over IB
153
Prerequisites
153
Boot Server Setup
153
Steps on the Gpxe Client
162
HTTP Boot Setup
162
Benchmark 1: Measuring MPI Latency between Two Nodes
165
Benchmark Programs
165
Benchmark 2: Measuring MPI Bandwidth between Two Nodes
168
Benchmark 3: Messaging Rate Microbenchmarks
170
OSU Multiple Bandwidth / Message Rate Test (Osu_Mbw_Mr)
170
Mpi_Multibw
173
SRP Configuration
177
SRP Configuration Overview
177
Important Concepts
177
Qlogic SRP Configuration
178
Stopping, Starting and Restarting the SRP Driver
179
Specifying a Session
179
Determining the Values to Use for the Configuration
182
Specifying an SRP Initiator Port of a Session by Card and Port Indexes
184
Specifying an SRP Initiator Port of Session by Port GUID
184
Specifying a SRP Target Port
185
Specifying a SRP Target Port of a Session by IOCGUID
186
Specifying a SRP Target Port of a Session by Profile String
186
Specifying an Adapter
186
Restarting the SRP Module
187
Configuring an Adapter with Multiple Sessions
187
Configuring Fibre Channel Failover
189
Failover Configuration File 1: Failing over from One SRP Initiator Port to Another
190
Failover Configuration File 2: Failing over from a Port on the VIO Hardware Card to Another Port on the VIO Hardware Card
191
Failover Configuration File 3: Failing over from a Port on a VIO Hardware Card to a Port on a Different VIO Hardware Card Within the same Virtual I/O Chassis
192
Failover Configuration File 4: Failing over from a Port on a VIO Hardware Card to a Port on a Different VIO Hardware Card in a Different Virtual I/O Chassis
193
Configuring Fibre Channel Load Balancing
194
Configuring SRP for Native IB Storage
197
Notes
199
Additional Details
200
Troubleshooting
200
1 Adapter Port and 2 Ports on a Single VIO
194
2 Adapter Ports and 2 Ports on a Single VIO Module
195
Using the Roundrobinmode Parameter
196
OFED SRP Configuration
200
Clean Termination of MPI Processes
203
Integration with a Batch Queuing System
203
Clean-Up PSM Shared Memory Files
204
D-1 LED Link and Data Indicators
207
LED Link and Data Indicators
207
Troubleshooting
207
Using Leds to Check the State of the Adapter
207
BIOS Settings
208
Kernel and Initialization Issues
208
Driver Load Fails Due to Unsupported Kernel
209
Rebuild or Reinstall Drivers if Different Kernel Installed
209
Infinipath Interrupts Not Working
209
Openfabrics Load Errors if Ib_Qib Driver Load Fails
210
Infinipath Ib_Qib Initialization Failure
211
MPI Job Failures Due to Initialization Problems
212
Openfabrics and Infinipath Issues
212
Stop Infinipath Services before Stopping/Restarting Infinipath
212
Manual Shutdown or Restart May Hang if NFS in Use
213
Load and Configure Ipoib before Loading SDP
213
Set $IBPATH for Openfabrics Scripts
213
SDP Module Not Loading
213
Ibsrpdm Command Hangs When Two Host Channel
214
Adapters Are Installed but Only Unit 1 Is Connected
214
To the Switch
214
Outdated Ipath_Ether Configuration Setup Generates Error
214
System Administration Troubleshooting
214
Broken Intermediate Link
215
Performance Issues
215
Large Message Receive Side Bandwidth Varies with Socket Affinity on Opteron Systems
215
Erratic Performance
216
Method 1
216
Method 2
216
Immediately Change the Processor Affinity of an IRQ
217
Performance Warning if Ib_Qib Shares Interrupts with Eth0
218
Open MPI Troubleshooting
218
Invalid Configuration Warning
218
Troubleshooting Virtualnic and VIO Hardware Issues
219
Checking the Logical Connection between the IB Host and the VIO Hardware
219
Verify that the Proper Virtualnic Driver Is Running
220
Verifying that the Qlgc_Vnic.cfg File Contains the Correct Information
220
Verifying that the Host Can Communicate with the I/O Controllers (Iocs) of the VIO Hardware
221
Checking the Interface Definitions on the Host
224
Interface Does Not Show up in Output of 'Ifconfig
224
Verify the Physical Connection between the VIO Hardware and the Ethernet Network
225
ULP Troubleshooting
219
Troubleshooting SRP Issues
227
Ib_Qlgc_Srp_Stats Showing Session in Disconnected State
227
Session in 'Connection Rejected' State
229
Attempts to Read or Write to Disk Are Unsuccessful
232
Four Sessions in a Round-Robin Configuration Are Active
233
Which Port Does a Port GUID Refer to
234
How Does the User Find a HCA Port GUID
235
Need to Determine the SRP Driver Version
237
Introduction
239
PAT and Write Combining
239
Write Combining
239
MTRR Mapping and Write Combining
240
Edit BIOS Settings to Fix MTRR Issues
240
Use the Ipath_Mtrr Script to Fix MTRR Issues
240
Verify Write Combining Is Working
241
Check Cluster Homogeneity with Ipath_Checkout
243
Commands and Files
243
Restarting Infinipath
244
Useful Programs
244
Summary and Descriptions of Commands
246
Dmesg
246
Iba_Opp_Query
246
Iba_Hca_Rev
251
Iba_Manage_Switch
261
Iba_Packet_Capture
263
Ibhosts
264
Ibstatus
264
Ibtracert
265
Ident
266
Ibv_Devinfo
266
Ipath_Checkout
267
Ipath_Checkout Options
267
G-2 Ipath_Checkout Options
268
Ipath_Control
269
Ipath_Mtrr
270
Ipath_Pkt_Test
271
Ipathstats
272
Lsmod
272
Modprobe
272
Mpirun
273
Mpi_Stress
273
Rpm
274
Strings
274
Common Tasks and Commands
274
Common Tasks and Commands Summary
275
Summary and Descriptions of Useful Files
276
Boardversion
276
Status_Str
276
Useful Files
276
Status_Str File Contents
277
Version
278
G-6 Status—Other Files
278
Status-Other Files
278
Summary of Configuration Files
278
Configuration Files
279
Books for Learning MPI Programming
281
Infiniband
281
Recommended Reading
281
Reference and Source for SLURM
281
References for MPI
281
Clusters
282
Networking
282
Openfabrics
282
Other Software Packages
282
Rocks
282
Advertisement
Advertisement
Related Products
Qlogic OFED+ Host
Qlogic Data Migration Service
Qlogic Fast Fabric
Qlogic Fibre Channel NPIV Storage Networking Guide
Qlogic QLogic Fibre Channel Switch
Qlogic SANbox-16
Qlogic SANbox-8
Qlogic SANsurfer FC HBA Manager
Qlogic SANsurfer FC HBA Manager Application
Qlogic SANsurfer iSCSI HBA Manager
Qlogic Categories
Switch
Adapter
Host Adapter
Network Router
Software
More Qlogic Manuals
Login
Sign In
OR
Sign in with Facebook
Sign in with Google
Upload manual
Upload from disk
Upload from URL