The NVIDIA-Certified Professional AI Networking (NCP-AIN)

Modern AI training (especially with LLMs) requires extremely high-speed, parallel access to large datasets. A dedicated storage fabric separates data I/O traffic from the training compute path and avoids contention.

From NVIDIA DGX Infrastructure Reference Architectures :

"Dedicated storage networks eliminate I/O bottlenecks by providing low-latency, high-bandwidth access to distributed storage for large-scale training jobs."

"Parallel access to datasets is key for performance, especially in multi-node, multi-GPU AI clusters."

Security (B) is important, but not the core reason for a storage fabric.

Cost (D) is typically increased , not reduced, with dedicated fabrics.

NVIDIA NetQ is a comprehensive network operations tool designed to provide real-time visibility into the health and performance of NVIDIA networking environments, including Spectrum-X. It offers detailed telemetry and analytics, allowing administrators to monitor adaptive routing behaviors, detect congestion, and analyze traffic patterns. By leveraging NetQ, you can ensure that adaptive routing is functioning as intended and that the network is optimized for AI workloads.​

Reference Extracts from NVIDIA Documentation:

"The NVIDIA NetQ network validation and ASIC monitoring tool set provide visibility into the network health and behavior. The NetQ flow telemetry analysis shows the paths that data flows take as they traverse the network, providing network latency and performance insights."​

"By leveraging telemetry from Spectrum Ethernet switches and BlueField-3 SuperNICs, NVIDIA NetQ can detect network issues proactively and troubleshoot network issues faster for optimal use of network capacity."​

In NVIDIA Spectrum-X, congestion is evaluated based on egress queue loads. Spectrum-4 switches assess the load on each egress queue and select the port with the minimal load for packet transmission. This approach ensures that all ports are well-balanced, optimizing network performance and minimizing congestion.

According to the NVIDIA Spectrum-X Switch Operating System (SX_OS) Troubleshooting Guide , the System Status LED behavior is a critical indicator of the switch’s internal operational state.

From the document:

“The System Status LED will blink green during system initialization. If the LED continues blinking for more than 5 minutes, it indicates that the Onyx OS has failed to load properly. The system may be stuck in the boot process, or the file system may be corrupted.”

This blinking LED beyond normal initialization time indicates that the system has either encountered a failure during software boot or is unable to transition from bootloader to the OS runtime environment (i.e., Onyx).

Key causes include:

Corrupted or missing system files.

Failed firmware or OS upgrade attempts.

Boot device (e.g., eMMC or SSD) issues or corrupted partitions.

Technically, during power-on:

The switch performs POST (Power-On Self Test).

Then the Onyx OS attempts to load from the boot partition.

If the Onyx OS kernel or root filesystem is invalid, the system halts boot, and the LED remains in a blinking state , as no successful OS load confirmation is triggered.

Remediation Steps (as per NVIDIA guide):

Access the switch through console and monitor boot logs .

Use ONIE recovery or re-flash a stable Onyx OS version.

Check system storage integrity using built-in diagnostics.

Exact Extract Reference:

Source: NVIDIA SX_OS 3.9.3000 Documentation

Topic: Troubleshooting System Status LED

Extract: "If the LED blinks for more than 5 minutes and the switch is not accessible via CLI, the Onyx software failed to load properly and recovery procedures must be initiated."

=============

The ibdiagnet utility is a fundamental tool for InfiniBand fabric discovery, error detection, and diagnostics. It provides comprehensive reports on the fabric's health, including error reporting, switch and Host Channel Adapter (HCA) configuration dumps, various counters reported by the switches and HCAs, and parameters of devices such as switch fans, power supply units, cables, and PCI lanes. Additionally, ibdiagnet performs validation for Unicast Routing, Adaptive Routing, and Multicast Routing to ensure correctness and a credit-loop-free routing environment.​

Reference Extracts from NVIDIA Documentation:

"The ibdiagnet utility is one of the basic tools for InfiniBand fabric discovery, error detection and diagnostic. The output files of the ibdiagnet include error reporting, switch and HCA configuration dumps, various counters reported by the switches and the HCAs."​

"ibdiagnet also performs Unicast Routing, Adaptive Routing and Multicast Routing validation for correctness and credit-loop free routing."​

Direct Memory Access (DMA) in InfiniBand networks allows data to be transferred directly between the memory of two devices without involving the CPU. This capability significantly reduces CPU overhead, lowers latency, and increases throughput, making it ideal for AI workloads that demand efficient data transfers.

To perform Flow Analysis with NetQ, the following prerequisites must be met:​

Cumulus Linux Version : NetQ Flow Analysis requires Cumulus Linux 5.x or later.​

Switch Hardware : The feature is supported on Spectrum-2 and later switch models.​

Lifecycle Management (LCM) : LCM must be enabled to utilize Flow Analysis capabilities.​

These requirements ensure compatibility and proper functioning of the Flow Analysis feature within NetQ.​

Before upgrading the DOCA SDK on a BlueField DPU , it is mandatory to uninstall the existing OFED drivers to prevent compatibility conflicts.

From the NVIDIA DOCA Installation Guide :

"Before upgrading DOCA or BlueField-related software, you must remove existing OFED packages using: /usr/sbin/ofed_uninstall.sh -force."

This ensures:

Clean driver state

No residual kernel modules or userspace libraries

Proper registration of new DOCA/OFED versions

Incorrect Options:

A and C may not resolve conflicts.

D installs but doesn’t remove conflicting packages.

In Cumulus Linux, configuring a bond interface with Link Aggregation Control Protocol (LACP) involves setting the bond mode to 'lacp'. The correct command to achieve this is:​

nv set interface bond1 bond mode lacp​

This command sets the bonding mode of 'bond1' to LACP, enabling dynamic link aggregation for increased bandwidth and redundancy.​

Reference Extracts from NVIDIA Documentation:

"To reset the link aggregation mode for bond1 to the default value of 802.3ad, run the nv set interface bond1 bond mode lacp command."​