Skip to Main Content
Set the user name from the user email address Animate email input

Dedicated SFP+ (10GbE+) network ring in ProxMox

ProxMox System Administration
Setting up a dedicated SFP+ (10GbE+) network ring in Proxmox VE for storage is a great way to get low-latency, high-throughput, and switch-independent connectivity—especially for clustered storage like Ceph or ZFS replication. Below is a clear, practical, step-by-step guide with design tips and exact config examples.

Topology (Ring Design)

Each node connects directly to two neighbors:

Node1 ↔ Node2 ↔ Node3 ↔ Node1

    •    No switch required
    •    Each node has 2× SFP+ ports
    •    Uses DAC cables or fiber
    •    Traffic can route around failures

Hardware Requirements

Minimum per node:
    •    2× SFP+ NIC ports (Intel X520/X710, Mellanox ConnectX-3/4/5 recommended)
    •    DAC cables or fiber modules
    •    Consistent NIC drivers (important for stability)

Physical Wiring Example (3 Nodes)

Node1:
 port0 → Node2
 port1 → Node3

Node2:
 port0 → Node3
 port1 → Node1

Node3:
 port0 → Node1
 port1 → Node2

Network Design Strategy

Simple Point-to-Point Links (Recommended)
    •    Each link = its own subnet
    •    Best for control + redundancy
    •    Most stable for Ceph

Configure Interfaces in Proxmox

vi /etc/network/interfaces

Node 1

auto enp1s0f0
iface enp1s0f0 inet static
   address 10.0.12.1/24
auto enp1s0f1
iface enp1s0f1 inet static
   address 10.0.31.1/24

Node 2

auto enp1s0f0
iface enp1s0f0 inet static
   address 10.0.23.2/24
auto enp1s0f1
iface enp1s0f1 inet static
   address 10.0.12.2/24

Node 3

auto enp1s0f0
iface enp1s0f0 inet static
   address 10.0.31.3/24
auto enp1s0f1
iface enp1s0f1 inet static
   address 10.0.23.3/24

Enable Routing Between Links

Enable IP forwarding:

echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl -p

Add static routes
Example (Node1):

ip route add 10.0.23.0/24 via 10.0.12.2

Repeat accordingly on all nodes.

Use It for Storage Traffic Ceph 

In /etc/pve/ceph.conf:

public_network = 10.0.0.0/16
cluster_network = 10.0.0.0/16

Or better:

public = management LAN
cluster = ring network (10Gb)

ZFS Replication

Use the ring IPs directly:

zfs send | ssh 10.0.12.2 zfs receive

Optimize Performance

Enable jumbo frames

mtu 9000

Tune NIC


ethtool -G enp1s0f0 rx 4096 tx 4096

CPU tuning
    •    Enable IRQ balancing
    •    Pin NIC interrupts if needed

Add Redundancy (Important)

A ring alone ≠ automatic failover.

Add:
    •    Static fallback routes
    •    Or dynamic routing (advanced):

Use:
    •    FRRouting (FRR)
    •    OSPF or BGP for automatic rerouting

This turns your ring into a self-healing mesh

Test the Network

Latency:

ping 10.0.23.2

Throughput:

iperf3 -s
iperf3 -c <peer IP>
www.1apx.com
_