Multi-Path Bonding aggregates several IP uplinks into one
reliable flow. This page is the network plumbing that sits underneath it:
how to connect several outdoor 5G modems — using the Teltonika OTD500 as a
running example, though any IP modem/router works the same way — behind a
single PoE switch, and present each one to the edge host as its own
independently-routable interface so the bonding scheduler can pin a path to
each modem.
There are two ways to do that, and you can mix them:
Design A — one physical NIC per modem. Dead simple, no VLANs on the host.
Design B — one NIC, VLAN-tagged. One cable carries every modem as a
tagged sub-interface. Scales to many modems on a single fast NIC.
The bond scheduler sends each path out a specific uplink. On the host that
means each modem must appear as a distinct, independently-routable network
interface — a real NIC, or a VLAN sub-interface. The bonded output then pins
each path to one of them (the interface field on a UDP path, which uses
SO_BINDTODEVICE on Linux).
If every modem shares one interface and subnet, the kernel routing table
collapses them onto a single default route and the “bond” is cosmetic — all
paths leave through whichever modem wins the route lookup.
Giving each path its own interface — a physical NIC or a VLAN sub-interface —
is the easiest and most robust way to do this, and it’s what this guide
recommends: the device binding is unambiguous and each interface needs only a
single route. A shared-interface alternative (one NIC, multiple IPs, policy
routing) exists for when you can’t add interfaces — see
Alternative: one interface, multiple IPs.
(Interface pinning no longer requiresCAP_NET_RAW either: without it the
edge falls back to the unprivileged IP_UNICAST_IF egress hint.)
A PoE switch that can power the modems. The OTD500 takes 802.3af/at
(PoE / PoE+) on its PoE-in port (42.5–57 V; ≤ ~30 W). PoE++ (802.3bt)
switches are backward-compatible and power it fine. For Design B the
switch must be managed / VLAN-capable; Design A only needs per-port VLAN
separation if all modems share one switch.
N outdoor 5G modems. The OTD500 is IP55, dual-SIM + eSIM, gigabit
Ethernet, and ships without a PSU — it’s powered over the PoE-in cable.
Outdoor/shielded Cat6 for each modem run, and an Ethernet surge
arrestor on each run where it enters the building (the modems sit outside;
that copper is a surge path into your switch and host).
An edge host with either N spare NICs (Design A) or one spare NIC
(Design B).
Many modems, one fast NIC (e.g. 25G), want to scale
The switch config is nearly identical in both designs — one VLAN per
modem, each modem on an access port. The only difference is how the edge
host consumes those VLANs: as separate untagged ports into separate NICs
(A), or as one tagged trunk into one NIC (B).
A clean, memorable scheme: VLAN N ↔ subnet 10.0.N.0/24, with the modem
at .1 (the gateway for that path) and the host at .2. Each modem must
be on its own subnet — the host needs a distinct gateway per path.
Each modem is a VLAN-unaware NAT router. You only change two things, and
they’re identical except the subnet:
Set a unique LAN IP per the table above.
Disable its DHCP server (the host uses a static address). On an isolated
per-VLAN segment you can leave DHCP on, but static is simplest for a
server host.
Leave the cellular/WAN side as your carrier needs it (SIM + APN), and do not
configure VLANs on the modem — the switch does the tagging.
OTD500 specifics (running example):
Insert an activated SIM, power it from the switch’s PoE port (single cable),
and browse to its default https://192.168.1.1.
Log in with admin / admin — you’re forced to set a new password, then the
Setup Wizard runs (time, then Mobile/APN; APN is usually auto-detected).
Change the LAN IP under Network → Interfaces → LAN → General Setup; toggle
DHCP on the DHCP Server tab. Saving the new LAN IP drops the WebUI session
— reconnect at the new address.
Gotcha: two factory-fresh modems are both 192.168.1.1. If they land on
the same segment before you re-IP them, they collide. Either set the switch
VLANs first (so each is isolated from the start), or configure the modems one
at a time on the bench, then deploy.
Step 2 — Configure the switch (common to both designs)
One host NIC carries every modem as a tagged VLAN. This is the
802.1Q trunk with an optional native VLAN pattern: a single physical port
can carry untagged frames (the NIC’s existing use, if any) and tagged
frames for each modem VLAN at the same time.
Switch: make the edge-facing port a trunk — a tagged member of every
modem VLAN. Optionally keep an untagged native VLAN if the NIC still serves
its existing role.
interface <edge-trunk-port>
switchport trunk allowed vlan 11,12 # tagged members
The interface-per-path design needs just one route per modem interface — a
default route via that modem, in the main table, at a high metric:
routes:
- { to: default, via: 10.0.11.1, metric: 1011 }
When the bond pins a path to an interface (the interface field →
SO_BINDTODEVICE), the kernel scopes the route lookup to that interface and
finds its default route → out the right modem. The high metric keeps these
off the host’s real default route, so un-pinned traffic still uses your
management/WAN link. That’s the whole routing story — no policy tables, no
ip rule juggling (those are only needed for the shared-interface
alternative below).
Then set loose reverse-path filtering on the modem interfaces so
asymmetric/multi-homed return traffic isn’t dropped (netplan can’t set this;
use a sysctl drop-in):
/etc/sysctl.d/90-bonding-5g.conf
net.ipv4.conf.wan5g11.rp_filter = 2
net.ipv4.conf.wan5g12.rp_filter = 2
Terminal window
sudosysctl--system# re-run after netplan apply (the sub-ifs must exist)
Per-interface rp_filter wins via max(conf.all, conf.<iface>), so this
relaxes only the modem interfaces — the rest of the host stays strict.
If you can’t give each path its own interface, put all modems on one L2
segment and the host on one interface with multiple IPs, then bind each bond
path to a different source IP (the path’s bind field) instead of an
interface. (You no longer need this just because you can’t grant CAP_NET_RAW —
interface pinning falls back to the unprivileged IP_UNICAST_IF hint — but
source-IP binding remains the answer when adding interfaces isn’t possible.)
Because every IP lives on the same device, SO_BINDTODEVICE can’t tell the
paths apart, so this is the one case that genuinely needs policy routing —
one table per modem plus a source rule:
# one shared interface carrying every modem subnet:
addresses: [10.0.11.2/24, 10.0.12.2/24]
routes:
- { to: default, via: 10.0.11.1, table: 11 }
- { to: default, via: 10.0.12.1, table: 12 }
routing-policy:
- { from: 10.0.11.2/32, table: 11 }
- { from: 10.0.12.2/32, table: 12 }
and the bond paths bind source IPs rather than interfaces:
Trade-offs: it’s unprivileged (no CAP_NET_RAW), but every modem shares one
broadcast domain — no L2 isolation, and you watch for DHCP/ARP cross-talk and
maintain the routing tables by hand. Prefer separate interfaces unless one of
those constraints forces this. rp_filter = 2 still applies.
Prerequisites: nothing to install. The ip/iproute2 tooling and the
kernel’s policy-routing support (CONFIG_IP_MULTIPLE_TABLES) ship enabled on
every mainstream distro; netplan writes the rules and tables for you (via the
systemd-networkd backend), and numeric table IDs need no /etc/iproute2/rt_tables
entry. Verify support with ip rule list — it should print the default rules,
not an error.
A Wi-Fi uplink — e.g. a Starlink Mini — works as a bond leg exactly like a
wired modem: pin the path to its interface ("interface": "wlo5", which uses
SO_BINDTODEVICE). Two things differ from a static wired modem:
It gets a DHCP address, so pin the route by interface, not source IP. A
Wi-Fi leg’s address changes on every re-associate or lease renewal, so a
from <ip> policy rule goes stale. Use an output-interface rule, which is
address-independent and survives lease changes:
Terminal window
iproutereplacedefaultvia192.168.4.1devwlo5table80
ipruleaddoifwlo5lookup80priority1080
sysctl-wnet.ipv4.conf.wlo5.rp_filter=2
The Starlink dish telemetry sits on its own subnet — but the edge routes to
it for you. The dish’s gRPC status endpoint (192.168.100.1:9200, polled for
the Starlink telemetry card) lives on its own /24, reachable
only over the Starlink Wi-Fi gateway. Once a Starlink uplink is configured, the
edge re-asserts 192.168.100.0/24 via the leg’s gateway (main table, replace
semantics) on every poll cycle, so it survives a re-associate or new lease with
no operator action. This needs CAP_NET_ADMIN, which the packaged systemd unit
grants. Only on a non-systemd run without that capability do you add it by hand:
The bond-leg egress route (the table 80 default + oif rule above) is yours
to maintain — it’s flushed whenever the Wi-Fi link cycles (re-associate / new
lease), so if the leg “disappears” after working, check ip route and
ip rule list first. The dish telemetry route is kept up by the edge itself,
so it survives link cycles automatically. The dev testbed re-applies the bond-leg
route from start-infrastructure.sh and post-reboot.sh.
ping-Iwan5g11-c21.1.1.1# device-bound → out the 5G path
iprouteget1.1.1.1oifwan5g11# shows: via 10.0.11.1 dev wan5g11
curl--interfacewan5g11-shttps://api.ipify.org# each modem → different carrier IP
Run the curl test per interface — each should report a different public
IP (one per carrier/SIM). That’s the clearest proof the paths are truly
independent.
Each bond path pins to one modem interface. In a
bonded output, every path targets the
same remote edge (your bond peer) but leaves via a different interface
(and a matching per-path port the receiver binds):
For Design A, set interface to the physical NIC instead (eth1, eth2).
The far-end bonded input binds matching
per-path ports (0.0.0.0:5000, :5001) with the same path ids.
SO_BINDTODEVICE prefers CAP_NET_RAW on Linux — it’s the hard
TX + RX device bind. Grant it without running the edge as root:
Terminal window
sudosetcapcap_net_raw+ep/path/to/bilbycast-edge
or add AmbientCapabilities=CAP_NET_RAW to the systemd unit. Without the
capability the edge falls back automatically to the unprivileged
IP_UNICAST_IF egress hint — the leg still leaves the right NIC, but the
hint is TX-only (it doesn’t device-bind the receive side), so on a host with
overlapping subnets the strict SO_BINDTODEVICE path is still preferred.
macOS/FreeBSD use IP_BOUND_IF and need no privilege.
Fit the datagram size to the cellular uplink, not the LAN. The host↔router
link is gigabit/1500, but each router’s cellular WAN is still MTU-constrained
and CGNAT paths black-hole IP fragments — so an oversized bond datagram is lost
wholesale. Set path_mtu on the bonded output to the smallest measured
uplink MTU (measure with ping -M do) so the sender re-chunks the TS to fit. See
Fitting datagrams to the path MTU.
Everything else about the bond — scheduler choice, ARQ/NACK tuning, monitoring,
stats — is on the Multi-Path Bonding page.
PoE budget: confirm the switch’s total PoE budget covers all modems
(OTD500 ≤ ~30 W each). PoE++ switches negotiate down to the OTD500’s
802.3af/at automatically.
Surge protection: fit an Ethernet surge arrestor on each outdoor run, and
ground it. The modems are outside; that copper carries surge into your gear.
Weatherproofing: seal cable glands, leave a drip loop, and use
outdoor/UV-rated, ideally shielded, cable.
MTU: keep modem-facing interfaces at 1500 (the modem LAN ports are
gigabit). A VLAN sub-interface’s MTU must be ≤ its parent’s.
All modems unreachable / collide on setup — factory default is
192.168.1.1 on every unit; re-IP one at a time or pre-assign switch VLANs.
Bond paths all leave via one modem — the interfaces aren’t pinned, or the
per-interface default route is missing. Check ip route get <peer> oif <if>
and confirm interface is set on each path.
Path interface up but no internet — missing/duplicate default route, or
rp_filter dropping returns. Verify with the ping -I <if> and curl --interface tests; set rp_filter = 2 on that interface.
Untagged traffic leaks between paths (Design B) — a modem VLAN is acting
as the trunk’s native VLAN; keep modem VLANs strictly tagged.
SO_BINDTODEVICE permission denied — CAP_NET_RAW not granted, so the
edge fell back to the unprivileged IP_UNICAST_IF hint (logged at startup).
The leg still egresses the right NIC; grant CAP_NET_RAW (see above) only if
you need the hard RX-side bind on a host with overlapping subnets.
