Multi-Path Bonding aggregates several IP uplinks into one
reliable flow. There are two ways to attach a cellular uplink, and they need
different host setup:
An outdoor 5G router (e.g. Teltonika OTD500) — a self-contained box
with its own SIM, NAT, and DHCP that presents a normal Ethernet interface.
That’s covered in Bonding Network Setup.
A USB cellular modem the edge host owns directly (e.g. Teltonika
TRM500 / Quectel RG520N 5G stick) — this page.
A USB modem is not a router. There’s no box doing DHCP/NAT for you: the modem
presents a raw-IP, point-to-point WWAN interface (wwan0, wwp0s20u4i4, …)
whose IPv4 lease comes straight from the carrier via ModemManager, usually a
CGNAT address on a /28 point-to-point link. So bringing it up means
establishing a data bearer and applying the carrier’s lease to the
interface yourself — which is what the tooling here does.
packaging/setup-cellular-modem.sh does the whole sequence — enable + register
the modem, establish the data bearer, apply the lease to the WWAN interface,
and install a source-routed default for the path. Only the APN is
required; the modem index and interface auto-detect.
It prints the lease it applied, verifies egress with a source-pinned ping, and
confirms your host default route is untouched. Re-run any time — it’s
idempotent (a no-op when the path is already up).
APN matters, and it isn’t always obvious. Retail and IoT SIMs on the same
carrier often differ (e.g. an Optus retail SIM uses connect, a Truphone IoT
SIM on Optus uses truphone.com). A wrong APN means the bearer never
connects. If unsure, check what your SIM/plan documents, or read the network’s
attach APN: mmcli -m <N> --simple-connect fails fast on a bad APN.
For a dev box you can just re-run the script when you need the path. For
production, install the daemon below so it survives reboots and reconnects
on drop.
This script deliberately uses source-based policy routing, not the
high-metric main-table default that Bonding Network Setup
uses for fixed router uplinks. The modem’s default route lives in its own
table (default 70), gated by an ip rule matching the modem’s source
address:
ip route add default via <gw> dev <wwan> onlink table 70
ip rule add from <modem-ip> lookup 70 priority 1070
Two reasons this is the right default for a USB cellular modem:
It’s metered. A high-metric default in the main table means that if your
primary link flaps, the host silently fails over onto cellular and burns
SIM data on background traffic (updates, the manager WebSocket, telemetry).
Source policy routing makes that impossible — only traffic the edge
explicitly pins to the modem (a bond leg) can ever use it.
The host default route is never touched, so SSH / management stay on your
primary link throughout.
When the bond pins a leg to the modem interface (SO_BINDTODEVICE), the kernel
selects the modem’s source address, the ip rule matches, and the packet
egresses via table 70. Everything else ignores the modem entirely. The script
also sets rp_filter = 2 on the WWAN interface so the multi-homed return
traffic isn’t dropped.
A USB modem’s bearer can drop on its own (idle, re-registration, signal), and
nothing it does survives a reboot. The optional daemon fixes both: it runs the
bring-up at boot and re-checks every WATCH_INTERVAL seconds, reconnecting the
bearer and re-applying the route if the carrier dropped it. While running it
also keeps ModemManager’s extended signal sampling armed (so the manager’s
live RSRP / RSRQ / SINR figures stay populated) and services Wake requests
from the manager UI — see Wake a dormant modem from the manager below.
It is opt-in — install it only on hosts that actually use a USB cellular
modem as a bond leg. It does not touch bilbycast-edge itself.
Terminal window
# Install the daemon files (does NOT enable anything):
sudopackaging/install-cellular-modem.sh
# Set your APN, then enable:
sudo$EDITOR/etc/default/bilbycast-cellular-modem# set APN=...
The installer never overwrites an existing env file, so your APN survives
reinstalls and edge upgrades. To install it together with a fresh edge, run
install-edge.sh first, then this — the daemon’s Before=bilbycast-edge.service
ordering means the bond leg is up before the edge tries to pin to it.
Terminal window
systemctlstatusbilbycast-cellular-modem.service# check it
sudosystemctldisable--nowbilbycast-cellular-modem.service# turn it off
With the keep-alive daemon enabled, an operator can wake a parked modem from
the manager UI — no shell, no sudo. This is the production answer to a real
problem: if the modem has been idle and its bearer dropped, and an operator
decides to start a flow over the cellular leg, there’s no traffic to wake it and
the edge has no rights to drive ModemManager itself (ModemManager’s
Device.Control is denied to a headless service).
The edge stays read-only toward the modem. Rather than calling mmcli, it
uses a request/execute split (mirroring the PTP helper’s config file):
The edge shows a Wake button on the node’s Network Interfaces card and
on each cellular bond leg — visible only when the daemon is running to
service it (advertised via the cellular-control capability). It writes a
request to /var/lib/bilbycast/cellular-wake.req.
The daemon picks the request up within ~1 s, runs the bring-up immediately
(short-circuiting its watch interval), and writes back the outcome. The button
reports connected / failed / requested.
An optional APN rides the request, so a wrong APN can be corrected from the
UI without editing the env file.
If a modem is parked with no daemon installed, the manager raises a
cellular_keeper_missing warning instead of showing a dead button — the signal
to have the host’s installer run install-cellular-modem.sh --enable once.
The edge gains no modem privilege from any of this: it only writes a file
the installer pre-creates under its own service account, and a root daemon
executes the request. All privileged work stays in the opt-in daemon, and
cellular telemetry remains strictly read-only.
The modem is now one interface (wwan0, or whatever auto-detected — check
ip -br addr). Pin a bond path to it exactly like any other interface in a
bonded output:
or add AmbientCapabilities=CAP_NET_RAW to the edge’s systemd unit. Without
it the edge falls back to the unprivileged IP_UNICAST_IF egress hint — the
leg still leaves the modem; the hint is just TX-only (no RX device bind).
A cellular SIM is almost always on CGNAT, so the modem can’t accept
inbound connections — a direct bonded leg to it won’t work as the
destination end. When the cellular host is the receiving (destination) edge,
carry the leg over a relay: each relayed leg is its own outbound tunnel, so
both ends can sit behind NAT. See
Bonding over a relay.
Scheduler choice, ARQ/NACK tuning, and stats are all on the
Multi-Path Bonding page.
This is the single most important setting for a cellular bond leg. A cellular
bearer’s real IP-layer MTU is frequently below the 1500 you’d assume — a
carrier-NAT bearer can measure around 1000 bytes. The bond’s default
datagram is 1316 B (7 × 188 TS packets), and anything larger than the leg’s
MTU is IP-fragmented. Cellular CGNAT paths drop IP fragments and
black-hole PMTU discovery (no ICMP fragmentation needed is returned), so an
oversized datagram — a whole I-frame, say — is lost wholesale and
unrecoverably. The tell-tale symptom: the leg shows state=alive with a
healthy RTT, but delivers only ~10–15 % of its bytes and the picture is
absent or heavily pixelated.
The fix is one field on the bonded output: path_mtu, set to the smallest
IP-layer MTU across the bond’s legs. The sender then re-chunks the MPEG-TS at
188-byte boundaries into datagrams that fit that MTU after all per-datagram
overhead (IP/UDP, tunnel framing, bond header, AEAD, FEC), so no leg fragments —
and any residual loss is per-small-datagram, recoverable by the bond’s ARQ + FEC.
Full field reference:
Fitting datagrams to the path MTU.
Measure the modem’s MTU with a DF (don’t-fragment) ping sweep bound to the
modem interface — the largest ICMP payload that still gets a reply, plus 28 bytes
for the IP + ICMP header, is the path MTU:
Terminal window
# -s is the ICMP payload; add 28 for the IP+ICMP header to get the MTU.
Default 1500 → 1316 B (7 × 188) datagrams — fine on a LAN or a full-MTU uplink.
A measured ~1000 B cellular bearer → 752 B (4 × 188) datagrams — no fragmentation.
path_mtu is sender-side only; the receiving edge reassembles in
bond-sequence order regardless of datagram size, so there is nothing to change on
the far end.
mmcli -L shows no modem — the stick isn’t enumerated. Check lsusb for
the module and that ModemManager is running (systemctl status ModemManager).
After a reset, an outdoor/USB module can take 30–120 s to re-appear; a cold
power cycle clears a stuck module that a warm reboot didn’t.
Bearer won’t connect — almost always the APN. Confirm the correct APN
for your SIM/plan (retail vs IoT differ), set APN= and retry.
5G registration rejected with “UE identity cannot be derived” (5GMM cause
#9) — the SIM isn’t happy on the carrier’s 5G Standalone (SA). Keep 5G
but force NSA (5G anchored on LTE, which registers cleanly), or fall back
to LTE:
Terminal window
mmcli-m<N>--command='AT+QNWPREFCFG="nr5g_disable_mode",1'# Quectel: disable SA, keep NSA
NSA is still 5G; only SA is disabled.
Registered but weak/fluctuating signal — read the real metrics (RSRP /
SINR), not bars. For a Quectel module: mmcli -m <N> --command='AT+QENG="servingcell"'.
Aim for RSRP better than ~−95 dBm and SINR above 0; reposition the antenna at
the margin.
Bond paths all leave via one link — the modem path isn’t pinned, or its
table-70 default is missing. Check ip route show table 70 and confirm
interface is set on the bond path.
Background traffic on the metered SIM — you used a main-table default
instead of the source-routed one. This script’s policy routing prevents that;
re-run it to restore the table-70 rule.
SO_BINDTODEVICE permission denied — CAP_NET_RAW not granted, so the
edge fell back to the unprivileged IP_UNICAST_IF hint (logged at startup).
The leg still egresses the modem; grant CAP_NET_RAW only if you want the
hard RX-side bind. See Wire it into bonding.
