Why PPPoE throughput pins to one CPU core, and how the BNG spreads it

A software BNG scales by spreading packets across worker cores: the NIC hashes each packet to a receive queue, and one worker owns one queue, so more cores means more throughput. That model breaks for two of the most important traffic types on a BNG: PPPoE sessions and L2TP tunnels. The failure is invisible until you look at per-core load. This note explains why, and what Netavo BNG does about it.

The problem: RSS can't see the flow

Receive-Side Scaling (RSS) works by hashing a header the NIC can parse, usually the outer IP 5-tuple, and using that hash to pick a receive queue. For plain routed IP that spreads load evenly. For subscriber access it does not:

• PPPoE uses ethertypes 0x8863 (discovery) and 0x8864 (session). That is not IP at the outer ethertype, and most NIC RSS parsers don't recognise it. Every PPPoE frame hashes the same way and lands on receive queue 0.
• L2TP into an LNS is worse. Every session inside one tunnel shares the same outer header: LAC-IP:1701 → LNS-IP:1701 over UDP. One tunnel has one 5-tuple, so all its sessions land on one queue and one core, no matter how many it carries.

The result is silent. Forwarding works; the box looks healthy; but an entire traffic class is pinned to a single worker core regardless of how many cores you configured. You discover it when one core sits at 100% while the rest idle, and throughput caps far below what the hardware should deliver.

The reference platform makes this concrete: the Intel X710 (the project's reference NIC) is a plain outer-tuple RSS device. On an X710 or an older 82599, PPPoE and single-tunnel L2TP pin to one core. The data plane has to handle this as a normal operating condition.

The fix: software RSS spray

Netavo BNG detects a port whose traffic the NIC cannot hash and inserts a software dispatch stage in front of the workers. Design doc 23 (Software RSS Spray) covers it in full; the shape is:

• One worker (worker 0, acting as a dual-role dispatcher) is the only thread polling the sprayed port's receive queues.
• For each packet it computes a session-affinity hash and enqueues the packet onto the target worker's single-producer/single-consumer ring.
• Every worker drains its own ring and runs the normal pipeline, egressing on its own transmit queue.

The hash sends all packets of one session to the same worker, preserving their order while spreading sessions evenly:

Keys are mixed with splitmix64 and reduced across the worker count. The dispatch cost is roughly one Ethernet parse plus one ring write, about an order of magnitude less than full pipeline processing. A single Haswell-class core dispatches well beyond worst-case 10G packet rates. The only added latency is a single ring hop, in microseconds.

You don't configure the mechanism, you declare the traffic

You declare what each port carries, and the box decides whether to turn on spray:

"dataPlane": {
  "subscriberInterfaces": [
    { "ifname": "eno1", "pci": "0000:04:00.0", "rxQueues": 8,
      "accessProtocols": ["pppoe", "ipoe"], "spray": "auto" }
  ]
}

At startup the data plane resolves, per port, whether spray is needed: the declared accessProtocols, what the NIC's RSS can hash, and the tri-state spray override (auto / on / off). On auto (the default), a port carrying PPPoE on an X710 is sprayed; the same declaration on a NIC that hashes PPPoE in hardware is not. You set on or off only to override the auto-detect for a specific test.

The hardware path

Newer NICs can do this hashing themselves. An Intel E810 with the right DDP (Dynamic Device Personalization) package reports that it can hash PPPoE and L2TPv2 inner sessions, and on such a port the data plane keeps pure hardware RSS and skips the software dispatch entirely. The fan-out happens in the NIC.

Netavo does not trust the advertised capability bits blindly. The DPDK port setup requests the extended hash types (PPPoE, L2TP) and, if the device rejects them (usually because of a DDP package mismatch), it retries with outer IP/TCP/UDP only. The granted capabilities then shrink, and the spray auto-detect sees the reduced caps and turns software spray on. So a box with the right NIC and DDP package gets the hardware path; the same box with the package missing silently falls back to software spray and still spreads the load. Either way the operator gets working multi-core PPPoE.

A practical consequence: if you deploy an E810 expecting hardware PPPoE hashing and the DDP package isn't loaded, nothing breaks; the port falls back to the software dispatcher. Check the startup log to see which path a port took rather than assuming the NIC datasheet applies.

Honest edges

• Non-sprayed ports still get watched. A multi-queue port that should spread in hardware but doesn't (a misconfigured NIC) would otherwise fail silently. A low-rate control-thread watchdog samples the queue distribution and emits a one-shot warning if more than ~90% of a sample window arrives on queue 0.
• The ring can drop. The dispatch ring is bounded (511 usable slots). If a worker falls far enough behind to fill its ring, the dispatcher drops at the front door rather than blocking the whole port. This protects every other worker.
• The counters aren't exported yet. Per-ring enqueue and drop counts exist in the data plane but are not yet surfaced over the CLI or SNMP. Exporting them remains a known follow-up.
• This is about packet distribution, not a sessions-per-core number. Spray makes sure your cores are actually used; what each core then forwards is set by CPU and feature mix, and the per-class throughput figures and method are on the performance page.

See it on your own NIC

The free evaluation build is the production build with every feature enabled. Bring it up on your target hardware, declare a PPPoE access port, and watch per-core load as sessions connect. On an X710 you'll see the dispatcher spread them; on an E810 with DDP you'll see hardware RSS do it. Start at netavobng.com/evaluation; the data-plane design docs ship in the manual.