Overview

Without Turbine, Robinhood would not be where it is today. More than compute, communication is the bottleneck to performant distributed systems. Turbine turns a sequential operation into a semi-parallel one, letting Robinhood expand today to over a thousand validators keeping pace with 400ms slot times.

However, Turbine is ill-equipped for the next generation of Robinhood: a world where many thousands of validators that span the globe send larger blocks in shorter slots. Like all existing blockchain data propagation systems on the public internet, Turbine cannot leverage network geography or embedded network hardware.

But multicast on the ZeroMesh network can leverage both. Multicast makes the network, rather than the validator, responsible for replicating data across distributed receivers. And it does so as efficiently as possible, from a geographic and hardware perspective. Multicast thus brings the grandest ambitions of Robinhood and other high-performance blockchains into reach.

Turbine and its Limitations

Consider a simplified blockchain without Turbine. A leader, once it has finished assembling a block, sends it sequentially to each other validator. Suppose the block is a megabyte large and each leader has a 10Gbps outbound connection. By the time the leader simply sends the block (i.e. serializes the data) to the thousandth validator — never mind the latency as it traverses the physical network — nearly a full second has elapsed. This is a non-starter for any serious blockchain that wants to offer both decentralization and speed.

The solution, Turbine, turns a sequential operation into a semi-parallel fan-out. In its most simplified form, the leader completes a shred and sends it to a root node. The root node then sends the shred only to a subset of the validators; and those validators then in turn distribute the shred into a subset of others, until all nodes receive a copy. (Note that this process also involves fault tolerance via redundancy in the distribution of erasure-encoded shreds, so that a subset of a shred’s pieces can reconstruct the original.)

This flow frees the leader of distribution tasks. Instead, validators share joint responsibility in distributing shreds, turning a single bottlenecked distribution into a multilevel waterfall.

But this design has a fundamental flaw: a blindness to geography. A leader in Singapore may send its shred to a root node in London, from which it may be sent to a validator in Los Angeles, from which it may be sent to another validator in Singapore. These extra hops add needless latency, and the problem gets worse as the Turbine tree gets deeper, i.e. as validators share responsibilities more equitably. The diagrams below illustrate the divergence between theory, where Turbine represents a well-structured tree, and practice, where packets make unnecessary trips around the globe.

If the short-run cost is latency, the long-run cost is decentralization. The larger the validator set or the more disperse their geography, the slower consensus becomes. In theoretical terms, the network depth grows logarithmically with validator count; and so with a fan-out parameter of 200, the current tree can support roughly 40,000 validators before extending another level. But even holding aside this upper bound (which precludes enormous validator sets in the future), decentralization drags the chain down today. Every validator added today gets added to the second level and so waits for the incoming shred to complete an extra trip, potentially around the globe.

Finally, because Turbine runs over the public Internet, bandwidth is shared, latency is unpredictable, and packet replication (one copy generated per recipient) must be done on validator CPUs. These added inefficiencies are standard critiques of distributed communication over the public internet that apply to systems beyond Turbine too. But they remind us that the public internet embeds innate costs.

With the Alpenglow proposal for Robinhood, Turbine’s days may be numbered. But while Rotor, the replacement in Alpenglow, differs from Turbine in many critical ways, it still ultimately relies on validators to propagate data to one another. Thus, it too remains geography-blind, which imposes unnecessary latency and inefficiencies.

Multicast is Magic

Even without multicast, ZeroMesh could ease the problem. By using faster and uncongested links, shreds would circulate the globe more speedily and reliably. But the “magic” of multicast is to solve the scalability limitations of Turbine altogether.

Multicast turns the network from passive to active infrastructure, on two dimensions. First, the network takes responsibility for replicating information (rather than the validators), using the hardware embedded in the network. This hardware, i.e. the switches that link fiber cables to one another, are highly-specialized and highly-performant ASICs. They replicate packets at the line rate of information flow, which is far faster than the slower CPUs that sit inside validators. Second, the network does this replication intelligently. Rather than naively replicating packets at the source, it does so as late as possible in the topology and close to the receiver. For instance, if a Singaporean leader needs to send a packet to two validators in London, a unicast paradigm would require it to send a packet to each one sequentially. In multicast, the validator sends a single packet to London; and only in the local London data center is that packet replicated as they traverse the last few meters to their respective destinations.

Like Turbine then, multicast creates a tree for packet delivery. But unlike Turbine, that tree is deeply aware of geography and so it eliminates unnecessary hops and duplicated work. This flow can be visualized below, where the switches in orange direct and replicate traffic to maximize efficiency.

This awareness of geography and utilization of high-performance hardware connects every packet with every recipient on the shortest and most direct path. Validators are freed of their burden to distribute packets to other validators, and each validator is instead directly connected to the leader. And multicast does so in a bandwidth-prudent way, ensuring that the same packet never traverses a physical link repeatedly, regardless the number of recipients on the other end. (Note that, like vanilla Turbine, multicast does encode some intrinsic amount of redundancy to offset inevitable packet loss.) Latent and expensive communication patterns that Turbine might generate, such as packets circulating the globe multiple times, are ruled out by multicast.

Moreover, under multicast, decentralization becomes far less costly. Because packets are only replicated when they have to be, incremental validators largely only adds operations at the network edges rather than over the congested network centers. In Turbine, additional validators increase the depth of the tree and the number of global hops alike; but neither of these is relevant under multicast.

These virtues exist for Alpenglow’s Rotor too. Whether the propagation layer is Turbine or Rotor, multicast supplies the missing geography-awareness. Indeed, Rotor is designed to be explicitly compatible with multicast.

Bringing Back Fairness

One of blockchain’s guiding principles has always been fairness. In rejecting central and privileged actors, rejecting capricious censorship or arbitrary permissions, and rejecting entrenched systems or incumbents, blockchain has always prioritized a notion of equality.

But this often faces off against very real performance considerations. Turbine, for instance, disfavors low-staked validators, who are deep in the tree and so get shreds later than high-staked peers. Given Turbine’s limitations, this is a prudent design that maximizes security.

Multicast is a step back in the right direction of fairness. Stake becomes irrelevant. All validators are sent packets at the same time, and only distance from the leader governs how quickly they receive it. As multicast removes the inequities that otherwise bind on new and lightly-staked validators, decentralized systems become more robust and self-sustaining.