Synchronizing data between Bitcoin nodes just got easier

Minisketch is a new solution that tries to solve an old problem.

Led by Blockstream Co-Founder Pieter Wuille, Bitcoin Core Contributor and Co-Founder of Blockstream, Gregory Maxwell, and Blockstream Software Engineer Gleb Naumenko, the open source initiative aims to achieve solid alignment between the mempools of each full node.

“Matching sets, in short, the problem is figuring out what the differences are between two sets stored on different computers while minimizing how much data has to be passed between them. In particular, it tries to do this while sending less data than the entire data set, ”Wuille told Bitcoin Magazine

For Bitcoin, this means recognizing the differences in the transaction data between the nodes. Maxwell likened set voting to the process of syncing your phone’s contact list with someone else who has many of the same contacts.

“You could send them your whole list, but it won’t fit on a postcard and it would be pretty wasteful as you already know most of your contacts … It is actually possible to communicate all of your contacts by sending just as much information as you can. what the differences are between your lists, even if you don’t know beforehand what the actual differences are, ”Maxwell told Bitcoin Magazine.

In short, set voting would reduce the bandwidth required to run an entire node on the Bitcoin network by minimizing how much data each node transmits to one another. This would effectively allow nodes to synchronize the data in their mempools more efficiently.

Set Vote Breakdown

The problem minisketch is trying to fix is ​​not blockchain specific.

Rate matching is a constant frustration that every distributed system struggles with. By and large, this simply means that two or more parties in a distributed network have different sets of data, and to reconcile that they need to figure out what pieces of data they are missing – and what pieces the other party is missing.

With Bitcoin, this data is a transaction. These transactions are routed from node to node until miners pick them up and put them in new blocks.

The problem is that the order of transactions can vary from mempool to mempool (the queues that list incoming transactions to be included in new blocks). This means that there can (and is) a discrepancy in the order of transactions between mempools and newly routed blocks.

“Bitcoin nodes have a… problem with relaying transactions between themselves. Any given node has mostly had the same transactions as each of its peers – received from other links, but not exactly the same. Nowadays, nodes waste a lot of bandwidth just trying to figure out who to send what to whom [data]”Said Maxwell.

How Minisketch bridges the gap

As an implementation of the PinSketch algorithm, minisketch constructs a library of data that is used to construct sentence sketches (ie, for this application, sets of transaction data). Nodes and miners can then use these sets for a compact set comparison.

Simply put, the solution would allow node operators to compare notes on transactional data. It would allow them to sketch (build) sets (lists) of transactions, and the program would cross-check those sets to see what data was in one set, but not both. Instead of spending the time and energy exposing all of this data to each other, however, with Minisketch nodes only need to know the difference between their transaction sets to sketch a complete set.

As Wuille explained, in practice it would look like this:

“If we simplify it down to a single difference, it’s easy to see how this can work:

  • Let’s say I have {3,5,7,11} and you have {3,5,7,9,11}, so the difference is {9}.
  • We both calculate the sum of our elements, so I get 3 + 5 + 7 + 11 = 26, and you get 3 + 5 + 7 + 9 + 11 = 35.
  • I will send you my sum 26 and you will deduct it from your sum; the difference is 9.

“It works, but it’s limited to finding a single difference. Minisketch generalizes this by sending different kinds of “sums” of the data. The result is that with N different sums you can find N differences … As long as the number of differences between the sentences is not greater than the number of ‘sums’ sent, minisketch will always find all the differences. “

If implemented successfully, this record matching could make transaction routing between nodes more efficient. Along with other improvements in the works to Bitcoin’s infrastructure, this could significantly reduce the transmission load on each node, Maxwell said.

“I took a measurement some time ago and found that the transaction relay accounts for about 87 percent of a node’s bandwidth usage. This was before compact blocks, so now the number may be larger. Our simulation results show that with a combination of improvements, including mini-sketch, we can reduce relay overheads by 40 times. “

According to Wuille, the minisketch solution could also create “a more robust network”. Maxwell expanded this thinking by suggesting that nodes with the protocol could use their saved bandwidth to connect to 16 to 24 other nodes instead of the standard 8, “which would make some theoretical attacks even more difficult without adding more bandwidth.” consume”. ,” he claimed.

Maxwell also hopes that one day Minisketch could also be used to revamp block distribution. “There were other protocols for block propagation with IBLT (Invertible Bloom Lookup Table) that Minisketch would do much better,” he said, although he admitted that such a solution was “not urgent” because “[compact] Blocks already ensure that the block transfer itself only consumes about 4 MB per node per day. Even if improvements were to reduce that number to zero, it wouldn’t make much of a difference to the users or the network. ”Instead, he envisions that this solution would be more useful for“ very low transmission like Blockstream’s satellite ”.

Wuille indicated that minisketch will reduce node bandwidth requirements and have a higher probability of success than IBLT, but admitted that IBLT would be faster for larger datasets. Maxwell added that “IBLT is very inefficient for less than a few hundred differences,” while Minisketch is more efficient for smaller data sets.

Minisketch, however, is still in its early stages; Wuille stated that an actual border crossing point is still a while away and that the assumption is subject to a number of factors.

Maxwell reiterated his observations up to a point and noted that the protocol is also not part of Bitcoin’s network consensus. If node operators felt comfortable enough and wanted to try to improve their transaction relays, they could use minisketch even without its ubiquitous adoption on the network.

“One useful point is that relay mechanisms are not part of the Bitcoin consensus. You and I could start using an improved protocol between us regardless of what other people are doing. This means that improvements to the relay will only be delayed by ordinary protocol / software engineering considerations – we need to build it, verify it, integrate it, etc. – but unlike a consensus change, it doesn’t depend on anyone agreeing to it except the people who choose to use it. “

Comments are closed.