Words on Bitcoin Vormgeving, Privacy, Security and Crypto, by Sergio Demian Lerner
(This postbode is an updated re-post of a previous postbode te RSK blog)
Te a nutshell, storage rent is a toverfee users pay te order to have their accounts, contracts and memory live on the network at any time, so their gegevens can be accessed quick and at a low cost. Storage rent does not fulfill any purpose te the short-term, but is required to assure the long term viability of a cryptocurrency toneel. However the vormgeving and implementation of efficient storage rent is very tricky. Te many cases the amount of memory a user persists te a contract is so puny that the rent becomes a micro-transaction, and the cost of processing the rent payment is higher than the amount paid itself. A seemly reasonable implementation of storage rent may fail to consider this cost imbalance. One also has to consider the CPU and space costs of storage accounting and the cost of managing misbehaving contracts. This management tasks can can lightly become bottlenecks to scalability. At RSK wij considered several designs, their pros and cons, until wij lodged on the current treatment. You can see the different RSKIPs signifying the distinct approaches te our github repository. Wij voorwaarde note that Ethereum community also discussed adding storage rent to their toneel, but the proposal wasgoed dismissed
The Cost of Utter Knots
Spil the blockchain state at the best block grows, the cost of maintaining a utter knot is predominated by the cost of quick access to the blockchain state and not from the cost of accessing historic blockchain gegevens, Accessing the blockchain state vereiste be swift to increase the throughput and security of the network (reducing the block propagation critical path), while accessing historic parts of the blockchain doesn’t have any hard efficiency constraint. Ter other words, if a fresh block verification takes too much time, then miners are incentivized to mine empty blocks or empty uncles. However, there is no need for rapid access to historic blockchain gegevens spil it can be downloaded leisurely te background from different peers even while the total knot verifies the latest blocks, if trusted checkpoints are used temporarily. Therefore the most valuable resource any cryptocurrency attempts to protect from bloat is the blockchain state, and not the blockchain blocks. The blockchain state te RSK grows when fresh accounts or contracts are created, and when contracts request extra contract storage.
Who Should Pay for Blockchain State Storage
One of the unsolved problems of Ethereum is that state storage can be acquired at a low cost, or even zero cost, and never released, forcing all utter knots to store that state information forever. The storage becomes free if there is no transaction backlog, so there is free block gas. There are almost no examples te real-world commerce where users acquire eternal rights overheen a property that requires continued maintenance performed by third parties, but it is acquired by a single non-recurring payment. But that is the case of blockchain state storage te Ethereum, and, to a lower extend, UTXO storage ter Bitcoin. Maintaining the state space requires paying for electric current and the amortization cost of storage hardware, and the cost vereiste be multiplied for every utter knot ter existence. It can be argued that total knots are altruistic, and therefore they are willing to incur ter any state storage cost the cryptocurrency users request. While this may have bot partially true for Bitcoin knots ter the past, the altruistic behavior has stopped greatly spil the blockchain size grown. The number of Bitcoin knots is enhancing leisurely but the number of Bitcoin users has enlargened considerably more, so it’s unclear if knot count will go after bitcoin request. It is expected that block pruning and sharding mechanisms permit fresh users to commit a certain smaller amount of historic blockchain storage, but yet the state voorwaarde be maintained ter total for block verification. Requesting all state information required to verify a block (read or written branches of the current state trie) is generally not possible ter real-time when propagating a fresh block, incurs te fat bandwidth consumption. RSKIP58 proposes a partial solution, where knots send the modified state records but not the records read by smart-contracts during execution. This permits a form of header-only block propagation.
If the use of state storage is not protected from manhandle, wij risk to price out utter knots. Controlling the state size reduces the centralization pressure while maintaining a free market. Considering long term risks of state bloat and the uncertainty of Moore’s law and similar trends te the future, is clear that preventively users should pay a state storage rent. Thesis central economic decisions cannot be straks applied without violating the community contract.
Who Should Rent be Paid?
At a very first glance, spil total knots store the blockchain and the state, it seems that storage rent should be paid to utter knots. However, HDD storage costs keep decreasing at a rate of 40% vanaf year (this trend is known spil Kryder Law), so under this trend the real cost of storage is bounded. A similar trend exists for SDD storage. Electro-therapy cost of persistent storage tend to zero if the storage is not accessed, but but it increases with the number of read or write accesses vanaf 2nd.
Storage rent targets mining knots storage much more than non-mining knots storage. The bloat of state affects mainly miners, who cannot embark mining a child block containing transactions before the parent block has bot fully validated. If the state does not gezond ter RAM, or te SDD, then state access is greatly slowed down, and miners vereiste mine empty blocks until they fully verify a block. This establishes strong incentives for centralization, spil fatter pools do not suffer this delay. Miners have incentives to use the quicker and more reliable storage and quicker CPUs to reduce the verification delay, so it seems natural that storage rent is mainly paid to miners. Even if wij would like to redirect part of the fees to total knots, there is no tested secure protocol to perform this payment. Ter RSK we’re testing the Proof of Unique Blockchain Storage (PoUBS) protocol I designed a few years ago and I introduced te 2018 at Devcon3. PoUBS is presently the only protocol that has the potential to solve full-node prize problem without trusted parties.
The Problem of Storage rent on DApps
For contracts that are managed by a central entity, it’s clear that the holder should pay the storage rent. But for some community clever contracts it is unclear who should pay for this rent. Many contracts (and most likely the most interesting ones) are crowd-contracts: programs that are fueled and used by the crowd, without any single manager. Crowd-contracts can consume a lotsbestemming of contract storage, but no single user is ter position of carrying the cargo of the paying the rent. No single user will be willing to.
One can imagine that a well designed crowd-contract should have a revenue generation method for paying for the storage rent. For example, each crowd-contract method call should also pay to a special rent account where the crowd-contract collects all rent-oriented income. However, this treatment has several problems:
- Most crowd-contracts are meant to be immutable, such revenue collecting method voorwaarde be defined before the contract is deployed. If there is a onmiddellijk relation inbetween a user and the memory it consumes, each user can pay a partial rent independently. But if this is not the case, and most users only receive a service, then it will be very unclear what proportion should be paid by each operation to collect enough funds before the rent deadline.
- The cost ter gas required to manage the rent collection process may be so high that makes the service suggested too expensive. The collection process involves several steps such spil computing the amount of rent each user voorwaarde pay, collecting rents, keeping a registry of which users have or haven’t payed the rent, removing the gegevens of users who did not pay the rent, etc.
Te an ideal world, a DNS-like contract would manage an independent balance to pay a rent for each name registered, However, spil previously stated, this treatment may be very inefficient spil each rent payment for a petite chunk of storage may represent a hundredth of a US cent. Why encapsulate this payment into a transaction that costs 100 times more?
The Problems with Collecting Rents for Immobilized Periods
There are several complications that arise when attempting to implement storage rent spil a payment for a immobilized period, ending at a certain specific deadline. Because RSK/Ethereum cannot schedule code execution, triggering the rent-paying code would need to be done from a message coming from the outside world, before the rent deadline comes. The payment interval should be long enough to prevent rent payment censorship from miners. If deterministic deadlines are set some motionless time after the contract creation time, then the simultaneous creation of numerous contracts te the same block can lead to a high number of contracts requesting deadline checks at the same time. Therefore rent-deadline events should be chosen randomly at contract creation time, so events become evenly distributed.
Monthly (or shorter) payment periods add too much pressure on users. Spil a comparison, owners of domain names choose to pay annual fees, rather than worrying about a monthly toverfee. With anual payments, plain account rent payments would consume more computing resources to verify than the rent being paid.
At RSK wij tested a vormgeving based on immovable rent periods and realized it increases the complexity and cost of contract programming. Therefore wij determined that the RSK verhoging would use a simpler treatment and collect rent for the intervals inbetween uses, rather than for long immobilized periods of time. Also it will pospone rent payments until the amount to pay is overheen a ondergrens threshold.
Solutions to Micro-rents
One way to tackle the problem of micro-rents is by using a probabilistic treatment. A random user (identified by its account) is pseudo-randomly chosen to pay the rent for all the users, for a period. Another probabilistic treatment is pseudo-randomly selecting one every 100 transactions that call a specific contract and force it to pay the total rent. The pseudo-random selection would be based on the block hash. However this implies that the result of this transaction cannot be reflected on the world-state of the current block, but on the next. Also if the parent block hash is used spil the random seed, then this permits miners to re-order transactions to favor certain users not to pay the rent everzwijn. Therefore a probabilistic treatment seems inadequate.
A better way to avoid thesis problems is that every operation on a contract pays a rent proportional to the the amount storage the contract has acquired multiplied by the last contract inactivity period. This is not entirely fair, spil a user who uses the contract a single time is compelled to pay for all memory previously acquired. However, this system is fair assuming that:
- miners do not reorder transactions ter a block to favor some users (because the rents are micro-payments, there is little incentive to).
- contract calls are spaced evenly overheen time, and not concentrated ter a few blocks.
Under thesis assumptions, users will pay a share of the contract rent ter proportion to their usage rate.
To Kill or not to Kill
One has to determine what to do if a contract does not pay the rent (or no user pays the rent for a contract). Killing the offending contract seems spil an shocking activity: the user asset balances would be burned if the user forgets to pay the rent. A softer alternative is required. At RSK wij analyzed two options:
Hibernation means a contract state is substituted by a single hash digest of it. Also, the block height where the contract wasgoed hibernated is stored. Zometeen, the user can recover the contract, including its balance, by providing the missing pre-image information. A user can inspect the stored hibernation block height to query a peer and obtain the missing gegevens, if a peer still has a copy of it.
Cache eviction means to budge the contract gegevens to a storage device with lower access time, and liquidate it from a swifter more expensive memory. For example, high availability contracts could be kept ter RAM, while infrequently used contracts are moved to SSD storage.
Wij determined to begin implementing cache eviction, and zometeen stir to compelled hibernation if cache eviction does not vertoning ter practice the scalability boost expected. Also because some RSK/Ethereum accounts consume so little space that hibernation actually increases the amount of gegevens to store. Hibernation of brief accounts can still be beneficial if wij permit cascaded hibernation of account knots te a binary tree. However, this feature requires an extra binary tree to uniquely number all leaf knots corresponding to contracts/accounts, spil specified by RSKIP18.
Packing Rent Payments into Transactions
Because of the low cost of HDD and SSD storage, most rent payments will correspond to micro-transactions. Wij estimate a rent rate te RSK to be approximately 3000 units of gas vanaf 32 bytes of storage vanaf year. If paying the rent requires sending a unique transaction, the fees paid for a transaction (21K gas) far exceed the amount transacted. This is a big overhead for the network and it’s against the common good. Therefore is preferable to piggy-back the payment to a transaction that performs another operation. This saves bandwidth and reduces computation time to a single digital signature verification. For RSK, wij’ve added a fresh field to transactions (gasRentCount), where the transaction holder specified how much gas it will pay for storage rent for all contracts called either directly or indirectly. By using a different gas count field for the rent, wij maintain compatibility with Ethereum contracts that make CALLs with embedded amounts of gas. When no rent needs to be paid, the field can be empty, and it consumes minimal transaction space.
Storage rent does not prevent short-term storage spam attacks but it’s needed for fairness and for long term scalability. Without storage rent, the community may be sacrificing resources for short-term gains and preventing long term success. Storage rent also protects the blockchain against miscalculations and erroneous predictions on technology or blockchain adoption rate. However, implementing storage rent is ingewikkeld, spil many rent payments are micro-transactions, the system implemented vereiste make sure the rent collection cost does introduce a fresh limiting factor to scaling. RSK will implement storage-rent payments te individual transactions, proportional to the target contracts last inactivity period and current storage consumption.
The most significant comparative properties of cryptocurrencies are decentralization, scalability, confidentiality, stability, usability, security. But scalability is always ter conflict with the surplus of the properties. To scale higher, some blockchains sacrifice security, usability or privacy. For example, Bitcoin sacrifices some security because it lacks stateful smart-contracts, so users cannot set daily withdrawal thresholds or use covenants. Ethereum sacrifices privacy, because reusing destination accounts (ter CALL payments) costs less than creating fresh accounts, incentivizing the former overheen the straks. The Meltdown vulnerability, which affects CPUs, or the CRIME vulnerability, which affected the TLS protocol, are catastrophic examples of the difficulty of having both secure/private and maximally efficient systems ter hardware and software.
When I designed QixCoin , the very first Turing-complete cryptocurrency, I determined to create a blockchain than could be turned private, usable and efficient and let the free market determine the value of each feature with respect to its cost. Each feature would be priced according to the resources it consumed. This is the same stance zometeen followed by Ethereum and by the RSK Bitcoin sidechain. Some niche cryptocurrencies adopt an opposed stance, subsidizing one feature, such spil finish confidentiality, to conquer a niche market. But at the same time thesis cryptocurrencies are pricing out users who do not need that feature. Spil an example, the cost of a private zCash transaction is only a fraction higher than a zCash semitransparent transaction, but it consumes much more resources. But the cryptocurrency killer app is still to be discovered, so niche cryptocurrencies only fragment the market and reduce the network effect.
Te last year Ethereum advanced one step te privacy by implementing elliptic curve and paring operations spil native contracts. You can now implement a contract that enables private token transfers. But te order to use this contract, you still need to pay the transaction fees from a standard see-through account, which gets linked to your private transaction, so privacy is still limited. To enable the use of truly private tokens, you need enable the use of the same private tokens to pay for transaction fees. With the advent of account abstraction (te case of Ethereum) and SigVerCode (ter case of RSK), the receiver can pay for the fees, so selective private transactions can be accomplished. This sounds excellent, but when wij take into account the costs of private transactions (te terms of gas consumption), wij realize that private transactions cannot scale to all the transaction volume.
Fortunately most of the transactions wij daily do, like shopping or having meals, do not need to be private. For a billion of unbanked people te the world, what matters most is transaction cost, and selective privacy can be an option.
The Cost of Privacy
Private transactions are costly, both te terms of blockchain space and CPU consumption. Several different methods to achieve transaction privacy have bot invented, Chaumian, zk-SNARKs based such spil zCash, Stadionring Signatures based, such spil Monero, Pedersen commitments based, such spil MimbleWimble, universal re-encryption based, such spil Appecoin. Each cryptographic construction has different resource requirements. Te general, private transactions take around 1-10 millisecond to verify, so 100 tps can be achieved. The current technological verkeersknelpunt is not CPU consumption, but block size and blockchain growth rate. Private transactions tend to be about 2-8 Kbytes te size. There are three components of a standard payment transaction that need to be hidden: transaction source, destination and transaction amount. Hiding transaction source and destination is effortless by using Stadionring signatures and Stealth Addresses. The tighter part seems to be hiding the amount. Three problems arise when attempting to hide the amount: very first, it requires “range proofs” to permit to privately combine and split amounts. 2nd, the range proofs proposed are based on perfect-hiding, computationally-binding commitments, and not the switch sides. This means that with reasonably computational power (e.g. large quantum computers) an attacker can create coins from lean air. Third, range proofs consume large space. For comparison, a range proof may take 2-8 KBytes while a see-through transaction te RSK can be made spil brief spil Ten bytes, using signature aggregation. Therefore, privacy is about 100 times more costly than transparency. Because of the homomorphic properties, both Pedersen and Appecoin commitments do not require range proofs to add two private values, so wij may think wij can reduce the cost when we’re adding value to a private savings account. The bad news is that to pay precies values, or to pay for the transaction fees, switch is required, and splitting the switch requires range proofs. Generally a transaction would need to split a hidden amount into a hidden payment, a hidden switch, and an open transaction toverfee amount. Therefore there is no way to avoid the need to verify range-proofs, even for mostly additive accounts, spil savings accounts are. A workaround is to use immobile denominations (e.g. powers of two), and pay with a list of coins. This solves the range proof problem, but it multiplies the number of private coins a user sends and receive. A utter private payment of 64-bits value would require the use of 64 private coins (some of them of zero value), and therefore the space consumed still stays around Four KByte vanaf transaction.
One method that can hopefully reduce the cost of private transactions is to create a zk-SNARK based system where many zk-SNARK verifications are aggregated by the end of the month with a mega zk-SNARK proof produced by the last miner, and the individual zk-SNARK proofs are afterward eliminated from the blockchain. Thesis are proofs of proofs . More hope comes from the latest discovery of zk-STARKs . By using zk-STARTKs it is theoretically possible for a miner to create a proof that a certain blockchain state wasgoed correctly reached by compressing the validation of the blockchain of all blocks prior to the one being proven, into a single succinct proof. If zk-STARKs that becomes practically possible, then the size of the blockchain won’t matter anymore, spil the historic gegevens won’t be needed anymore, and wij’ll have a free pass for scalability and privacy.
So even if difficulties te reducing privacy overhead seem a bit depressing, cryptography is evolving quick, and fresh methods will be developed that will hopefully decrease the cost of privacy. For example, a fresh resource optimized proof system that provides brief and practically efficient range proofs called BulletProof wasgoed recently invented.
Te 5-10 years one billion people will be using a blockchain along with an instant, global, and massive off-chain payment network. The one which win will be the one suggesting the cheapest payment cost, even without privacy. That’s how the market has worked for Google or Facebook for individual information gathering. People are willing to give private information for cheaper or free services. But that doesn’t mean all users are compelled to: the same blockchain can support on-chain and an off-chain payment network that offers fully private payments. The off-chain private-oriented payment layer will be multi-hop routed or use Bolt technology, or any other privacy-preserving technology yet to be invented, while the cheap payment layer will be more hub-and-spoke. If wij can reach a billion cryptocurrency users, it just needs some petite percentage of users to be willing to have some privacy for a private system to be bootstrapped, with a large enough anonymity set, and for the network effect to form. I envision that each person with a smartphone will manage some semi-private payment channels and one private payment channel. Both private and semitransparent channels will be tied to a savings account (hopefully private, with the advent of more efficient cryptography).
It is very probable that a decentralized financial system develops ter the following years to serve one billion unbanked users. Bitcoin and the lightning network have the advantage. However, because of the limitation of on-chain volume, Bitcoin alone cannot serve one billion people, not even with an off-chain payment network. I estimate Bitcoin (the blockchain) can serve Ten million active users, at most. RSK, the very first working Bitcoin sidechain, is specially well placed to become the toneelpodium of choice for several reasons. Very first, Bitcoin has the most network effect, security and market cap, so it seems natural that people will choose to use Bitcoin rather other cryptocurrencies. 2nd, RSK’s community core values are aligned with financial inclusion. Third, RSK permits selective privacy. Fourth, RSK is adding innovative technology for scalability. Several unique protocols will soon to be deployed for RSK, such spil LTCP (Lumino Transaction Compression Protocol) and DSTCP (Double-Signed Transaction Compression Protocol). Thesis protocols are targeted to reduce on-chain transaction costs to a ondergrens, freeing more space for payment channel top-ups and settlements.
Regarding the currency, I’m not sure if the billion unbanked people will be transacting te Bitcoin, te foreign or te local fiat-denominated tokens. I suppose the straks. But even when transacting tokens, still people may be using Bitcoin under the rubber hood to pay for transaction fees. And if Bitcoin gets more stable ter the forthcoming years, then maybe Bitcoin becomes the preferred unit of account.
I have a fixation with algorithm complexity. When I wasgoed youthfull I wasgoed an early optimizer, and, I vereiste admit, that didn’t help mij much ter dates. Today I sometimes code a sub-optimized algorithm when there is no need for high spectacle, but it stills bothers mij when I do. When I review code, it also bothers mij to see quadratic complexity, so I pay special attention to loops. That helped my detect the O(N^2) hashing problem, and straks the FindAndDelete() problem te Bitcoin.
Last week I began a research on Segwit scaling. I wasgoed particularly interested te the maximum resources that Segwit-enabled knot requires for verifying a Segwit block. So I went again to review the code, and more specifically, the EvalScript() function. A few seconds later…voila! I came across two more quadratic complexity loops ter Bitcoin Core. By exploiting edge cases for each of thesis two sub-optimal algorithms, I manage to simulate a Segwit block that takes up to Five.6 seconds to verify on a Ubuntu VM running on a single Core i5 processor. The simulation is based on a single thread executing EvalScript(), the Bitcoin script execution function. The tests were not performed processing actual blocks. Thesis results should not make anyone worry, because there are worse problems ter Bitcoin block verification, and because Bitcoin employs several worker threads for verifying scripts ter parallel. For example, a Segwit block can request 80000 signature verifications when all transactions are P2WSH. It is said that Bitcoin Core (ter a modern multi-core machine, using its multi-threading verification capabilities) can verify 8000 ECDSA signatures vanaf 2nd. Therefore a malicious miner can create a Segwit block that requires approximately Ten seconds to be verified. Since the examples introduced ter this postbode consume less than Ten seconds, I don’t consider my findings spil vulnerabilities. However, if the block size is to be enlargened te the future, thesis problems should be solved prior enhancing the block size. The scripts introduced here spil examples do not leave the value stack empty, but the Bitcoin protocol does not require it. Bitcoin only requires the top value to be true to accept the script.
The unsatisfied questioner: OP_IF manhandle
Every time a Bitcoin script executes the OP_IF opcode, a boolean value indicating if the condition wasgoed true, false or the conditional wasgoed skipped (also represented spil false) is shoved into the vfExec stack. Every time an opcode is executed, the number of false values ter the vfExec stack is counted using the following line:
If the count is non-zero, all subsequent instructions except OP_ELSE and OP_ENDIF are skipped. It is clear that the longer the conditional stack is, the more it takes to count the false elements.
The following scriptPub or ScriptSig exploits this problem:
The vfExec vector is packed with 100 elements, and then each factor is scanned 9799 times, totaling more than 979K items scanned. This took Two.Five seconds ter my test VM (for a block packed with thesis scriptSigs).
To re-write this logic with a O(1) algorithm, one simply has to count the number of true conditions ter one variable (trueCount), and the number of false or skipped conditions following all true conditions ter another (ignoreCount). Detecting if code needs to be executed or not requires just testing if ignoreCount is zero.
The treating of OP_IF / OP_NOTIF / OP_ELSE should be like the following pseudo-code:
You may have noticed the strange behavior of Bitcoin’s ELSE statement. Bitcoin permits one to switch inbetween true and false conditions several times. For example, the following script is valid and leaves the value Two on the stack:
The 2nd problem lies te the OP_ROLL opcode. This opcode liquidates a value at a given index from the value stack, and thrusts it on top. Spil the Bitcoin Core stack stores a list of char std::vector by value (not by reference), and because the stack is itself a std::vector (not a linked list), then removing the very first elements requires moving all elements one position te memory. The value stack can store a maximum of 1000 elements. The following script fills the stack and then moves each stack factor 200 times, so the number of moved elements is 200K. This took almost Five.6 seconds te my test VM (for a block packed with thesis scriptSigs).
I attempted other scripts, such spil packing the stack with values of size 520 using DUP3, and then performing rolls, but all of them led to a block that took less time, if the block is to be packed with the scripts.
One solution to this problem is use a linked list gegevens structure instead of a std::vector, to permit O(1) removal of items, but it still requires O(N) for factor lookup. A balanced tree where each internal knot is augmented with the number of children underneath can be used to provide efficient indexed access and efficient factor removal. However, the overhead of such gegevens structure (pointer hopping) may kill its benefits.
What if the attacker has time to prepare outputs?
The results introduced ter this postbode correspond to the maximum times I could get without preparing UTXOs ter previous blocks. By putting the slow-to-validate scripts te scriptPubs instead of scriptSigs, and creating a block with empty scriptSigs, it is possible to execute the slow-to-validate script approximately 25000 times, spil each input will only consume 40 bytes from the block space. For example, it is possible to create a block that takes 33 seconds to verify using ready scriptPubs packed with OP_ROLLs. Preparing 25000 OP_ROLL outputs requires the omschrijving of packing 11 blocks with non-standard outputs created by the attacker. But again, this is not something to worry about, spil with a prep stage an attacker can also create a block that takes Ten minutes to verify by packing 200 OP_CHECKSIGs on each output, and consuming all thesis outputs te a single block. The 20K signature limit does not apply to spent scriptPubs, but only to scriptSigs and created scriptPubs (Bitcoin could soft-fork to also count sigops on consumed scriptPubs). However, preparing blocks packed with OP_CHECKSIGs ter scriptPubs requires almost 250 blocks, because of the 20K signature verification limit restricts the number of outputs that can be included te each block, while preparing a block packed with outputs packed with OP_ROLLS only requires 11 blocks because there is no such limit.
Albeit a lotsbestemming has bot done to optimize block processing, there a few chunks of old code that still require some minor optimizations to prevent future surprises ter the scaling path, whatever path turns out to be followed.
Note: If you find a reason why thesis scripts cannot be included ter a block (spil I toevluchthaven’t attempted it), please send mij a comment. Also if you find prior information on thesis same problems, send mij the linksom.
I will attempt to explain the relation inbetween Segwit and AsicBoost, ter both the covert and overt forms, ter certain detail. I will also attempt to explain why a method wasgoed recently proposed to reduce the interference inbetween covert-AsicBoost and some protocol improvements, by reducing the incentives for covert AsicBoost. The proposal makes covert AsicBoost more expensive, but not unlikely.
“AsicBoost” is two things:
1. A Bitcoin Proof-of-Work ASIC vormgeving that can potentially mine quicker that the standard vormgeving by fixing the last 64 bytes of the very first application of SHA256 (the tail of the block header), and switching the mid-state (the head of the block header), instead of the opposite. This requires fixing and repeatedly using the tail of the header.
Two. Several methods to modify the very first 64 bytes (head) of the block header, while keeping the tail onveranderlijk.
I will concentrate on the methods for (Two).
There are Two methods to modify the head (64 bytes) of the block header.
a) Switching the nVersion field of the block header.
b) Switching 28-bytes of the transaction Merkle root ter the block header.
The method (a) wasgoed called “overt” AsicBoost, because all users can inspect the nVersion field of a block. An nVersion field that has bot flipped would have several random vinnig activated. Since the semantics of the nVersion onverdraagzaam switched according to BIP9, presently a random switch ter this field is interpreted by Bitcoin Core knots spil the signaling of non-existent soft-fork proposals.
The method (b) wasgoed called “covert” AsicBoost, because it’s very difficult to distinguish a block whose Merkle tree has bot “rolled”. Even if detecting a single covert AsicBoost may be difficult, detecting if a certain miner that identifies itself is performing covert AsicBoost may be possible using statistical methods.
Covert AsicBoost requires the miner to find “collisions” of the last Four bytes of the Merkle root field. This is because the last Four bytes of the Merkle root field are stored ter the tail of the block header, not ter the head. This is a pre-processing stage that voorwaarde be done every time a fresh block vereiste be created (height-changed).
There are several algorithmic methods to find collisions: which method to use depends on the number of collisions required, and the size of the colliding groups (e.g. it’s not the same to find two (different) 2-way collisions that one 4-way collision). The group requirements depends on how the SHA256 core wasgoed designed. To find collisions where are several ways to create fresh Merkle root candidates:
b1) Switching the content of a transaction T.
b2) Switching the order of the transactions.
Switching the content of a transaction T (method b1) requires re-computing the transaction id (a dual SHA256 operation) and updating the Merkle tree from the leave of T to the root. At each level, a dual SHA256 of a 128-byte gegevens chunk voorwaarde be performed (this is because two 32-bytes hashes are concatenated, and SHA256 requires reserve space to store the bit length). Let’s assume SHA256 compression and message expansion take the same time Q. Te total, 6Q operations voorwaarde be performed vanaf level. If the block has 2048 transactions, then there will be 11 levels, and 66Q would need to be executed every time the transaction is switched. If the tail of the transaction T is modified (instead of the head), then it may be possible to perform only 4Q to recompute the transaction id. Ter total, 70Q are needed. If the block contains only the coinbase transaction (what is generally called “empty” block), then it is possible to compute the Merkle tree root with only 4Q, because te this case the coinbase transaction id is the Merkle root. Therefore, using the covert AsicBoost by switching the coinbase incentivizes creating blocks with a lower number of transactions. It’s 17 times swifter to find collisions for an empty block than for a block with 2048 transactions, using method b1.
Switching the order of transactions (method b2) can be done te several ways. There are two sub-methods to do it:
c1) The easiest way is the create a set of left subtrees and a set of right subtrees ter memory and them pair them te all possible ways. This can only be accomplished if there are no dependencies inbetween right and left subtrees. If there are n candidates for each subtree, then one can build n^2 roots. The number of transactions te each subtree is almost irrelevant, because each subtree can be created by switching the content of a single transaction. This method requires 6Q to produce each root candidate, so it’s better than switching the coinbase even if the the block is “empty”. This method is best suited for a CPU or GPU.
c2) Another way is to compute the left-subtree and right subtree candidates on-the-fly by interchanging knots at different levels. This method requires much less improvised memory, and it’s more suited for an FPGA o ASIC.
The number of SHA256 operations required to find collisions establishes a motionless delay to commence AsicBoost mining. During the collision finding period the system needs to mine blocks using the traditional nonce-rolling method until the collisions are ready. There is no limitation that the blocks are empty during this period. If the mining hardware only supports AsicBoost but not standard mining, then the miner cannot generate any block during this period. Therefore, there is a high incentive to reduce the cost of collision finding only if the chip is AsicBoost-only capable (not dual-mode). If the chip is dual-mode, a delay of up to 1 minute may be tolerated (e.g. a 2% revenue loss) so the efficiency of the method to generate collisions is of less importance.
If voorwaarde be noted that even if transaction reordering is the fastest covert method, the overt method is still swifter: it doesn’t require to find any collision at all.
When a segwit (BIP141) enabled blockchain receives a block that contains at least a single segwit transaction, it only accepts the block if the block has a wtxid tree Merkle root, which is stored ter the coinbase transaction. This tree contains the transactions wtxids. A wtxid comprises both the transaction gegevens and its witness gegevens. Therefore, if the order of transactions switches (but not the wtxid commitments), then this mismatch makes the block invalid.
This is the reason why Segwit makes the use of transaction reordering more difficult. Transaction reordering is the most efficient covert method for AsicBoost, and the one more suited for FPGA implementation. It coerces the covert-AsicBoost miner to mine only blocks not containing Segwit transactions.
The recently proposed method blocks a specific form of covert Segwit using transaction reordering (c2). The other less-efficient covert methods (b1) te a Segwit block still exists.
Te 2013 I found a Bitcoin transaction that takes Three minutes to verify (CVE-2013-2292) related to O(N^2) hashing te signatures. Since then, the O(N^2) argument has popped up ter many contexts, mainly te discussions about a block size increase. Now the problem is partially solved by Segwit. During January 2016 I attempted to ritme the record, for joy. To do it, I determined I needed to dive into the least explored parts of the source code to find a flaw going undetected for 7 years te archaic code. About 28% of the original Bitcoin v0.1 source code remains te Bitcoin Core v0.12 (albeit that 28% now represents less than 1.9% of the total source lines), so the possibility existed. But after 7 years of code review, I wasgoed looking for a pearl. I wasgoed looking for the last Satoshi bug.
It is known that deprecated and backwards-compatibility functions are vulnerability-prone parts of any source code. So I focused on FindAndDelete() function: the oldest, most cracked and less used part of the Bitcoin overeenstemming code. Peter Todd wrote an excellent review of the nasty consequences of FindAndDelete() here. Fortunately, after not much time, I found my pearl: O(N^2) gegevens movement. The next step wasgoed to find a way to build a Bitcoin transaction that would exploit it. If you let mij give you an advise, this is it: never trust your own vulnerability discoveries and go reporting before you have indeed tested them. That is a lesson I learned the hard way: it’s very effortless to be blinded by excitement. So I began coding an exploit with my friend Oscar Guindzberg, and after several attempts wij managed to convert it into an attack. Te the worst case (te my pc) the transaction took Five.Five hours to verify, but it required a loterijlot of prep.
Before going into the details, I want to point out that OP_CODESEPARATOR, FindAndDelete() and the O(N^2) hashing are utterly annoying bugs ter the original vormgeving that should be stationary by either by Segwit plus a soft-fork, or via a hard-fork, someday. Removing archaic code not only prevents vulnerabilities but also permits much secure re-implementation of the overeenstemming code (albeit re-implementing overeenstemming code is still a subject of debate).
This vulnerability had an effortless fix so I reported it and it wasgoed quickly motionless ter Bitcoin Core v0.12.1 (on 14 Apr). The fix didn’t require a fork of any kleuter, just rewriting the FindAndDelete() function to be O(N) and not O(N^2).
The way to exploit O(N^2) memory movement te FindAndDelete() is by building a transaction having the following scriptPub:
The OP_IF / OP_ENDIF prevents stack overflows by skipping the execution of the OP_0 (which thrust an empty vector into the stack). The scriptSig voorwaarde be just an empty script (or a NOP). Each of the initial 200 OP_0 thrusts represents a signature consumed by for each CHECKSIG. Since the signature is zero, and CHECKSIG attempts to eliminate the signature from the scriptPub with FindAndDelete(), all zeros are liquidated. The tail of the script contains an extra 9601 zeros, so each OP_CHECKSIG performs 9801^2/Two byte moves. The total number of bytes moved is 200*9801^2/Two (=9.6 GB).
The validation of a transaction consuming this prevout takes 1 sec on my virtual machine.
Since an input that redeems a prevout containing only an empty script consumes not more than 50 bytes, the number of redeems that can be performed te a block is about 20K (ter a single transaction), and therefore the time it would take to process such block would be 20K*1 sec = Five.Five hours. However, it still requires 200 blocks total of thesis garbage outputs spil prep stage. So the worst case attack is more theoretical than practical.
To test it ter a practical attack (a standalone block attack) I built a block containing a very first transaction having 95 of the zuigeling of scriptSigs described, plus another transaction consuming those 95 prevouts and this is the resource usage pattern when accepted by bitcoind:
It takes 92 seconds to verify the 96 txins, so my guess that a transaction that takes Five.Five hours to verify can be built wasgoed juist.
It is possible to use a standard p2sh transaction spil the attack vector, but it seems to be a transaction that takes no more than 500 msec to be verified.
The Elementary Solution
Te this commit Bitcoin developers re-implemented FindAndDelete() to build a fresh script by copying the parts that are not liquidated, instead of removing in-place. I don’t know if any other re-implementation of the overeenstemming code is more vulnerable to this type of resource consumption (I checked BitcoinJ and python bitcoinlib and were not vulnerable).
Other cryptocurrencies based on old versions of Bitcoin may still be vulnerable, but I can’t check every Bitcoin clone to see which are. I’ve lost the count of how many alt-coins there are.
Functions implementing old violated features that nobody use anymore are good places where vulnerabilities can hide, so it’s better to eliminate code that implements unused functionality.
Bugs are everywhere, keep an eye open and distrust source code.
This report wasgoed made public since inbetween 80% and 98% of the network presently has their knots patched. Special thanks to Oscar Guindzberg who helped mij by coding the exploit using bitcoinj. The source code is available if anyone wants to re-test.