When it comes to data storage, we have only a few options. We can store small files using USB drives. Centralized cloud storage is another option, where a third-party entity owns and controls the storage. However, opting for centralized cloud storage means that the said entity operates the cloud, resulting in the loss of freedom and security factors. Decentralized storage networks provide a solution. In this case, a network spread among multiple users across the globe stores the data. Users get incentives to join and operate the network to keep it decentralized and ensure data accessibility. A group of people hosts the servers used, rather than a single authoritative body.
Anyone can join a decentralized network based on a blockchain. Smart contracts ensure the integrity and authenticity of users, and users get incentives with native tokens to run the network. These tokens offer a range of benefits, including governance rights.
Why should you choose a decentralized storage network over centralized storage?
Firstly, decentralized storage networks provide stronger security as blockchain technology resists data breaches more effectively. They also reduce the chances of attacks like DDoS. Moreover, decentralized storage grants complete ownership of personal data. Finally, decentralized storage networks are less prone to censorship and surveillance compared to centralized storage.
Storj
Storj is an open-source decentralized network that stores data. It aims to solve high durability with minimum expansion. It does not follow the typical approach of replicating files in multiple nodes. Instead, it fragments files into multiple nodes, maintaining high durability (probability of surviving outage) with low expansion (requiring additional storage). The network enables anyone with spare disk space and internet to join as storage nodes and applications. Mechanisms like Uplink clients and developer tools facilitate data upload and download. In the absence of a consensus mechanism, trusted satellite nodes manage metadata, node information, data repair, and payouts.
Storj uses erasure code to ensure data persistence without increasing network traffic. It breaks data into fragments, encodes them, and saves them in multiple nodes. This approach requires much less expansion, i.e., additional storage, compared to replication for the same durability or probability of data recovery in case of an outage. The math behind it is available here. There is an audit that detects system performance. Failed audits result in recovering the file from remaining nodes and reconstructing and saving missing pieces in other nodes.
Modifying metadata entries is necessary for any changes. Currently, there is no mechanism relying on satellite operators for consensus, but plans are in place to build one in the future. The cost of storage is $0.004 per GB per month. Storj offers unique features like high durability with fragmentation (erasure code) and compatibility with AWS S3. It also ensures privacy through encryption.
IPFS
Juan Bennet is the founder of InterPlanetary File System (IPFS). IPFS serves as a distributed file system for storing and sharing data. It functions similarly to torrents, but instead of hosting files in a single location, anyone with a copy can host them.
Components of IPFS
The concept of IPFS includes several components. These components are:
- Distributed Hash Table: It is used to store and retrieve data between network nodes.
- BitSwap: It is a peer-to-peer file-sharing protocol that coordinates data transmission between untrusted swarms.
- Merkle DAG: It is used to track changes to files on the network in a distributed-friendly way.
IPFS runs a process called “Garbage collection” to optimize memory usage. This process involves deleting objects that are not frequently used. Users can add a local “Pin” to ensure that the data is saved locally. For larger data sizes and enhanced safety, users can use “Pinning services” to store the data in multiple nodes. However, this has a fee. IPFS does not guarantee long-term storage. However, it can be integrated with multiple tools like Filecoin to ensure permanence.
The users can use content addressing with Content Identifiers (CID). Any change in content will generate a new content identifier, ensuring immutability. For web 3.0 use cases such as NFT, integrity is ensured by linking on-chain “metadata.” This metadata includes the IPFS URL created using the CID of the content stored on IPFS. The default layer of IPFS does not have a consensus mechanism. However, there are layers on IPFS, such as IPFS cluster and Filecoin, that have their own consensus mechanisms.
IPFS cluster offers CRDT (Conflict-free Replicated Data Type) and Raft consensus mechanisms.
CRDT (Conflict-free Replicated Data Type)
CRDT is for a liquid peer network where peers frequently enter and exit, and not all peers have modification permissions. It also supports batching operations for pinning/unpinning.
Raft Consensus Mechanism
On the other hand, when the peer network is fully trusted, users utilize the Raft consensus mechanism, which is older but proven. Filecoin, on the other hand, implements its own consensus mechanism known as “Proof of storage.” While users can freely use IPFS itself, uploading files on IPFS does not guarantee permanence since the data solely resides on the user’s computer.
To ensure data replication in multiple places, users need to use pinning services and permanence tools. However, these services and tools come with a fee. Some unique features of IPFS include content addressing using Content Identifiers (CID) for immutability, optimized linking and addressing of content using the Merkle Directed Acyclic Graphing technique, and content discovery using Distributed Hash Tables (DHT).
BitTorrent File System
BitTorrent is a decentralized file-sharing protocol that was created by developer Brad Cohen in 2001. In 2018, the Tron Foundation acquired the BitTorrent Foundation and subsequently launched the BTT token in 2019. Instead of relying on a single server for downloading or uploading files, users join a network of computers running software that facilitates the exchange of files and data among participants. This network empowers users to share content directly with one another. Additionally, the BTT token powers various decentralized applications, including DLive, BitTorrent Speed, BitTorrent File System, and others.
When it comes to persistence, BTFS leverages multiple smart contracts that govern interactions between file owners (renters) and storage providers (hosts). These contracts ensure the proper storage of files, and hosts are required to provide periodic proof of storage. Failure to comply with these requirements can result in fines. It’s important to note that BitTorrent servers do not store data. Once users download the BitTorrent client, they retain control over their data for a lifetime. Even if the program goes offline, users of Sync, the BitTorrent client, can continue to utilize the software.
In terms of tokenomics, the total and maximum token supply of BTT tokens is 990,000,000,000. At present, the circulating supply stands at 923,767.70B BTT. It uses Content Identifiers (CID) which uniquely identify the content and guarantee its integrity. To establish consensus within the network, hosts stake BitTorrent Tokens (BTT), and their staked tokens determine which files are assigned to them for storage. Considering the cost of storage, it amounts to approximately $.00675 USD per gigabyte per month. Moreover, BTFS incorporates several unique features. These include token economics with BitTorrent Token (BTT) integration, file encryption, and the ability for hosts to remove files from their storage.
Arweave
Arweave is a decentralized storage network originally named Archain. The Arweave protocol operates on two layers. The first is the Blockweave, where one can store datain a graph of blocks. Each block links to two earlier blocks in Arweave, forming a structure known as a “blockweave.” The second layer is the Permaweb, which ensures that everything published on it remains available forever. The Permaweb provides low-cost, zero maintenance, and permanent hosting for web apps and web pages.
Arweave utilizes token economics with AR tokens for persistence. It establishes multiple smart contracts between renters (file owners) and hosts (storage providers) to ensure the storage of files. Hosts provide periodic proof of storage, and failure to do so can result in fines. The blockweave data structure incentivizes miners to store blocks, ensuring immutability. The blockweave employs a probabilistic approach to ensure the appropriate number of replications, with higher incentives for storing rare data.
Proof of Access Mechanism
Arweave employs Proof of Access as a consensus mechanism. Miners validate a proof provided by a node with access to a randomly selected “recall” block. This mechanism enhances Proof of Work and incentivizes miners to store blocks and win rewards. The one-time lifetime storage cost on Arweave is approximately $8 per gigabyte, with no monthly subscription.
Arweave offers the unique feature of paying only once for permanent storage. Additionally, the blockweave structure links each block to two prior blocks: the previous block and a random block from the blockchain’s previous history, known as a recall block. Miners incentivize storing rare blocks by following a probabilistic and incentive-driven approach to replicate data in the network.
Moreover, memoization of state allows new nodes to download only the current blocks from trusted peers or use block data structures such as the Block Hash List and Wallet List. Miners synchronize and update these lists, enabling each node to prioritize storage based on preferences or available resources while guaranteeing storage and replication within the network.
FileCoin
Filecoin allows anyone to rent storage space. Instead of entrusting your documents to one company, you can divide and store them on computers all over the world. It is an incentive layer on top of IPFS that incentivizes users to rent out their storage space by paying them in FIL tokens.
Proof of Replication & Proof of Space Time
In terms of data verification, Filecoin utilizes ‘Proof of replication’ to verify that a node has stored data. Additionally, it employs ‘Proof of space time’ to ensure that data remains saved over a range of time. Moreover, Filecoin leverages FIL tokens and multiple associated mechanisms to incentivize long-term storage.
The network scales up block rewards until a certain baseline, to prevent miners from exiting the network after cashing out early rewards. Miners guarantee storage for a specific period by pledging collateral funds. If a miner fails to provide storage or goes offline during the guaranteed duration, the network deducts a portion of the collateral and block rewards. Filecoin offers the option to store files with multiple independent miners and verifies the existence of independent copies, making it a fault-tolerant solution. Filecoin uses Content Identifiers (CIDs) to address content, ensuring immutability. Notably, any change in content generates a new CID.
Useful Work Protocol
Filecoin employs a ‘Useful work’ consensus protocol. Under this protocol, miners allocate storage in proportion to their assigned storage, determining the probability of being allocated storage. Furthermore, some storage providers, like web3.storage, offer free storage on the Filecoin network. This is due to the fact that miners’ rewards are higher when they already store data. The average cost for storage on Filecoin is $0.0000009 USD/GB/Month.