What is IPFS?

Prabhaker Mateti

Wright State University

cecs.wright.edu/~pmateti

Table of Contents

1 Detailed Table of Contents

1.1 2. What does "IPFS" Offer? TOC

  1. What is "IPFS"?
  2. Object Storage
  3. Addressing Stored Values: Key-Value Stores
  4. Location Based Addressing
  5. Content-Hash Addressed
  6. Immutable
  7. Permanency
  8. Permissions
  9. Ownership
  10. Decentralized + Distributed
  11. Network Accessible Uniformly
  12. Security

1.2 3. What is IPFS? TOC

  1. Technologies Used/ Related
  2. Nodes
  3. IPFS Layers (Subsystems)
  4. IPLD InterPlanetary Linked Data
  5. LIBP2P
  6. IPNS InterPlanetary Naming System
  7. IPFS Applications

1.3 4. IPFS FAQs TOC

  1. IPFS Implementation Done?
  2. IPFS Integration with BitCoin, et al.
  3. Content Availability
  4. Content Permanency
  5. Content Secrecy
  6. Delete Content
  7. Does IPFS Guarantee Anonymity?
  8. Hosting a Website
  9. IPFS Abuse
  10. IPFS Hash Collision
  11. Public v Private IPFS

1.4 5. Related Projects TOC

  1. FileCoin
  2. Orbit
  3. DTube
  4. ZeroNet
  5. Akasha
  6. Competing Projects
  7. Distributed Web Applications (DApps)

1.5 6. IPFS Why Should We Care?

  1. Future Web? Web 3.0?
  2. Un-censorable.
  3. "Trust, but Verify." No 3rd parties. Content can be trusted.
  4. Digital ledgers built-in.
  5. More at the end of these Lectures.

1.6 On to the Content of this Lecture

  1. The next few descriptions are loaded with "terminology".
  2. Often, Circular.
  3. These are going to leave you with a number of questions. Wait for FAQ below.

2 What does "IPFS" Offer?

  1. (Linux/ …) OS File System.
  2. Web content: As it is now 2018, Web 2.0, Web 3.0 soon.
  3. Globally + Uniformly Network accessible.
  4. A platform for Distributed Apps (DApps).

2.1 What is "IPFS"?

  1. The current section is a collection of ideas.
  2. Together they can be the basis for a future design of web.
  3. The next section is a description of IPFS, the specific "product".
  4. IPFS, an acronym for "InterPlanetary File System".

2.2 Object Storage

  1. Digital Objects are stored in "nodes": distributed across the globe, decentralized.
  2. The "FS" in the name does not stand, strictly, for files.
  3. A "file" is any digital object, small or large.
  4. Objects are stored in a MerkleDAG.

2.3 Addressing Stored Values

2.3.1 Arrays and Indices

  1. a[x]
  2. array name a,
  3. index integer x,
  4. x >= 0 mostly.
  5. address of a[x+1] == address of a[x] + size-of(a[x])

2.3.2 Associative Arrays

  1. a collection of pairs (x, a[x]) of indices x and values a[x]
  2. unordered
  3. no duplicated indices
  4. cache memory
  5. Often drawn as an array, but no address relationships among elements

2.3.3 Key-Value Stores

  1. Generalize the indices in assoc arrays to arbitrary types called keys.
  2. No address relationships among elements.
  3. Given a key k, retrieve value val(k)
  4. https://redis.io/ | https://memcached.org/

2.4 Location Based Addressing

  1. Web 1.0/ 2.0 URLs have the following syntax: scheme: //[userName [:password@]] serverMachineName [:port]/ [path] [/resource] [?parm1=parma &parm2=parmb]
  2. A simple example of the above is http://www.cs.wright.edu/~pmateti/ where the scheme was chosen to be http, the port defaults to 80, and the path given is ~pmateti/.

2.5 Content-Hash Addressed

  1. Apply a crypto-hash (e.g., sha256) to the content, resulting in, e.g., QmZpfhN5rucQC4kx7Gu5udS8FXxsxMddiDCvG8WFjG8SMv
  2. The above is in multihash self-describing format and Base58 encoding. The letters Qm correspond to SHA-256 and 32 byte length.
  3. IPFS makes sure the hash is unique.
  4. An IPFS address hash should be stored in bytes. The Qm... should be stored as a string.
  5. IPFS link /ipfs/QmW2WQi7j6c7UgJTarActp7tDNikE4B2qXtFCfLPdsgaTQ/cat.jpg
  6. http://localhost:8080/ipfs/QmR96iiRBEroYAe955my8cdHkK4kAu6SApGC2eS5kSzHwt/

2.5.1 MultiHash

  1. struct Multihash 
    struct Multihash {
  bytes32 hash
  uint8 hash_function
  uint8 size
}
  2. ipfs add -w example.txt 
    $ ipfs add -w example.txt
added QmbFMke1KXqnYyBBWxB74N4c5SBnJMVAiMNRcGu6x1AwQH example.txt
added QmVFDXxAEC5iQ9ptb36dxzpNsQjVatvxhG44wK7PpRzsDE

2.5.2 Retrievability

  1. To retrieve content, you must have its hash.
  2. Ex: https://ipfs.io/ipfs/QmR7GSQM93Cx5eAg6a6yRzNde1FQv7uL6X1o4k7zrJa3LX/…
  3. User friendly? Work-in-progress. See IPNS.

2.6 Immutable

  1. The content (of a stored object) is unchangeable.
  2. The content is versioned, with a link to its previous version.
  3. Recall digital ledgers.
  4. Deletion? See FAQ.

2.7 Permanency

  1. IPFS is a permanent storage system for digital objects. Lives forever.
  2. An object stored in IPFS is retrievable at any time, 24x7, from a node.
  3. Motivated by incentivization.

2.8 Permissions

  1. As long as you have the hash of an object, you can retrieve it from IPFS.
  2. Can we have at least Linux-style read-write-etc permissions? WIP.
  3. FAQ: Public v Private IPFS.

2.9 Ownership

  1. What is ownership? Define. Related to immutability.
  2. No single entity, human or org, owns the system, or any part of it.
  3. No "destructive" operations can be performed.
  4. FAQ: Who uploaded what?

2.10 Decentralized + Distributed

  1. "It's nowhere because it's everywhere!"
  2. Sanskrit: Sarvantaryami (in Telugu script సర్వాంతర్యామి)

2.10.1 Client-Server v P2P

Client-Server v P2P

Figure 1: Client-Server v P2P

2.10.2 Centralized Decentralized Distributed

Centralized Decentralized Distributed

Figure 2: Centralized Decentralized Distributed

2.10.3 Distributed

  1. Spatially Distributed
  2. Possibly replicated
  3. We use duplication versus replication with different semantics.
  4. Sharded content?

2.10.4 Decentralized

  1. Harder to define.
  2. Centralized: One node exists that controls all content.
  3. Multiple centers.
  4. Decentralized: Single or multiple centers do not exist.

2.10.5 Decentralized implies Distributed

2.10.6 Distributed does-not-imply Decentralized

2.11 Network Accessible Uniformly

  1. All IPFS nodes access the objects using the same addresses.
  2. No client is denied access.
  3. Performance can vary.

2.12 Security

  1. "Security" is an ill defined term, in general. So, let us focus on the following aspects.
  2. Unauthorized content modification: Not possible.
  3. Read-Access control Built-in: Non existent.
  4. Read-Access control DIY: Encrypt the object and share the (private) key.
  5. Learn to securely share on the IPFS.

3 What is IPFS?

  1. A peer-to-peer hypermedia protocol: ipfs:/.../....
  2. A (networked) storage system that can be mount -ed.
  3. Funding: $52 million from Union Square Ventures (??) and others.
  4. IPFS is FOSS.
  5. This section is a sketchy design description.

3.1 Technologies Used/ Related

  1. DHT Distributed Hash Tables - Kademlia
  2. Block Exchanges - BitTorrent
  3. Version Control Systems, merkledag - Git
  4. Self-certifying File System SFS
  5. FileCoin

3.2 Nodes

  1. A node is a computer system running the "IPFS daemon": ipfs init
  2. A node stores IPFS objects.
  3. Obviously, not all of them.
  4. No assumptions are made about the life-cycle of a node.
  5. A node is a running process that can find, publish, and replicate merkledag objects.

3.2.1 Nodes #2

  1. Access requests originate from a node.
  2. Retrieved content is cached.
  3. Idle cached content is garbage collected.
  4. Nodes cannot be forced to fetch anything or to pin anything.
  5. A node uses the DHT, IPLD, IPNS – see below.

3.2.2 Nodes #3

  1. There are already many (thousands?) nodes.
  2. Content becomes more readily available as it gets more popular.
  3. Often content "cannot" be reached.

3.3 IPFS Layers (Subsystems)

IPFS Layers

Figure 3: IPFS Layers (the left half names pre-existing systems)

3.3.1 IPFS Subsystems

Figure 4: IPFS stack, divided

3.4 IPLD InterPlanetary Linked Data

Figure 5: InterPlanetary Linked Data

3.4.1 IPLD InterPlanetary Linked Data

  1. IPLD InterPlanetary Linked Data https://github.com/ipld
  2. "IPLD is the data model of the content-addressable web. It allows us to treat all hash-linked data structures as subsets of a unified information space, unifying all data models that link data with hashes as instances of IPLD."
  3. "IPLD is a single namespace for all hash-inspired protocols. Through IPLD, links can be traversed across protocols, allowing you explore data regardless of the underlying protocol."

3.5 LIBP2P

IPFS Layers

Figure 6: libp2p included protocols

3.5.1 LIBP2P #2

IPFS Layers

Figure 7: libp2p routing

3.6 DHT Distributed Hash Tables

  1. DHT: Given the Key, retrieve Value.
  2. Similar to Python dict or Perl hash. Not (necessarily) crypto-hash.
  3. The DHT may be distributed over multiple nodes.
  4. Multiple designs/ implementations exist: e.g., Kademlia.
  5. IPFS key is crypto-hash of object content.

3.7 IPFS Networking

  1. "IPFS nodes communicate regularly with hundreds of other nodes in the network."

3.7.1 IPFS Routing

  1. IPFS nodes require a routing system that can find (a) other peers’ network addresses and (b) peers who can serve particular objects.
  2. IPFS achieves this using a DHT.

3.7.2 DHT Interface

type IPFSRouting interface {
  FindPeer(node NodeId) // gets a particular peer’s network address
  SetValue(key []bytes, value []bytes) // stores a small metadata value in DHT
  GetValue(key []bytes) // retrieves small metadata value from DHT
  ProvideValue(key Multihash) // announces this node can serve a large value
  FindValuePeers(key Multihash, min int) // gets a number of peers serving a large value
}

3.8 IPFS Exchange

  1. In IPFS, data distribution happens by exchanging blocks with peers using BitSwap.
  2. BitSwap nodes provide direct values to each other in the form of blocks.
  3. BitSwap nodes keep ledgers accounting the transfers with other nodes

3.9 IPNS InterPlanetary Naming System

  1. The Peer ID is the hash of the public RSA key of the node. IPNS maps peer IDs to content hashes.
  2. The mapping can be modified. 
ipfs name publish QmNUhKfcGJyQJnZu3AKn8NoiomDwDCRBicgqPt1YRqJBCz
Published to QmYmmfn68vkcFDeZz1NTZyEXTixjjUnUS6UaPdMSsUBWxs:
 /ipfs/QmNUhKfcGJyQJnZu3AKn8NoiomDwDCRBicgqPt1YRqJBCz
  1. In the above, QmYmmfn68vkcFDeZz1NTZyEXTixjjUnUS6UaPdMSsUBWxs was the peer ID.
  2. IPFS stores the private RSA key in the configuration. No passphrase.

3.9.1 Readability

human-readable.png

3.9.2 Readability #2

ipld-foo-bar-baz.png

3.9.3 IPFS Naming

  1. Bind IPNS to a domain using DNS TXT records

3.9.4 IPFS MerkleDAG

A Merkle Bin Tree

Figure 10: A Merkle Bin Tree of Four Blocks

3.9.5 Details of MerkleDAG

  1. Merkle trees in general are n-ary.
  2. A forest is a collection of trees.
  3. Merkle Trees are already in use by Git, BitCoin, Ethereum, …

3.9.6 IPFS Digital Objects

  1. block: a variable-size chunk of data. An addressable unit of data.
  2. list: a collection of blocks or other lists.
  3. tree: a collection of blocks, lists, or other trees.
  4. commit: a snapshot in the version history of a tree.

3.10 IPFS Applications

web20to30.jpg

3.10.1 PubSub

  1. Publishers-Subscribers is an object oriented architectural design pattern.

3.10.2 Observer Pattern

##

Observer Pattern

Figure 12: Observer Pattern

3.10.3 PubSub (Publisher Subscriber) Pattern

Figure 13: PubSub Pattern

4 IPFS FAQs

4.1 IPFS Implementation Done?

  1. Is the design and implementation of IPFS done?
    1. Go https://github.com/ipfs/go-ipfs exists,
    2. JavaScript https://github.com/ipfs/js-ipfs exists,
    3. Python https://github.com/ipfs/py-ipfs is in progress.
  2. Alpha. Targeted at developers.

4.2 IPFS Integration with BitCoin, et al.

  1. Can IPFS integrate with BitCoin, Git, …? Yes.
  2. See IPLD.

4.3 Content Availability

  1. Will something added to IPFS remain available to others if I turn off my computer?
  2. Turning off is not the crucial point.
  3. Unreachable because un-cached in reachable nodes.
  4. https://discuss.ipfs.io/t/397

4.4 Content Permanency

  1. No guarantees of replication; need people to request content data.
  2. Incentivization in progress.
  3. https://discuss.ipfs.io/t/storing-a-few-tb-of-data/476

4.5 Content Deletion

  1. Can I delete (my) content from the network? No.
  2. See above.
  3. https://discuss.ipfs.io/t/301

4.6 Content Secrecy

  1. Objects can be made illegible using public key cryptography. Those holding the private key can read. Others can only tell how big they are but they cannot know more.
  2. Incentivize to keep such content available.

4.7 Content Privacy

  1. Private swarms, WIP.
  2. Separate name spaces, WIP.
  3. Public v Private IPFS, WIP.

4.8 Does IPFS Guarantee Anonymity?

  1. IPFS makes no claims about anonymity.
  2. We can discover the IP address of a peer hosting any given block.
  3. https://discuss.ipfs.io/t/387

4.9 Hosting a Website

  1. The main site https://ipfs.io/ is on IPFS.
  2. Example: http://www.dispersion.io/
  3. Example: "The Ramblings of a Random Programmer" https : //ipfs.io/ipfs/ QmdPtC3T7Kcu9iJg6hYzLBWR5XCDcYMY7HV685E3kH3EcS/ 2015/09/15/ hosting-a-website-on-ipfs/ [Blanks inserted for readability]
  4. Google does not index IPFS (yet).
  5. See https://discuss.ipfs.io/t/is-ipfs-fit-for-personal-blog-pages/331

4.10 IPFS Abuse

  1. Forgoing reputation and indiscriminately servicing leaches https://discuss.ipfs.io/t/444
  2. Is it possible to limit the size of a local storage? https://discuss.ipfs.io/t/364
  3. Can I control the amount of storage/ bandwidth dedicated to IPFS? https://discuss.ipfs.io/t/395

4.11 IPFS Hash Collision

  1. What to do in case of hash collision? Multiple objects crypto-hashing to the same.
  2. https://discuss.ipfs.io/t/482

4.12 Who Uploaded What?

  1. We can discover the IP address of a peer hosting any given block.

4.13 Misc

  1. {IPNS works badly now, but this is because of the implementation.}
  2. IPFS uses a lot of bandwidth now. WIP

5 Related Projects

  1. Sister Projects
  2. Competing Projects

5.1 FileCoin

  1. FileCoin is a sister project, by Protocol Labs. FOSS.
  2. http://filecoin.io/ as an incentive to replicate data. (Get paid "payed" for storing files.)
  3. {You offer to pay for storage, and the most efficient, capable, and cheap hard drives win the bid and they host your files.}

5.2 FileCoin #2

  1. Filecoin is a protocol that pays nodes to keep a copy of your stuff without having to trust them.
  2. When you use Google Drive, you are trusting google.
  3. Google Drive has a user interface and editors for your spreadsheets etc, sharing functionality and much more.
  4. Filecoin is just a system to have others store files for you and that's it.

5.3 Orbit

  1. Orbit P2P Chat https://orbit.chat/
IPFS P2P Chat

Figure 14: Orbit P2P Chat on IPFS

5.4 DTube

  1. https://d.tube/ A decentralized video platform using STEEM and IPFS
  2. Image "hosting": https://github.com/ipfspics/ipfspics-server | http://ipfs.pics/ [Broken link]

5.5 ZeroNet

  1. https://zeronet.io/ "Open, free and uncensorable websites, using Bitcoin cryptography and BitTorrent network"

5.6 Akasha

  1. AKASHA (a Next-Generation Social Media Network) https://akasha.world/ TBD Sanskrit ākāśa आकाश
  2. orbit-db (Key/Value database; Peer-to-Peer Databases for the Decentralized Web)

5.7 Competing Projects

5.7.1 LBRY

  1. Content Freedom https://lbry.io/

5.7.2 DAT

  1. http://datproject.org/ A distributed data community, dat

5.7.3 PEERGOS

  1. https://peergos.org/ An end-to-end encrypted, peer-to-peer file storage, sharing and communication network. It is an open source, secure, self hostable file storage and sharing platform. All your files are encrypted locally and your private keys never leave your machine. FOSS.

5.8 Distributed Web Applications (DApps)

  1. IPFS is a platform for dApps.
  2. Orbit P2P Chat https://orbit.chat/
  3. https://peerpad.net/ PeerPad is a realtime P2P collaborative editing tool, powered by IPFS and CRDTs (conflict-free replicated data type)

6 IPFS Why Should We Care?

  1. Un-censorable.
  2. Trust, but Verify. No 3rd parties. Content can be trusted.
  3. Digital ledgers built-in.
  4. More at the end of these Lectures.

7 ACK

  1. Juan Bennet, @article{benet2014ipfs, title={IPFS-Content Addressed, Versioned, P2P File System (Draft 3)}, author={Benet, Juan}, journal={arXiv preprint arXiv:1407.3561}, year={2014}, pages = {11},} https://ipfs.io/ipfs/QmR7GSQM93Cx5eAg6a6yRzNde1FQv7uL6X1o4k7zrJa3LX/ipfs.draft3.pdf
  2. @Article{Filecoin2017, author = {Protocol-Labs}, title = {Filecoin: A Decentralized Storage Network}, journal = {White Paper}, year = {2017}, pages = {36}, month = {July}, }
  3. Mark Pors, https://decentralized.blog/ Catching the Blockchain Train, Understanding-the- IPFS white-paper 2017.
  4. All figures are borrowed from the web.

8 End

Copyright © 2018 pmateti@wright.edu • www.wright.edu/~pmateti 2018-08-19