Structure of shards

(NOTE: for encryption, check this and other similar guides, read libsodium docs, check what SSB and Dat are doing)

Metadata, returned by shard servers but not part of the main shard (FORMAT TO BE SPECIFIED):

encryption suite info		# e.g. aes-ctr-sha256
{shardHash}^authorPriv		# shard signature
{stream-key||256bit-random}^targetPub1		# list of allowed recipients
{stream-key||256bit-random}^targetPub2		# the random is necessary so that one of the targets
...						# cannot figure out who else is a recipient by encrypting
{stream-key||256bit-random}^targetPubN		# the key with other people's keys

The data shard itself (this is what gets hashed into the shard ID):

{stream-key}^authorPub|{contents}^stream-key

Contents of the links (structure either as in magenc, or to be defined):

<shard-hash>;<optional-modifier-signature>;optional-author-info

stream-key:

• stream, because we should be able to request n bytes (for the manifest), check if that's enough, and then request more
• this way partial decryption possible
• no need to store the length of the manifest (and encrypt it separately)
• the key should be included in the hash to make constructing colliding shards more difficult
• this means, the hash might have to be calculated by the client only as we don't want the server to know about any keys

optional-modifier-signature:

• to allow replacing shards with new ones, a modifier signature needs to be used
• it's the hash encrypted with the modifier group's private key
• thus all new shards that are advertised as replacements / updates for this one can prove it by proving the ownership of the modifier private key
• for private shards, this is the authors private key and the same author can overwrite their shards
• for co-operative shards, this is a group key, and users that know the key can provide updates or changes to this shard
• this doesn't have to be the same key as is used for encrypting the stream-encryption-key though
  • this allows, read-only, read-write, as well as write-only access capabilities

contents:

• free-formed s-expressions
• should at least contain "type" with e.g. the following
  • txt
  • exe (for lisp code as s-expr)
  • list (of additional shards)
  • tree (hierarchical list)
  • view-template (think HTML templates like in Clotrine, but with hashes for sub-components)
  • image (with additional "encoding" or something for "jpg"/"png", ...)
  • etc. etc. for all kinds of well-known stuff
  • optionally the author, and if present, signature should also be included
  • but the signature needs to be outside of the content, as including it in the hash would be semi-impossible