Unstructured volatile notes (partly stream-of-thoughts and perhaps already obsolete, see later posts for updates and clarifications)

random thoughts, to be refined

• if one should be able to share e.g. a post between different social media services, all non-universal metadata needs to be stored separately
  • we want to avoid data being tied to a service due to its format
  • how is such a post formatted? Markdown? Which flavour? Is styling possible? All that should be compatible with different services.
  • that's perhaps more a requirement for the services though; add some metadata about the format and mandate support for it?
• how to keep track of all shards in general? Tagging helps but might not be enough. This should be a database for multiple services, private notes, random files etc. How to keep that all organized?
  • each shard should anyway contain all the necessary information about its origins to allow reconstruction after loss of volatile metadata
    • actually no, if you are planning to share those shards without wanting to leak the metadata
  • of course if each shard has tags, any tags added later would have to be mutable metadata and thus not part of the shard
    • ... but as long as this metadata is also a shard with a pointer back to the original, we should be okay
    • wait, should each individual main shard (note, post, whatever) be its own Dat feed...?
• the private master key should be held in memory using something like memfd_secret system call on Linux: that memory is not accessible for any other process or even the kernel, which makes it very hard to get the key even if the host is compromized.

local shard cache

• one file per shard
• file hierarchy with at least two layers
  • first char of sha, then second char?
    • dynamic beyond that? or fixed files after two layers?
    • this way we have ~ 40 dirs per layer
      • 1600 possible dirs
      • with 10000 shards (just random example), we'd have only a few files per dir
• needs metadata
  • directly accessed or a dependency shard?
  • to enable garbage collection if primary shard deleted and dependencies no longer needed
  • perhaps also last access time for stuff not marked as permanent?

Code is data

• system where everything, including programs, are handled by such hash shards
• shards can contain executables in the form of LISP programs
• DSL for displaying data, a bit like HTML but as s-expressions and with very few features
  • multiple implementations, e.g. terminal, GTK, browser-based (conversion to HTML)
  • NO STYLE SHEETS! It's important that different device types (laptop vs phone vs smart watch) can render optimal representations
  • current adaptive web approach puts too much load on devs -> we get shitty experience all around the board

Mutable vs. immutable

• all "documents" are immutable shards with a hash
  • updates / changes via links to follow-up shards
  • these need to be managed separately by the database as the original shard cannot be modified
• MAIN PHILOSOPHY: everything should be saved in immutable shards, but mutable stuctures can be used for quick access
  • e.g. metadata for a blog post showing which shards update the original post
  • but if this was to be lost, one could browse through all the main shards and retrieve the same information from there
    • slow / computationally expensive, but possible in case things go south
• things like programs (again shards) should have mutable context (settings etc.) which should be stored as map 'key -> shard-id-hash'
  • these maps again are in a higher-level map keyed by the original (program) shard id
  • when things change, a new data shard is stored and the map is updated with the new id -> occasional garbage collection necessary -> store maps in SQLite with flags for mark&sweep...?

Web sites

• either you can host your own instance that searches for shards and converts them to HTML
• ... or you could use open public gateway services with something like "example.com/<shard-id>"
  • gets automatically searched by the gateway and either redirected or returned
  • with a DNS alias one could implement a website like this without private hosting
• or the shard logic could be fetched from the shard server as well with a token that validates that this code is really doing the work
  • i.e. my.private.shards.me/foobar.js needs to be fetched for each session
  • it finds and fetches all objects with some data- attribute that contains the shard id
  • offers an API (functions) for storing new shards
  • one still has to trust the service, as it could upload the contents to the origin server
  • the delivered JS could of course wrap some sending functions, but this is brittle: perhaps they were already wrapped / stored by some other code etc.
  • THUS, WEB AS IT IS CAN NEVER BE FULLY TRUSTED

Things I'd like to reuse between decices / services (i.e. use cases to consider when designing data formats)

• playlists (access to songs via capabilities of course)
• notes
• media
• settings