# streamful

WORK IN PROGRESS -- this specification is not finalized at all

streamful is a customizable, distributed social network platform. Its purpose is
to enable the darker parts of social networks (attention capitalization) to be
able to serve higher purposes by allowing people to rapidly experiment with new
expressions of their lives safely. It does this with a novel capacity for
controlling access to your data and visualizing it in the exact ways you want it
to show up.

## Stream

A stream contains arbitrary content that is authenticated as coming from whoever
controls the private side of a cryptographic key pair.

## Stream server

A stream server hosts streams. It provides various kinds of access control as to
what kind of content is allowed on the server. It also provides an access
control list of which keys are allowed to perform which operations on which
streams.

## Domain

A stream can be identified by a domain, which means that you don’t necessarily
need to know which server the content is on — you can use the domain to fetch
the content for that stream.

A stream may only be identified by one domain, and must make that choice at
inception. The domain must be registered in a public DNS server which contains a
TXT record that contains the public key of the stream’s owner.

A server that has chosen to host the stream will then use that public key to
verify ownership.

## Message

The only person that can publish a message to a stream is the owner of the 
stream's keys. When someone else fetches messages from a stream, they should 
verify that the message was signed by the stream owner. The first message in
every stream identifies the public key of the stream.

### Correction

A message may be corrected after the fact. This works just like a retraction
except new content is put in place of the old. The new message should show up in
the same position in the stream as the old one. The fact that there was a
correction should appear in the stream at the time of the correction.

### Replies

Anybody may submit a reply to a message on a stream that they own. If people
subscribe (see below) to the stream containing the message, and the stream 
containing the reply, then their server or client may choose to show replies in
context somehow. 

If a message that was replied to is ever retracted, then the reply will simply
show as a reply to a retracted message. Any cached content must be purged.

### Thread

Messages may be grouped into threads. The ID of the message at the beginning of
the thread must be used as the thread ID of every message in the thread.

If the message forming the root of a thread is ever retracted, then the messages
in the thread will show as belonging to a retracted thread. Any cached content
must be purged.

## Substream

A substream is a stream contained within another stream. A substream may be
identified by a domain or not.

A substream can be moved between streams, but it may only have one primary
stream. The process to change streams involves notifying the previous primary
stream of the new stream. If the new stream is on a different server, the new
server may fetch all content from the previous server before the transfer is
complete. The previous stream will refuse all future messages submitted to that
substream. There is no obligation for a server to retain messages of a substream
that has moved to another server. However, servers must retain the fact that a
particular substream has moved and where it has moved to in perpetuity.

### Substream moderation

A substream may be created within streams that have specifically been configured
by their owner to allow substreams. Those substreams may also be required to be
approved. If your substream is denied you may or may not be allowed to submit
again with or without changes to the creation request.

## Stream ID

A stream ID tells you how to access the messages on a stream. The format is 
`id@domain`. The `id` portion is for streams that are not identified by domains,
but are used to access substreams of the given domain.

If a substream is contained within another substream, add another `id@` to the
left. So, `cats@pets@animals.com` means there is a stream called animals.com 
which contains a substream `pets@animals.com` which contains a substream
`cats@pets@animals.com`

## Subscription

A stream may subscribe to another stream. If the messages for the destination
stream are on another server, the subscriber’s server must maintain a mirror
copy of the given stream so long as there are active subscriptions.

The subscription is recorded as a special subscription-type message in the
subscriber stream.

Subscription requests may only be submitted if signed by an authorized key. They
may only be retrieved by those who sign their fetch requests with an authorized
key.

When a stream with subscriptions moves to another server, the old server stops
fetching messages.

### Signed subscription

A signed subscription allows the destination stream "subscribe back," which
gives the destination the ability to read replies from the subscriber. What 
happens is:

1. The subscriber's stream publishes a signed message requesting subscription
   to the destination stream
2. The subscriber's server holds onto that message amongst all its other
   subscriptions for the destination stream
3. When the subscriber's server fetches new messages from the destination
   stream, it includes a reference to all subscription messages for the
   destination server to check
4. When the destination server receives a new subscriber for the first
   time it notifies the destination stream of the subscriber stream, at
   which point it becomes possible for the destination stream to subscribe
   back if it wants to

Subscribers are identified by their public key and their stream ID, since
the stream ID may change. If the same public key is used for multiple

subscriptions from different stream IDs, the destination stream may choose
how to handle the collision. It is possible one of the streams may have moved
or the stream's keys have been compromised.

### Subscription updates

Servers may configure how often other servers may fetch subscription updates.
They may also choose to offer some servers preferential treatment in whatever
way they wish.

### Push style

Servers may also choose to offer “push” style updates, with a few caveats:

1. Messages destined for multiple subscribers on a server may only be delivered


   once per server
2. Messages may be published in batch sizes configurable by the push recipient
   within parameters that the publisher allows
3. A server may not fetch messages if push is enabled

## Private messages

Messages sent to a single subscriber have no right to privacy. If the
subscriber decides to make those types of messages available on their stream
they may do so. However, they must make that choice on a message-by-message
basis.

Likewise, servers have no obligation to store messages encrypted at rest.
This means that, while a server will only deliver private messages to the
subscriber they are destined for, anybody with access to that server's database
can see the messages. 

The only real protection is end-to-end encryption. 














## Stream moderation






A stream may set moderation rules that apply to itself and all its substreams.

These rules include which streams may be subscribed to or not. It may use a


combination of allow and deny lists, which may specify any level of granularity.


For example, a denial of `pets@animals.com` also denies `cats@pets@animals.com` 

but not `wildlife@animals.com`, whereas a denial of `animals.com` blocks all 



content from that stream and all its substreams, recursively.




Moderation rules may be set only by those who have keys authorized to change








them. They may only be viewed by those who have keys authorized to view them.






A server may also set moderation rules which apply to all streams it hosts.





## Replies and threads across streams

If a message in one stream replies to a message in another stream, it must
subscribe to the other stream. This way a full picture of the thread may be
maintained.

If the stream of the original thread also subscribes back, it can make all the
messages your stream submits in reply to its messages available to its other
subscribers.

There is one caveat: if a stream requests messages from another stream, it may
only receive that stream's messages.

## Reading a stream

When fetching stream messages, you only get the messages of that stream. If your
server is configured to do so, it may "auto-subscribe" to messages from other

streams mentioned in said messages. Those rules can be configured 
programmatically by the server to save on storage costs. 

## Stream access

A stream’s discoverability may be controlled by the stream’s owner, or an
authorized representative:

1. Can only be viewed by the owner (private, the default)

Every message published to a stream must have a type. If it doesn’t have a type
it is assumed to be a plain text note.

Servers must accept all message types, regardless of if it “knows” how to
process it. Messages of unknown types must be stored as-is. Clients may choose
to perform their own processing of these message types.

## Server side aggregates

In order to minimize duplication of effort in generating aggregate data, servers
can provide aggregates for specific event types. For example, a reaction event

type could produce an aggregate across a specific message ID. The server would


then feed back only the most recent aggregate and not return the individual
reaction messages that led to it, unless requested (for audit purposes).



Aggregates can be configured at any level. An aggregate at the root of the 
server is invoked once per message fetched across all subscriptions. One 
attached to a specific stream is only invoked for its messages and those of


its substreams, unless otherwise configured.


Since the server would need to read the content of these messages, any 

messages with end-to-end encryption would not be submitted to any aggregate.



## Stream customization



Any stream can customize their stream using arbitrary code that can be uploaded
without needing to be compiled on a computer. In this way, stream managers could
even use their phones to program their own stream experience.

### Customizable parts


Users can upload scripts to their server that customize various things about


their stream:

1. “Page” layout — a stream has a home page and sub-pages. A page experience is
   one of structure, not style. It shows how data are organized. Pages can
   choose to display data from the stream, and any substreams, however they 
   want. Further, apps that render pages have a lot of freedom as to how
   a page is rendered
2. Aggregators — as messages come in, a script can be tied to a specific message
   type, updating server side aggregates
3. Widgets — have access to raw messages and aggregates and can display them in
   interesting ways, using charts and a basic layout inspired by html. A page is


   a container for widgets




### Security




Stream scripts do not have access make arbitrary network connections or perform

other kinds of IO. They can only perform pure functional transformations based
on data that is passed to them, as well as access data within their scope.


For example, an aggregate on the root stream of a server has access to all
other aggregates on the server. An aggregate applied only to a substream has 
access only to that substream's aggregates.

(defn- stream-agg-k [sid] (keyword (format "stream-agg-%s" sid)))

(defn- put-data
  ([table kt k vt v] [:put table k v kt vt])
  ([table k v] (put-data table :data k :data v)))




(defn- put-encoded
  ([table kt k v] (put-data table kt k :bytes (cbor/encode v)))
  ([table k v] (put-encoded table :data k v)))

(defn- decode-pair [[k v]] [k (cbor/decode v)])

(defn- get-decoded

                  {"id" id
                   "psk" client-psk
                   "srt" t
                   "m" msg
                   "o" original-msg-bytes})
     (mapping-put-msg id k)]))

;; todo allow substreams of substreams
;; for now all substreams are implicitly of the root stream

(defn is-owner? [tx client-psk]
  (let [existing-owner (get-db-owner tx)
        result (creq/eq? existing-owner client-psk)]
    {:is-owner?
     {:given client-psk
      :actual existing-owner
      :result result}}
    (and existing-owner result)))

(defn claim-server! [db client-psk]
  (db/with-transaction-kv [tx db]
    (let [existing-owner (get-db-owner tx)]
      (if existing-owner
        :already-claimed

; You should have received a copy of the GNU Affero General Public
; License along with streamful.
; If not, see <https://www.gnu.org/licenses/#AGPL>.
;

(ns streamful.transport
  (:require [clj-cbor.core :as cbor]
            [clj-cbor.tags.content :as tags.content]
            [clj-cbor.tags.numbers :as tags.num]
            [clj-cbor.tags.text :as tags.text]
            [clj-cbor.tags.time :as tags.time]
            [clojure.walk :as walk]))




(def ^:private write-handlers
  (merge tags.content/content-write-handlers
         tags.num/number-write-handlers
         tags.time/epoch-time-write-handlers
         tags.time/epoch-date-write-handlers
         tags.text/text-write-handlers))

(def ^:private read-handlers
  (merge tags.content/content-read-handlers
         tags.num/number-read-handlers
         tags.time/instant-read-handlers
         tags.time/local-date-read-handlers
         tags.text/text-read-handlers))

(def ^:private cbor-codec
  (cbor/cbor-codec
    :write-handlers write-handlers
    :read-handlers read-handlers))

(defn encode-msg [msg]
  (cbor/encode (walk/stringify-keys msg)))

(defn decode-msg [msg]
  (cbor/decode cbor-codec msg))

(ns streamful.protocol-test
  (:require [clj-cbor.core :as cbor]
            [clojure.test :refer :all]
            [crypto.equality :as creq]
            [streamful.client :as client]
            [streamful.crypto :as cr]
            [streamful.protocol :refer :all]
            [streamful.test-aggregates :as tagg]
            [streamful.test-asserts :as ta]
            [streamful.test-cfg :as tcfg]
            [streamful.test-crypto :as tcrypto]
            [streamful.test-protocol :as tp]
            [streamful.test-protocol :refer :all]
            [streamful.transport :as transport])
  (:import (java.io ByteArrayInputStream)

                (is (= expected-test1-messages
                       (map tp/get-m test1-messages)) r2))

              (testing "we can validate and reassemble from original binaries"
                (is (= expected-root-messages
                       (map reassemble-original root-messages)))
                (is (= expected-test1-messages
                       (map reassemble-original test1-messages))))))









          (testing "good error message when stream is missing"
            (is (= {:response {"status" "stream 'missing-stream' not found"}}
                   (get-as client keys1 "missing-stream"))))

          (testing "streams are private by default, show as missing"
            (is (= {:response {"status" "stream 'root' not found"}}

         "m"
         {
          ;; when created on the client, ms since epoch
          ;; this would normally be a bigger number, but the above test has
          ;; time hard-coded for consistency of test assertions
          "ct" 22,

          ;; command -- what should subscribers do with this message?
          ;; `put` just means "put it in your subscription"
          "c" "put",

          ;; base64-encoded form of `:psk` above for verification of internal
          ;; consistency to prevent forgery
          "k" "fCJ8C+MBKGuwLPz771W4IhuAxYStrwnFHKkpRGquLV8=",

          ;; message itself

        ;; performs all the validations and disassembles
        reassemble-original

        ;; should be identical to the outer `"m"` above
        clojure.pprint/pprint))
  )

(deftest server-side-subscriptions-test
  (testing "subscribe"
    (testing "same server"
      (ta/pending "updates destination properly"))
    (testing "different server"
      (ta/pending "notifies remote only once")
      (ta/pending "notifies remote again if remote fails")))
  (testing "fetch subscriptions"
    (testing "per stream"
      (ta/pending "all messages")
      (ta/pending "individual deny list")
      (ta/pending "by rules")
      (ta/pending "only once even with multiple subscribers"))
    (testing "server defaults"
      (ta/pending "allow list")
      (ta/pending "deny list")))
  (testing "unsubscribe"
    (testing "same server"
      (ta/pending "updates destination properly"))
    (testing "different server"
      (ta/pending "notifies remote only once")
      (ta/pending "notifies remote again if remote fails")
      (ta/pending "purges remote on last"))))