Merge pull request #7 from pubky/chore/review

feat: clearing concepts for hackweek

Author: SHAcollision

.obsidian/workspace.json

@@ -13,13 +13,26 @@
             "state": {
               "type": "markdown",
               "state": {
-                "file": "readme.md",
+                "file": "App Architectures/2. Client - Homeserver.md",
+                "mode": "source",
+                "source": false
+              }
+            }
+          },
+          {
+            "id": "eaab76dfc5b558e5",
+            "type": "leaf",
+            "state": {
+              "type": "markdown",
+              "state": {
+                "file": "App Architectures/4. Custom Backend.md",
                 "mode": "source",
                 "source": false
               }
             }
           }
-        ]
+        ],
+        "currentTab": 1
       }
     ],
     "direction": "vertical"
@@ -85,7 +98,7 @@
             "state": {
               "type": "backlink",
               "state": {
-                "file": "readme.md",
+                "file": "App Architectures/4. Custom Backend.md",
                 "collapseAll": false,
                 "extraContext": false,
                 "sortOrder": "alphabetical",
@@ -102,7 +115,7 @@
             "state": {
               "type": "outgoing-link",
               "state": {
-                "file": "readme.md",
+                "file": "App Architectures/4. Custom Backend.md",
                 "linksCollapsed": false,
                 "unlinkedCollapsed": true
               }
@@ -125,7 +138,7 @@
             "state": {
               "type": "outline",
               "state": {
-                "file": "readme.md"
+                "file": "App Architectures/4. Custom Backend.md"
               }
             }
           }
@@ -147,8 +160,24 @@
       "hide-folders:Show hidden folders": false
     }
   },
-  "active": "8fd5812ca70f0ab7",
+  "active": "eaab76dfc5b558e5",
   "lastOpenFiles": [
+    "Pubky App/Introduction.md",
+    "App Architectures/4. Custom Backend.md",
+    "Pubky Core/Introduction.md",
+    "App Architectures/1. Introduction.md",
+    "Pubky Applications.md",
+    "App Architectures/5. Complex - Custom Aggregator.md",
+    "App Architectures/2. Client - Homeserver.md",
+    "App Architectures/3. Global Aggregators.md",
+    "Pubky Core.md",
+    "images/complex_custom_aggregator.png",
+    "images/global_aggregator.png",
+    "images/global-aggregator.png",
+    "images/custom_backend.png",
+    "images/client-homeserver.png",
+    "images/custom-backend.png",
+    "readme.md",
     "index.md",
     "ELI5.md",
     "Technologies/Paykit.md",
@@ -158,24 +187,14 @@
     "Technologies/DoH.md",
     "Technologies/DNS.md",
     "Technologies/DHT.md",
-    "Pubky Core/Introduction.md",
     "Pubky Core/Data Stores.md",
     "Pubky Core/Pkarr/4.Architecture.md",
     "Pubky Core/Pkarr/3.Expectations.md",
     "Pubky Core/Pkarr/2.Getting Started with Pkarr.md",
     "Pubky Core/Pkarr/1.Why Pkarr?.md",
     "Pubky Core/Pkarr/0.Introduction.md",
     "Pubky Core/notes/3.Protocols.md",
-    "Pubky Core/notes/2.Incentives.md",
-    "Pubky Core/notes/1.Adoption.md",
-    "PubKy App/Introduction.md",
-    "PubKy App/Client/Introduction.md",
-    "PubKy App/Backend/Web Server.md",
-    "PubKy App/Backend/Introduction.md",
-    "PubKy App/Backend/Indexer.md",
-    "PubKy App/Backend/Aggregator.md",
     "images/obsidian.png",
-    "Concepts/3.Credible Exit.md",
     "quartz/public/404.html",
     "quartz/public/static/og-image.png",
     "quartz/public/static/icon.png",
@@ -188,7 +207,6 @@
     "quartz/public/index.html",
     "quartz/public/Technologies/index.html",
     "quartz/public/Technologies/Paykit.html",
-    "quartz/public/quartz/quartz/static/og-image.png",
-    "quartz/public/quartz/quartz/static/icon.png"
+    "quartz/public/quartz/quartz/static/og-image.png"
   ]
 }

App Architectures/1. Introduction.md

@@ -0,0 +1,9 @@
+Leveraging the [[Pubky Core/Introduction|Introduction]] protocol as the foundational layer, we can envision app a diverse array of client applications with distinct functionalities. While each app can implement its own unique design patterns and user interfaces, they all share a common underlying architecture: interacting with the distributed data stores, colloquially referred to as [[Data Stores |home servers]], through standardized read and write operations.
+
+The [[Pubky Core/Introduction|Pubky Core protocol]] enables a decentralized approach to data management, facilitating seamless communication between clients and their respective data stores. This architecture promotes data sovereignty and enhances system resilience.
+
+Below, we present several high-level designs that showcase the versatility of this architecture, from simple client (browser) apps that interact directly with one or several homeservers to more complex applications with custom aggregators and backends capable of powerful inference:
+
+- [[2. Client - Homeserver|Client-Homeserver]]
+- [[4. Custom Backend|Custom backend]]
+- [[3. Global Aggregators|Global aggregators]]

App Architectures/2. Client - Homeserver.md

@@ -0,0 +1,13 @@
+![[client-homeserver.png]]
+
+In this architecture, we implement a direct communication model between the client application and the home server. This approach minimizes latency and reduces system complexity by establishing a direct data flow pathway.
+
+This design pattern is particularly well-suited for applications with straightforward functionality, especially those that don't require real-time interaction or data normalization. This architectural approach demonstrates optimal performance in use cases characterized by intermittent data operations, where asynchronous read/write cycles are adequate for maintaining data consistency and fulfilling application requirements.
+
+To illustrate the practical applications of this architectural paradigm, consider the following implementation scenarios:
+
+1. Bookmark Management System: A client application designed to store and retrieve user bookmarks directly from the home server.
+2. File Synchronization Utility: Similar to the open-source [Syncthing](https://syncthing.net/) project, this type of application would facilitate direct file synchronization between the client and the home server.
+3. Text Snippet Repository: A lightweight application for creating, storing, and retrieving short text fragments or code snippets as [pastebin](https://pastebin.com/)
+
+These implementations leverage the Pubky Core protocol to establish secure, efficient, and direct data exchange channels between the client and the home server, while the user remains with ownership of their data.

App Architectures/3. Global Aggregators.md

@@ -0,0 +1,10 @@
+![[global_aggregator.png]]
+
+This architectural pattern implements a distributed system model centered around a global aggregation layer, eliminating the need to fetch data from a multitude (maybe thousands!) of homesevers by the client. The core component of this design is a centralized global aggregator that interfaces with multiple home servers, consuming events from each in a unified manner.
+
+Key features of this architecture include:
+
+1. **Centralized Event Processing:** The global aggregator serves as a single point of convergence for event streams originating from disparate home servers across the network.
+2. **Policy-Driven Filtering:** The aggregators can optionally implement a configurable set of policies and filtering rules, allowing for dynamic event processing based on predefined criteria.
+3. **Client Flexibility and Aggregator Choice:** Clients consume data from the global aggregator stream. However, if a client finds the enforced rules of one aggregator unsuitable, it retains the flexibility to switch to an alternative global aggregator that better aligns with its requirements or selectively look for the homeservers itself.
+4. **Scalable Event Distribution:** By centralizing the aggregation process, this architecture facilitates efficient event distribution to multiple clients, potentially reducing redundant processing and network overhead.

App Architectures/4. Custom Backend.md

@@ -0,0 +1,20 @@
+![[custom_backend.png]]
+
+This architectural design introduces a more sophisticated data flow model, incorporating an intermediary backend layer between the client application and the home server. This backend functions as a middleware, enhancing the system's flexibility and data processing capabilities.
+
+The backend can be potentially comprised with many components. These components will depend on the client app needs, but these are the main ones
+
+1. __Indexer__: Responsible for data normalization, ensuring consistent data structures and optimizing query performance.
+2. __Aggregator__: Implements event filtering logic, allowing for selective data propagation based on predefined criteria.
+
+
+This architecture supports two distinct data consumption patterns:
+
+a) For scenarios requiring both data normalization and event filtering, the client interacts with the backend layer, as an endpoint. The aggregator processes the event stream from the home servers, applying filtering rules before passing the data to the indexer for normalization.
+
+b) In cases where only data normalization is necessary, the backend can bypass or not implement the aggregator, consuming events directly from the home server via the indexer.
+
+This modular approach allows for fine-grained control over data processing, enabling efficient resource utilization and optimized client-side performance based on specific application requirements.
+
+An example of a complex backend that does aggregation, normalization, indexation, filtering (compliance) and powerful inference is [Pubky-Nexus](https://github.com/pubky/pubky-nexus), the backend that powers the [[Pubky App/Introduction|Pubky App]] social features.
+

PubKy App/Backend/Aggregator.md Pubky App/Backend/Aggregator.md

@@ -8,4 +8,4 @@ Imagine you're trying to find a specific document in a large library. The backen
 
 - [[Aggregator|Aggregators]] execute a **data retrieval protocol** to obtain data from **data storage**, initiating a process that retrieves and collects data from various sources.
 - [[Indexer|Indexers]] receive aggregated data from the **Aggregators** and initiate a rigorous **data normalization** process, transforming and converting the data into a standardized format to ensure consistency and accuracy.
-- [[Web Server|Web servers]] provide the requested data to [[PubKy App/Client/Introduction|Pubky client]]
+- [[Web Server|Web servers]] provide the requested data to [[Pubky App/Client/Introduction|Pubky client]]

PubKy App/Backend/Indexer.md Pubky App/Backend/Indexer.md

@@ -2,9 +2,9 @@
 
 ![pubkey-client](../../images/pubky-header.png)
 
-The Pubky client will be available as both a desktop application and a hosted website service (standard website) that allows users to interface with the social media layer of [[PubKy App/Backend/Introduction|Pubky Backend]] using [Synonym](https://synonym.to/) hosted services.
+The Pubky client will be available as both a desktop application and a hosted website service (standard website) that allows users to interface with the social media layer of [[Pubky App/Backend/Introduction|Pubky Backend]] using [Synonym](https://synonym.to/) hosted services.
 
-Using the library analogy again, the Pubky Client is like a personalized research assistant who takes the prepared documents from the librarian ([[PubKy App/Backend/Introduction|backend]]) and creates a customized report just for you. This report is designed to be easy to read and understand, with all the relevant information presented in a clear and concise manner.
+Using the library analogy again, the Pubky Client is like a personalized research assistant who takes the prepared documents from the librarian ([[Pubky App/Backend/Introduction|backend]]) and creates a customized report just for you. This report is designed to be easy to read and understand, with all the relevant information presented in a clear and concise manner.
 
 - Users are able to take control of the data and exit the Synonym hosted services and run their own without hampering discoverability ([[3.Credible Exit|credible exit]]).
 

PubKy App/Backend/Introduction.md Pubky App/Backend/Introduction.md

@@ -1,6 +1,6 @@
 ![pubkey-app](../images/pubky-app.png)
 
-> Synonym will be initially hosting: [[Data Stores|Data stores]] and [[PubKy App/Introduction|Pubky App]]
+> Synonym will be initially hosting: [[Data Stores|Data stores]] and [[Pubky App/Introduction|Pubky App]]
 
 ## Overview
 
@@ -13,12 +13,12 @@ Pubky-app's initial focus is building a decentralized social media protocol.
 - [[Aggregator|Aggregators]] collecting social graphs
 - Feeds of followings' activities
 - [[Search|Searching]] profiles and posts
-- Notification delivery through [[PubKy App/Backend/Introduction|aplication backends]] 
+- Notification delivery through [[Pubky App/Backend/Introduction|aplication backends]] 
 - Distributed moderation through user blocking
 
 ## Components
 
-The Pubky App is a complex system that can be broken down into two main components: the [[PubKy App/Backend/Introduction|backend]] and the [[PubKy App/Client/Introduction|client]]. These two pieces work together to provide a seamless user experience.
+The Pubky App is a complex system that can be broken down into two main components: the [[Pubky App/Backend/Introduction|backend]] and the [[Pubky App/Client/Introduction|client]]. These two pieces work together to provide a seamless user experience.
 
 ##### Backend: The Data Organizer
 
@@ -33,4 +33,4 @@ It is the part of the Pubky App that you interact with directly. It's responsibl
 The early versions of Pubky app take some shortcuts over the [[Pubky Core/Introduction|Pubky Core]] design. The MVP app is centralized, therefore we saved time and complexity by aggregating functionality into fewer components. The main two components are the `Homeserver` and the `Indexer`
 
 - The `Homeserver` fulfils the function of [[Data Stores|data stores]], [[0.Introduction|Pkarr]] republishing for users, identity-provider (Oauth-like sign-in). Users maintain a trust relationship with the `Homeserver`.
-- The `Indexer` fulfils the function of the [[PubKy App/Backend/Introduction|backend]] for the Pubky App. 
+- The `Indexer` fulfils the function of the [[Pubky App/Backend/Introduction|backend]] for the Pubky App. 

PubKy App/Backend/Web Server.md Pubky App/Backend/Web Server.md

@@ -10,12 +10,14 @@ Pubky Core is built on a few core concepts:
 
 Pubky Core's distributed architecture aims to provide user autonomy through [[3.Credible Exit|credible exit]] between interchangeable components.
 
+- **[[1. Introduction|Pubky App Architectures]]** can be very diverse. From simple web client applications that connect to a single [[2. Client - Homeserver|homeserver]] for data storage and retrieval to [[4. Custom Backend| complex backends]]  that do aggregation and powerful inference over many homeservers creating abstractions such as social media with [[1.Web of Trust|web of trust]].
+
 ## Target Users
 
 Pubky Core is made for developers and builders of internet software products. We will actively work to cultivate a community of builders and startups using our tech.
 
-[[PubKy App/Introduction|Pubky App]] is made for anyone interested in social media and online publishing that wants new ways to control their data and more control over which posts they see.
+[[Pubky App/Introduction|Pubky App]] is made for anyone interested in social media and online publishing that wants new ways to control their data and more control over which posts they see.
 
 ## Why? What is the reward?
 
-The reward for everyone is a more open, privacy-focused, usable, modular and secure web. The reward for [Synonym](https://synonym.to/) is to be positioned to disrupt Big Tech as an industry, gaining user recognition and reputation through the effort of building a decentralized ecosystem. The ultimate potential is for [Synonym](https://synonym.to/) to become a major player in online publishing & social media, while also monetizing in ways similar to Google, as well as opportunities to introduce new users to our ecosystem of products and services.
+The reward for everyone is a more open, privacy-focused, usable, modular and secure web. The reward for [Synonym](https://synonym.to/) as lead of this project is to be positioned to disrupt Big Tech as an industry, gain user recognition and hopefully reputation through the effort of building a decentralized ecosystem. The ultimate potential is for [Synonym](https://synonym.to/) to become a major player in online publishing & social media, while also monetizing in ways similar to Google (e.g., search and aggregation over the web -> search / aggregation and tools for sovereign data stores), as well as opportunities to introduce new users to our ecosystem of products and services (e.g., bitcoin payments infrastructure)

PubKy App/Client/Features/Bookmarks.md Pubky App/Client/Features/Bookmarks.md

@@ -1,4 +1,4 @@
-[[PubKy App/Introduction|Pubky App]] is a decentralized alternative to social media platforms like Twitter. It uses a distributed architecture where:
+[[Pubky App/Introduction|Pubky App]] is a decentralized alternative to social media platforms like Twitter. It uses a distributed architecture where:
 
 - Users host their own data on independent [[Data Stores|data stores]]
 - Services [aggregate](Aggregator.md) data across stores into a unified view

PubKy App/Client/Features/Layouts.md Pubky App/Client/Features/Layouts.md

@@ -1,12 +1,12 @@
 # Welcome to the Pubky Knowledge Base
 
-This is the [[Pubky Core/Introduction|Pubky Core]] and [[PubKy App/Introduction|Pubky App]] knowledge base. Currently these docs are a mix of reality, dreams, visions, technicalities, actuality and plans.
+This is the [[Pubky Core/Introduction|Pubky Core]] and [[Pubky App/Introduction|Pubky App]] knowledge base. Currently these docs are a mix of reality, dreams, visions, technicalities, actuality and plans.
 
 1. Cloud storage with [[3.Credible Exit|credible exit]] (in the current implementation, it is just a web server + [[0.Introduction|pkarr]])
 2. On top of that we build a [[1.Web of Trust|Web of Trust]]
-3. [[PubKy App/Introduction|Pubky App]] is the main demonstration of how should this distributed data be crawled, indexed and searched at scale (albeit at a centralized setup).
+3. [[Pubky App/Introduction|Pubky App]] is the main demonstration of how should this distributed data be crawled, indexed and searched at scale (albeit at a centralized setup).
 4. By decoupling features from centralized control, independent development remains open and vibrant.
 
 ## Get Started
 
-A good place to start is the [[PubKy App/Introduction|Pubky App]], [[TLDR]] or our pubky-core [[ELI5]]
+A good place to start is the [[Pubky App/Introduction|Pubky App]], [[TLDR]] or our pubky-core [[ELI5]]