Dear backers,
It's humbling to see new supporters joining, thank you so much for believing in what we're doing. Last month I mentioned how the conceptual phase of our new protocol is done, and we have started implementing it, fleshing out all the details.
This starts with getting the foundation right, so I'll update you on low-level components that we worked on.
Log
The log is at the heart of the database, and everything else depends on it. In SSB, this was typically called an "append-only log" and in Manyverse we used async-append-only-log built by Arj. In the new protocol, PPPPP, the log is in some ways simpler and in other ways more complex. It's more complex because it's not "append-only", allowing you to delete records entirely or partially, which in honesty is a way of rewriting the past. However, only a specific type of rewriting is allowed: removal of information. Then, it's in some ways simpler, because the implementation can be tailored to the specific needs of our database module, which loads up everything in memory. This means we don't need to implement streams that scan all records, we just need to load up all records into memory, at startup.
Last month I worked on the log, implementing erase() (partial record deletion), and compact() (rewriting the log to fill all empty spaces), and tests for both.
Message format
Mikey started implementing PPPPP in Rust (woah!) which proved to be a valuable exercise to understand the protocol better. This led to feedback on details on the message format, which led to fixes in the JavaScript implementation and better alignment between implementation and documentation.
We're doing our best to foresee issues with the message format before the wider community can experiment with it, but we arrived at an even better idea. When the protocol is launched, it will operate in a "testnet" mode for a few months, with a pre-defined end-date for the testnet. People will be able to use the testnet for real world use cases, but as a consensus, implementations will agree to wipe out everything once the end-date arrives. This will allow us to gather feedback and reserve the opportunity to radically change the message format if needed, before the protocol is launched in "mainnet" mode.
Greenfield cryptography
Another thing Mikey learned while implementing PPPPP in Rust is that certain networking and encryption components of the protocol can be easily upgraded. libsodium's secretbox, based on XSalsa20-Poly1305, is considered "legacy" in Rust Crypto. The modern replacement is (X)ChaCha20-Poly1305, used by TLS 1.3, which is also provided by libsodium. Thus began a quest to upgrade PPPPP's cryptographic primitives, why otherwise start a greenfield project with legacy crypto?
When Dominic created SSB, there was no good module for authenticated and encrypted streams, so he created box-stream for SSB. Mikey designed a successor named Secret Channel, with a complete specification, which shares similarities with box-stream and Noise's post-handshake transport. The goal is to be easier to implement than box-stream, and include more application responsibilities than Noise's post-handshake transport.
Speaking of Noise, a well-established cryptographic framework, we studied whether we could simply use Noise for SSB or PPPPP. Unfortunately, we were unable to adapt Noise for our needs. Instead Mikey is taking the opportunity of PPPPP (a breaking change from SSB) to tweak and upgrade Secret Handshake, our authentication protocol, with a fresh specification.
Misc updates
A few other things I worked on: a new major version of muxrpc which removes some legacy code, and improvements to the garbage collection module to guarantee that it schedules cleanup and log compaction correctly.
All that leaves our current roadmap status at this:
What's coming up next? For me personally, I'll focus on the "conductor" module, which is the glue that manages replication and garbage collection for your account and all the accounts you follow.
Mix is meeting up with Keks to talk about the encryption format. Some years ago, the two created envelope-spec as an upgrade of SSB message encryption to accommodate for private groups. Now, they will discuss how to adapt envelope-spec to PPPPP.
Mikey will finish Secret Channel and Secret Handshake for our JavaScript implementation, and then will continue with Rust, potentially helping write the hub server in Rust.
Last but not least, happy holiday season! Spend it with your loved ones, and let's touch base again in 2024. It will be the year when PPPPP is launched. ;)
– @andrestaltz