An overview of the Politeia system architecture

Originally published on Decred on Medium (Medium, Sep 2018).

Politeia is a powerful off-chain governance tool. It empowers users to submit detailed proposals, provides a platform to review them, and allows users to vote which proposals should receive funding from the project treasury. Politeia is designed for Decred, but its code is open source and can be used by anyone. This post gives an overview of the Politeia system architecture.

For those of you who are not familiar with what Politeia is, I will recite the title of this blog post from Decred’s project lead (Jake Yocom-Piatt):

Politeia: Proposals in a Timestamped Filesystem

Expanding this a bit more: Politeia combines a file storage backed with a timestamping backend by anchoring the commit hash into the Decred blockchain, using git and dcrtime. Quoting Jake once more:

[…] (dcrtime) creates a time-ordering that is computationally infeasible to recreate. […] This time-ordering, combined with git’s standard process of hashing commit data to generate a commit hash, is a means of demonstrating the provenance, i.e. origin or source, of that data.

In summary, Politeia is a blockchain-anchored proposal system where all user actions, despite their access level (admins or regular users), are completely transparent and auditable. Let’s find out how this system works.

Politeia system architecture

If you take a look into Politeia’s repository, you will quickly notice some of the words in the figure above. While there are other pieces which are not included in this diagram, I found it best to highlight the essential parts of the system to create a complete yet concise overview of Politeia’s main functionalities.

Now we will go over each part of the system to describe how they work and how they compose the big picture.

1) Politeia www

Politeia www (or just “www”) holds the business logic of the system. It handles creating proposals, editing proposals, creating users, modifying users, etc. It exposes a REST API for external clients to access its resources. Politeiawww can be subdivided into four high-level blocks:

• Backend: is the www boss. The backend handles communicating with politeiad (d) to send and receive data. This is also the one in charge of handling incoming API requests and performing updates on the inventory and the database.
• Inventory: is in-memory storage that caches all proposal records and its comments in the www layer. When www starts, it initializes the inventory by fetching every proposal and comments from d and saving it into a map cache, the inventory. It will then be updated on every action which affects the proposal records permanently.
• Paywall: is the chunk of the code which handles processing payments. It processes the payments by keeping track of which payments are still waiting for confirmations. Once it detects that a payment was successfully executed, it will update the database accordingly. Right now, paywall payments are necessary for registering to the platform and also for submitting proposals.
• Database: is a key-value database where all user’s information is stored. Most of the user data, such as username, id, and email aren’t sent to d; instead, they are kept into the www database. The only information sent to d which allows connecting a piece of information (proposal, comment, action, etc.) to a user is the user public key, which is part of the record metadata. A user can have multiple public keys and www keeps a record of all of them, associating it to its author id.

There are two main clients for www: the politeiagui and the politeiawwwcli. The former is a web application using React/Redux, it is the front end of the Politeia system, and it’s being tested on testnet and mainnet. The latter is a command line interface application utilized to interact with the www API. Both of those applications allow you to access every action or resource available through the API.

The www system is 100% functional right now; however, there are a few significant improvements which the Decred’s team is looking forward to apply regarding the system architecture once Politeia is fully released. For instance, optimize the caching by replacing parts such as the Inventory by third party software, for example Redis or Cockroachdb. The same kind of replacement could happen to the Database where the key-value storage approach would give rise to a better-suited database option.

2) Politeia daemon

Politeia daemon (or just “d”) takes care of many different tasks such as saving records into git repositories and “anchoring” data into Decred’s chain. It also exposes an API for communicating with other parts of the system and external clients. Let’s go through its composition pieces:

• Git Backend: is the most significant part of d. It handles managing the file storage by updating the two git repositories (vetted and unvetted), so it will ultimately be processing the outside requests that are intending to change the records data.
• Journals: is a system that handles journaling (or caching) information that would take too long on the backend to execute. Marco Pereboom, the tech lead of Politeia, stated: “[…] Since everything has cryptographic attestation and uses git, it isn’t feasible to always write it directly into the repository. So the data is journaled and then replayed into the git repo every hour.” This technique is currently utilized for comments and casted votes, while proposal changes are directly committed to the git repository.
• Anchor: is the part of the system that handles anchoring the commit hashes into Decred’s chain in a pre-defined period. It checks periodically for commit hashes that haven’t been anchored yet. It will then encrypt the commit hashes, generating what we call “digests”. A Merkle tree is made from those digests. The Merkle root and the digests are sent to dcrtimed which will apply some more data processing and finally store the data into Decred’s chain, generating a transaction (along with some other details) which is sent back to Politeia and then committed into the git repository. The diagram below shows the process of taking the Politeia data and anchoring it on Decred’s blockchain.

Politeia anchor flow chart

• Decred plugin: is the part of the system that handles votes and comments. Both comments and votes are journaled before getting committed into the git repository. This plugin makes sure that comments and votes are in sync with the git repository. It also communicates with dcrdata to fetch pieces of information necessary to manage the voting process, such as starting the voting on a proposal and defining which tickets are eligible for casting votes on that proposal.
• The unvetted git repository: stores the proposals records which aren’t publicly visible. The records committed into the unvetted repository can be either unreviewed or censored by an admin.
• The vetted git repository: stores the proposals records which are publicly visible. The records committed into the vetted repository are already reviewed and approved by an admin. This repository will also be published on GitHub. You can access the vetted repository for testnet here. The comments and votes journals are also committed into this repository.
• Journals on storage layer: merely refers to the files holding the information waiting to be committed into the git repository.

The main client written for politeiad is the politeiavoter which allows users to cast votes on proposals. This program requires a wallet setup because the user tickets are used as the base for defining how many votes a user can cast to a proposal. Note that it is also possible to cast votes using the politeiawwwcli. Furthermore, the support for casting votes is already integrated into the Decrediton GUI wallet and should be released soon.

There are also a few significant improvements expected to be done on politeiad regarding its architecture. They follow the same trend of politeiawww which is to replace some parts of the code with better suited storage software. In politeiad, it is expected to happen on caching elements where there is information stored in in-memory variables.

3) External systems

Politeiad and politeiawww need to communicate with two external systems: dcrtimed and dcrdata. The former is used to write into Decred’s chain as explained before. The latter is used to fetch information about the Decred’s chain so it’s possible, for instance, to determine the status of proposal voting (not started, on voting, voting finished) by checking the chain height and also to process payments by checking for transaction confirmations.

Conclusion and reflection

This post has clarified what the most critical parts of the Politeia system are. If the content was more technical than you expected, you may be looking for a broader explanation of Politeia. If that’s the case, or if you are still wondering what Politeia is about, check this excellent article from Richard Red.

If you want to dive deep into the code or even start contributing to the project, go through the git repositories referenced along the text (links at the bottom). In any case, we welcome you to reach the # politeia channel on the Decred Slack or Matrix and ask your questions.

I am also in the process of learning what great possibilities Politeia can bring to Decred’s governance and in other contexts. I see a world of opportunities enabled by decentralized governance where the system designed reflects all the principles of the community. Politeia’s system architecture reflects every meaningful point made into the governance topic so far and still has a long way to go. I’m excited to see what’s coming next. What about you?

Repositories:

Politeia: https://github.com/decred/politeia

Politeiagui: https://github.com/decred/politeiagui

Dcrdata: https://github.com/decred/dcrdata

Dcrtime: https://github.com/decred/dcrtime