Design and implementation of the g/toolkit
rc scripts for a decentralized, fault-tolerant grid
The g/toolkit is a set of interrelated Plan 9 rc scripts for indexing and connecting to a diverse set of external 9p services. The assumption is that the set of services begins as unknown, and may change unpredictably. The goal is to enable a diverse set of nodes (assumed to be a random collection of mostly residential desktop computers) to provide services to one another, and for nodes to maintain knowledge of each other despite possible network disruptions and the random appearance and disappearance of different nodes on the grid.
Design goals and priorities
- Ease of use
- Fault-tolerance
- Decentralization
- Follow Plan 9 principles
- Interoperate with Inferno
Ease of use is the primary goal and motivation for the project. The scripts are written to provide a set of tools for a diverse group of clients/servers to share information about services they are providing using simple, semantically meaningful commands. Ease of use for inexperienced users is the motivation for certain decisions such as causing the scripts to prompt the user to input required information if run without the necessary arguments.
Fault-tolerance is an absolute requirement for a system presumed to be comprised of 'whoever feels like connecting' at any given moment. Within any decently sized collection of random worlwide residential internet users, quality of interconnection is likely to vary wildly, and users should be allowed to connect and disconnect in whatever fashion they choose. This motivated the use of tools such as recover for handling grid connections, and managing and cleaning up broken or lost services is the function of a sizable percentage of the code.
Decentralization is both a goal, and a requirement. Fault-tolerance in a grid of symmetrical peers with potentially low quality connections requires decentralization and redundancy to succeed, and decentralization is a valuable goal for reasons that might be labeled political or ethical - except that such concerns cannot be divorced from design in general. A decentralized system prevents a controlling authority from acting to censor or harm the grid as a whole, and it also limits the potential harm of any malicious actions taken against the grid. It is assumed that a grid composed of equivalent peers should all have both responsibility and authority in equal measure for their own nodes.
Follow Plan 9 principles is an obvious requirement for Plan 9 software, but it is a beneficial requirement. Most of the Plan 9 principles are general principles of good design, and following them also means that the power of the Plan 9 tools can be leveraged more effectively. In practice, this means focusing on the semantics of the grid namespace and how it is constructed and maintained. The toolkit is inteded as a set of small, orthogonal commands to allow the user to build their own connection to the grid by connecting to whatever services they choose.
Interoperate with Inferno is another design goal that is constructive, not restrictive. The Inferno registry system serves as a model for much of the toolkit's semantics and functionality, and an attempt has been made to provide limited interoperability. Complete interoperability is a goal for further development. Another goal is a complete port of toolkit to the Inferno environment, with whatever modifications are needed to integrate seamlessly with the existing registry tools.
Implementation: Plumbing, mntgens, recover, filesystem as database
The architecture created and maintained by the g/toolkit can be summarized simply. A plumber and a series of stacked mntgens are used to create a shared namespace anchored at /g for a rio session. At least one 'service registry' (either Inferno registry or a Plan 9 g/mklocalreg created registry) is attached at /g/r (with a unique mntgen for each registry or other service). Information about available services and what services are currently being provided and exported are stored in a set of files within /tmp/grid. Any node can create and export a registry, and it is assumed that each node will attach to and use multiple registries simultaneously. Once an initial seed connection has been made, further connections are made with simple name based semantics, and the current updated status of the grid (available services, missing or dead connections) can be maintained automatically.
All connections are made via recover to provide fault tolerance for periods of downtime, and to protect actions taken targeting services that have left from getting stuck in failed rpc syscalls. Recover also helps keep the /g namespace from becoming globally unresponsive due to broken connections. The use of a unique mntgen for each serice attached is also intended to support this goal.
The grid scripts track and maintain the state of the grid via a series of small files organized with the /tmp/grid namespace. This 'filesystem database' provides the foundation for semantic 'magic' such as g/getsvc SERVICENAME which locates and attaches the named service based on saved registry data, and then stores the information about the connection. Storing information about the current status of the grid enables that status to be maintained and restored even if the active 9p connection is lost.
The grid state is checked and maintained by scripts which use the information in /tmp/grid to tell them what services exist, check the status of those services, and update the indexes of available services. Nonresponding services are marked for attempted repair via reconnection.