Timeline: Mar-Apr 2022
In this project, a friend and I created a fault-tolerant distributed graph processing engine based on Google’s Pregel API.
Our project:
- is written in Go
- is tested on a 75 MB graph with 875,713 nodes and 5,105,039 edges
- uses MongoDB and SQLite for storage
- is containerized using Docker
We implemented:
- a failure recovery protocol that uses master-slave replication
- a custom heartbeat-ack protocol library to monitor node failures
- RPC communication among the cluster of nodes
- a gRPC client that uses an Envoy proxy, built with React & TypeScript
