At Hootsuite, we process millions of social messages every day. We strive to keep our service stable and secure, and rely on open source projects such as Scala and Akka to achieve our goals. We are excited to contribute back to the community today with a Redis plugin for Akka Persistence.
So what is Akka and how is it useful? Akka provides means to make a distributed system scalable, asynchronous, and fault-tolerant. It achieves this by sending messages among lightweight processes called actors. These actors only communicate through Akka’s message delivery mechanism such as mailboxes and certain level of delivery guarantee. This allows developers to concentrate on actor behaviours, and not on message delivery. This is certainly the use case at Hootsuite. Most of the time, we focus on defining actors and trust Akka to deliver those millions of messages.
Akka Cluster, Sharding, and Persistence
The Akka that we all love and use is a lot more than just actors. It provides scalability and resiliency with Akka Cluster, Sharding, and Persistence. Akka Cluster enables a cluster of Akka instances and membership management. Usage of vector clocks, failure detection, and gossip protocol helps in forming and maintaining the cluster.
Cluster Sharding enables location transparency of actors within the cluster. It assigns a shard and relays messages based on a unique identifier for each actor. Neat thing is that Akka takes care of rebalancing of shards and queuing messages when cluster membership changes. Under the hood, there is an actor named ShardCoordinator that manages and persists the shard state. Where is it stored? Who stores it? The answer is in Akka Persistence.
More on Akka Persistence
Akka Persistence provides a stateful actor that can persist state in an event sourced fashion. Storage is pluggable, so theoretically any data store can be used. Plugins for popular databases such as Cassandra, Kafka, or SQL can be found at Akka Community Projects.
Akka Persistence powers ShardCoordinator, and the application doesn’t stop there. It can be used without Akka Cluster and is certainly useful when dealing with a stateful actor that needs to recover from failure. Examples include event sourcing and CQRS.
Our Use Case
Our experiment evolved around notifications that are associated with an event within Hootsuite. Each notification is connected to a user and available for few days. Our application was to spawn an actor per user to transform events to notifications, persist them for durability, and expire old notifications. Actors would be distributed with Akka Cluster and Sharding.
When it comes to database, there is no shortage of options. The road to Akka Persistence is well-paved with many open-sourced plugins from Akka Community. At the time of our experiment, we were toying with the idea of ephemeral notifications with expiry, or Time To Live, of few days. Redis was one that came to our mind since we have used Redis in other projects. We fell in love with it for its native support for data structures, TTL, and fast performance. It was also easy to setup and query for data.
The major caveat of this solution was the lack of a Redis plugin for Akka Persistence. We could have resorted to other database solutions but also wanted to have a single data store during our experiment. We decided to create a solution internally by developing a new plugin for this functionality. Akka documentation, code comments, and existing plugins for other data stores were excellent guides to follow. Paired with Technical Compatibility Kit from the Akka team, our plugin has been tested and validated throughout development. We are not currently using the plugin on production, but it played an important part in our experiment on Akka Cluster and Sharding.
Where to Find and How to Contribute
- The plugin is on GitHub: akka-persistence-redis.
- Outstanding issues and suggestions are recorded in Issues.
- We would love to hear about ideas for making this plugin better, so please record them in Issues.
About the Author
Steve is a Software Engineer on the Platform Team at Hootsuite. When not working on writing distributed and scalable microservices in Scala, he likes to explore beautiful Vancouver, read books, and discover technology. Follow Steve on Twitter @ssongvan.