Pekko Cluster Implementation in a Microservice

[joesan]

I need some help around implementing a Pekko cluster in my Microservice. The set up is like this:

1. I have a Microservice A that runs as a replica set in a Kubernetes cluster
2. This Microservice holds the life cycle state of a physical asset (Asset START, PENDING, ACTIVE, REJECTED... etc.,)
3. I have implemented an Actor to mutate over this state based on the messages that I get from these physical assets
4. This is all good if I run my Microservice A as a single instance, but as a replica set, it would mean that I need to have the same actor instance replicated across all 3 instances of my Microservice A

My questions:

1. How can I leverage Pekko Cluster to help with my situation where a message for a physical asset comes in, it gets processed by any one instance (Node 1) of my Microservice A and the actor state is mutated accordingly based in the message
2. The mutated actor state is replicated to all the other nodes of this Microservice A, so that if the next request comes and is handled by Microservice A (running in Node 3 or Node 3), it has the same state as Node 1
3. Is there anything else that I should consider? Like for example Node 3 going down and being re-started in another Node. How will this new instance catch up with the actor state from the other 2 nodes?

I'm looking for some help to accomplish this with Pekko cluster. Of-course I could use an external cache like Redis, but I would like to avoid additional infrastructure and see if I can build this into my Microservice itself.

[pjfanning]

• You could have a singleton remote actor instance that could be accessed by any actor instances in your cluster.
• I would suggest that persisting the state using Pekko Persistence or some other mechanism is probably how I would approach a problem like this though

[joesan]

@pjfanning Thanks for the suggestion. Are there any examples of implementation of a Microservice using Pekko cluster, deployed to Kubernetes? I see one from the pekko-examples. but I'm looking for even more detailed examples if available?

[Susmit07]

We are just now trying to solve the exact problem.

There are multiple ways you can solve

• Implement Offset Persistence Using Pekko Persistence. You can read this
  https://pekko.apache.org/docs/pekko-persistence-r2dbc/current/durable-state-store.html
• Implement Offset Persistence Using PVC (persistent k8S volume to store the state)
• Use S3 Object Marker Files for Tracking or Redis or a KV store like Dynamo. S3 are cheap and extremely durable but there will be a slight added latency

Would wait to hear from you, how you solved the issue