Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Promote a slave to master when the master goes down in replication mode #775

Open
C-Ravina opened this issue Feb 14, 2024 · 7 comments
Open
Labels
enhancement New feature or request

Comments

@C-Ravina
Copy link

C-Ravina commented Feb 14, 2024

In replication mode, when master goes down write operations couldn't be performed as slaves are read-only. So, when a slave is promoted to master in case of master failure then write operations won't be restricted. So, it would be nice, if code changes are made to promote a slave to master in case master goes down.

@C-Ravina C-Ravina added the enhancement New feature or request label Feb 14, 2024
@kannanvr
Copy link

kannanvr commented Feb 15, 2024

Its a good feature addition on OT Redis. shall we implement this feature?

When there is a problem on master redis. We need to promote slave as Master. This will really help in all of our use case.

@C-Ravina
Copy link
Author

HI @drivebyer ,

I think it could be a valuable addition to the project.
Can we implement this feature?

@drivebyer
Copy link
Collaborator

@C-Ravina Thank you for your feedback. If I understand your question correctly, I believe the feature you're referring to is already supported in this operator; it's the replication combined with sentinel mode. A sentinel cluster monitors the replication cluster and, in the event that the master goes down, the sentinel will promote a replica to master, thus ensuring read/write operations continuity.

@C-Ravina
Copy link
Author

Hi @drivebyer ,
Thank you for your response. But why should it be like replication combined with sentinel mode and not simply replication with the ability to promote slave to master when master fails?
Also, when I tried what you said earlier, I am able to set and get keys through the sentinel client (using sentinel service IP) and also through Redis client (using replication service IP). Is this behavior intended and why so?

@drivebyer
Copy link
Collaborator

Thank you for your response. But why should it be like replication combined with sentinel mode and not simply replication with the ability to promote slave to master when master fails?

Providing high availability in replication mode initially would be more user-friendly.

Also, when I tried what you said earlier, I am able to set and get keys through the sentinel client (using sentinel service IP) and also through Redis client (using replication service IP). Is this behavior intended and why so?

I'm not sure which client you are using. When one uses go-redis, a sentinel client can be created with NewFailoverClient. This client issues the SENTINEL GET-MASTER-ADDR-BY-NAME command to retrieve the master's address, enabling it to execute set/get commands to the master.

@RavinaChidambaram
Copy link

Hi @drivebyer ,

Providing high availability in replication mode initially would be more user-friendly.

So, can we incorporate the change to enhance replication mode by enabling automatic promotion of a slave to master when the current master fails?
I have tested this approach, and it works well—during each reconcile, the system checks for master availability, and if it’s unreachable, a slave is promoted to master , and the other slaves are reconfigured to follow the new master. This ensures continuous write operations without interruptions. If this sounds good, I can raise a PR with these changes.

@drivebyer
Copy link
Collaborator

@RavinaChidambaram
After careful consideration, I don't think we should implement sentinel functionality (failover) directly in the redis-operator code. This would essentially be reinventing the wheel.

Redis Sentinel was specifically designed to handle high availability and automatic failover for Redis deployments. By implementing this functionality ourselves within the operator, we would be duplicating existing, well-tested logic that's already part of the Redis ecosystem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants