(An admin removed and re-added search head cluster (SHC) members as part of patching the operating system. When trying to re-add the first member, a script reverted the SHC member to a previous…

Question

Options

• AReview splunkd.log for configuration changes preventing the addition of the member.
• BDelete the [shclustering] stanza in server.conf and restart Splunk.
• CForce the member add by running splunk edit shcluster-config --force.
• DClean the Raft metadata using splunk clean raft.

• A
  6% (3)
• B
  11% (5)
• C
  21% (10)
• D
  62% (29)

Explanation

According to the Splunk Search Head Clustering Troubleshooting Guide, when a Search Head Cluster (SHC) member is reverted from a backup or experiences configuration drift (e.g., an outdated Raft state), it can fail to rejoin the cluster due to inconsistent Raft metadata. The Raft database stores the SHC's internal consensus and replication state, including knowledge object synchronization, captain election history, and peer membership information. If this Raft metadata becomes corrupted or outdated (as in the scenario where a node is restored from backup), the recommended and Splunk-supported remediation is to clean the Raft metadata splunk clean raft This command resets the node's local Raft state so it can re-synchronize with the current SHC captain and rejoin the cluster cleanly. The steps generally are: Stop the affected SHC member. Run splunk clean raft on that node. Verify that it successfully rejoins the SHC. Deleting configuration stanzas or forcing re-addition (Options B and C) can lead to further inconsistency or data loss. Reviewing logs (Option A) helps diagnose issues but does not resolve Raft - Troubleshooting Raft Metadata Corruption in Search Head Clusters - splunk clean raft Command Reference - Search Head Clustering: Recovering from Backup and Membership Failures - Splunk Enterprise Admin Manual - Raft Consensus and SHC Maintenance

No community discussion yet for this question.