The Test-and-Set instruction is defined by the following atomic (indivisible) sequence:

Most people see this error in the context of Raft log replication.

Raft requires strict persistence. To become the leader, a node must write a "no-op" entry to disk using a test-and-set to ensure no split-brain occurs.

If a node gets false for equality on its own disk block:

The log will say: "Cannot guarantee log consistency. Shutting down."

Symptom: A 4-node GlusterFS cluster began throwing “atomic test and set of disk block returned false for equality” errors after a power outage. Metadata operations hung, and thick provisioning failed.

Root cause: The power outage caused two nodes to believe they owned the same disk block region (split-brain). The DLM’s internal block version counter had reverted to 0 on one node after unclean shutdown.

Fix:

If a system is logging "Atomic Test and Set returned false" unexpectedly or excessively, the following issues should be investigated:

| Issue | Description | Review Recommendation | | :--- | :--- | :--- | | Lock Starvation | The thread holding the lock is taking too long (e.g., slow I/O, page fault). | Implement exponential backoff in the spin-loop or switch to a blocking semaphore if wait time exceeds a threshold. | | Deadlock | Thread A holds Lock X and waits for Lock Y; Thread B holds Y and returns false on X. | Review lock ordering policies. The false return is a symptom of a cyclical dependency. | | Forgotten Release | A thread acquired the lock but crashed or returned without releasing it. | The TS will return false indefinitely. Implement watchdog timers or recovery mechanisms to reset "stuck" locks. | | Priority Inversion | A high-priority thread spins on false returns, while the low-priority thread holding the lock is preempted and never runs. | Use priority inheritance protocols. |

Many "false for equality" errors stem from firmware bugs in:

Check vendor release notes for terms like "COMPARE AND WRITE," "atomic," or "reservation."

atomic test and set of disk block returned false for equality is not a software bug. It is a physics vs. logic error.

Your code expects the disk to obey causality (Write A happens before Read A). The disk decided to be a chaotic neutral trickster. When you see this error, stop debugging the database and start debugging your storage stack.

Have you seen this error in the wild? Drop a comment below with your hardware specs. I’ll bet it was an NVMe drive from 2018.

Understanding the "Atomic Test-and-Set of Disk Block Returned False for Equality" Error

In the world of distributed systems, high-availability clusters, and storage area networks (SANs), data integrity is the highest priority. One of the most cryptic yet significant errors a systems administrator or storage engineer might encounter is: "atomic test and set of disk block returned false for equality."

At its core, this message indicates a failure in a fundamental synchronization primitive used to prevent data corruption. When this fails, it usually means the system’s "source of truth" regarding who owns a piece of data has been compromised or contested. What is Atomic Test-and-Set (ATS)?

To understand the error, we first have to understand the mechanism. Atomic Test-and-Set is a hardware-offloaded locking mechanism (often part of the VAAI—vSphere Storage APIs for Array Integration—feature set in VMware environments).

In traditional storage, locking a file required "SCSI Reservations," which locked an entire LUN (Logical Unit Number). This was inefficient. ATS allows for discrete locking. Instead of locking the whole "parking lot," the system only locks a "single parking space" (a specific disk block). The process works like this:

Test: The host checks the current metadata of a disk block to see if it matches what it expects.

Set: If it matches (equality), the host updates the block with its own signature to claim ownership.

Atomic: This happens in a single, uninterruptible operation. Decoding the Error: "Returned False for Equality"

When the system reports that this operation "returned false for equality," it means the Test phase failed.

The host sent a command saying: "I want to lock this block. I expect the current owner ID to be 'X'." The storage array looked at the block, saw that the ID was actually 'Y', and replied: "False. The data is not what you expected." Common Causes

Why would the equality test fail? Usually, it's one of three scenarios: 1. "Split Brain" or Multi-Host Contention

The most common cause is that two different hosts are trying to access the same metadata at the exact same time. If Host A updates a block while Host B is still holding onto "old" information about that block, Host B’s next ATS command will fail because the block's state changed behind its back. 2. Storage Array Firmware Incompatibilities

Not all storage arrays implement VAAI/ATS the same way. If there is a bug in the array's microcode or if the host's driver is sending a malformed request, the array might reject the ATS heartbeat, leading to "false for equality" errors even if no real contention exists. 3. Network Latency and Heartbeating Issues

In clustered environments (like VMware VMFS datastores), hosts use ATS as a "heartbeat" to tell other hosts they are still alive. If the network between the host and the storage has high latency or dropped packets, the update might arrive late or out of sync, causing the "equality" check to fail because the host is working with stale metadata. Impact on Operations When this error occurs, you will typically notice:

Virtual Machines freezing: If the host cannot "set" the lock, it cannot write to the disk.

Datastore disconnects: The host may mark the storage as "All Paths Down" (APD) or "Permanent Device Loss" (PDL) to protect data integrity.

Log Spam: The VMkernel logs will fill with ATS Miscompare or Status: Op: 0x89 messages. How to Troubleshoot and Fix

Check Firmware and Drivers: Ensure your HBA (Host Bus Adapter) drivers and the storage array firmware are on the vendor's "Compatibility Matrix."

Review Storage Latency: Look for spikes in command latency. ATS is very sensitive to timing; if the storage is overloaded, ATS failures will increase.

Disable ATS Heartbeating (Last Resort): In some specific storage environments (notably certain older NAS or SAN setups), the ATS heartbeating mechanism is too aggressive. VMware allows you to revert to traditional SCSI reservations for heartbeating while keeping ATS for other tasks, though this should only be done under the guidance of support.

Verify VAAI Support: Use command-line tools (like esxcli storage core device vaai status get) to ensure the array is actually reporting ATS as "supported." Conclusion

The "atomic test and set of disk block returned false for equality" error is a protective measure. While it causes disruptive downtime, it exists to prevent the "silent killer" of enterprise computing: data corruption. By failing the operation when the state doesn't match, the system ensures that two hosts never write to the same block simultaneously, preserving the integrity of your databases and virtual machines.

Report: Atomic Test and Set of Disk Block Returned False for Equality

Introduction

The following report documents an issue encountered during a recent testing phase, where an atomic test and set operation on a disk block returned an unexpected result, indicating that the block's contents were not equal as anticipated.

Test Environment

Test Description

The test in question involved performing an atomic test and set operation on a disk block. This operation typically checks the current value of a disk block and, if it matches a specified expected value, atomically sets it to a new value. The goal was to verify the integrity and consistency of disk operations under various conditions.

Observed Issue

During the execution of the test:

Analysis

The return of false for equality during an atomic test and set operation on a disk block suggests that:

Steps to Reproduce

  • Expected Result: The operation should return true, indicating the block's value matched the expected value before being updated.

    • Recommendations

    Conclusion

    The observation that an atomic test and set operation on a disk block returned false for equality highlights a potential issue with data consistency or concurrent access. Further investigation and debugging are necessary to resolve the root cause and ensure the reliability of disk operations.

    Action Plan

    Responsibilities

    Timeline

    Status Update

    This report will be updated with findings from the investigation and any corrective actions taken.

    "Atomic test and set of disk block returned false for equality" is a low-level status message typically found in VMware ESXi VMkernel logs It indicates a failure in the Atomic Test and Set (ATS) , which is part of the vStorage APIs for Array Integration (VAAI) Core Concept: What is ATS?

    Atomic Test and Set (ATS) is a hardware-assisted locking method used by ESXi to manage metadata updates on shared storage (VMFS datastores). WordPress.com Traditional Method

    : Used SCSI reservations to lock an entire LUN (Logical Unit Number), preventing other hosts from accessing it entirely during updates. ATS Method

    : Locks only specific disk blocks (sectors) rather than the whole LUN. This allows multiple hosts to perform metadata operations simultaneously on the same LUN, significantly improving performance and scalability. Hitachi Vantara Community Meaning of the "False for Equality" Error

    The error occurs when the ESXi host attempts to update a block but finds that the existing data on that block does not match what it expected (the "test" part of "test and set" failed). This typically signifies a lock contention mismatch in state between the host and the storage array. Broadcom support portal Common Causes Performance issues with VM operations

    T##:##Z cpu2:#######)ScsiDeviceIO: 4167: Cmd(0x45d90f0d4e48) 0x89, CmdSN 0x2163b3 from world 2101333 to dev "naa..################ Broadcom support portal

    vSphere connection to datastore error Atomic test : r/vmware