Hard Drive Failures

RustFS ensures read/write access can still be provided when some disks fail through mechanisms similar to erasure coding, and automatically heals data after replacing disks.

Table of Contents

1. Unmount Failed Disk
2. Replace Failed Disk
3. Update /etc/fstab or RustFS Configuration
4. Remount New Disk
5. Trigger and Monitor Data Healing
6. Follow-up Checks and Notes

Unmount Failed Disk

Before replacing the physical hard drive, you need to safely unmount the failed disk from the operating system level to avoid I/O errors in the file system or RustFS during the replacement process.

# Assume the failed disk is /dev/sdb
umount /dev/sdb

systemctl stop rustfs

Replace Failed Disk

After physically replacing the failed disk, you need to partition and format the new disk, and apply the same label as the original disk.

# Format as XFS and apply the same label as the original disk (RustFS requires XFS, like the other disks in the deployment)
mkfs.xfs -L DISK1 /dev/sdb

Requirements

• New disk capacity ≥ original disk capacity;
• File system type consistent with other disks (XFS, per the installation guide);
• Recommend using labels (LABEL) or UUID for mounting to ensure disk order is not affected by system restarts.

Update /etc/fstab or RustFS Configuration

Confirm that the mount item labels or UUIDs in /etc/fstab point to the new disk. The mount point must stay identical to the one listed in RUSTFS_VOLUMES (in /etc/default/rustfs), so the healed disk rejoins the cluster at the same path.

# View current fstab
cat /etc/fstab

# Example fstab entry (no modification needed if labels are the same)
LABEL=DISK1 /data/rustfs0 xfs defaults,noatime 0 2

blkid /dev/sdb
# Get the new partition's UUID, then replace the corresponding field in fstab

mount -a # If no errors, configuration is correct

Remount New Disk

Execute the following commands to batch mount all disks and start the RustFS service:

mount -a
systemctl start rustfs

Confirm all disks are mounted normally:

df -h | grep /data/rustfs

Trigger and Monitor Data Healing

Once RustFS detects a freshly formatted disk at a known mount point, its background scanner heals the missing data onto it automatically — no manual command is required. Confirm that recovery has started and track its progress through the service logs:

# For systemd-managed installations
journalctl -u rustfs -f

# Or view the log files under the directory set by RUSTFS_OBS_LOG_DIRECTORY
tail -f /var/logs/rustfs/rustfs.log

You can also open the RustFS Console and check the disk status of the affected node.

Follow-up Checks and Notes

1. Performance Monitoring

• I/O may fluctuate slightly during healing, recommend monitoring disk and network load.

2. Batch Failures

• If multiple failures occur in the same batch of disks, consider more frequent hardware inspections.

3. Regular Drills

• Regularly simulate disk failure drills to ensure team familiarity with recovery processes.

4. Maintenance Windows

• When failure rates are high, arrange dedicated maintenance windows to speed up replacement and healing.