File systems organize data storage on a disk. Sometimes, file systems may be corrupted hence making them inaccessible. This may cause systems to fail boot or services to fail starting.

When a file system is corrupt, sometimes it changes its mode to Read-Only mode. This means it becomes impossible to perform any action on the file system e.g copy, move or remove data. Also, services e.g apache or any other service running on the file system will fail to start. This is because these services normally need to write events on a log file as they start and a system being in read only mode means files cannot be written on..

One circumstance this can occur is after replacing a defective disk on a RAID system. The new disk may have Inode inconsistency or other issues with its new RAID disk partner, leading to the above problems.

To fix these, you need to run a file system check. The steps below show how to achieve this

1.Boot your server into Rescue Mode.

2. Check partitions names to determine which one to fix.

root@rescue ~ # lsblk

3. Run the command below to check that all partitions have a valid partition table

root@rescue ~ # parted /dev/sda print all

Change /dev/sda with your partition name

If the output is something like this, it means the partition needs to be assigned a partition table.

root@rescue ~ # parted /dev/sda print all
Error: /dev/sda: unrecognised disk label
Model: ATA INTEL SSDSC2KW25 (scsi)
Disk /dev/sda: 256GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Model: ATA HGST HUS726020AL (scsi)
Disk /dev/sdb: 2000GB
*******more content below*******

4. Fix the error above by running the commands below

root@rescue ~ # parted /dev/sda

(parted) unit GB
(parted) mklabel msdos
(parted) mkpart primary 0 -1
(parted) quit

Now when you run the command in step three above, the output will be as follows

root@rescue ~ # parted /dev/sda print all
Model: ATA INTEL SSDSC2KW25 (scsi)
Disk /dev/sda: 256GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
1 1049kB 255GB 255GB primary

Model: ATA HGST HUS726020AL (scsi)
Disk /dev/sdb: 2000GB
Sector size (logical/physical): 512B/4096B
******more details below*******

5. Now, you can proceed to run a file system check on any partition that is in Read Only mode as follows

root@rescue ~ # fsck -fy /dev/md2

Replace /dev/md2 with your LVM or Partition Name

6. Once done, boot your server back to Normal Mode.

Automatic filesystem check

To perform automatic file system check you can run the following commands to configure

root@rescue ~ # tune2fs -c 3 /dev/md2 #[where 3 is the number of boot processes in between the checks]

root@rescue ~ # tune2fs -i 1m /dev/md2

You can repeat those two commands for all your partitions