From grlug at grlug.org  Fri Mar 20 11:51:56 2020
From: grlug at grlug.org (Grand Rapids Linux Users Group)
Date: Fri, 20 Mar 2020 10:51:56 -0500 (CDT)
Subject: [GRLUG] XFS on Softraid
Message-ID: <mailman.31669.1584719528.100365.grlug@grlug.org>

----- Message Text -----
Looking a weird problem with a 7T/RAID6/6 disk softraid: Over the past few
months, there have been nine occasions when the last rsnapshot has not
been able to delete due to a dirty filesystem.

This has occurred a few times over the years (that array is about five
years old), but rebooting and repairing always has fixed the problem, but,
unfortunately, after some time it occurrs again.

The weird part is that on this 7T volume, **ALL** of the issues have
occurred with ONE snapshot set, which seems very unlikely to me with 20
active sets.

So, .. my question is:

How likely are nine different file system problem in the SAME snapshot
directory with 20 active snapshots?

        Thanks!

From grlug at grlug.org  Fri Mar 20 12:05:19 2020
From: grlug at grlug.org (Grand Rapids Linux Users Group)
Date: Fri, 20 Mar 2020 12:05:19 -0400
Subject: [GRLUG] XFS on Softraid
In-Reply-To: <mailman.31669.1584719528.100365.grlug@grlug.org>
References: <mailman.31669.1584719528.100365.grlug@grlug.org>
Message-ID: <mailman.31673.1584720323.100365.grlug@grlug.org>

I am not familiar with rsnapshot. But if this an `md` of mdadm software raid, you might be able to isolate and track down the problem with
 
./mdadm --monitor —mail=you at yourhost --syslog --delay=300 /dev/md0 --daemonise

where md0 is your container


cheers

-j





> On Mar 20, 2020, at 11:51 AM, Grand Rapids Linux Users Group <grlug at grlug.org> wrote:
> 
> ----- Message Text -----
> Looking a weird problem with a 7T/RAID6/6 disk softraid: Over the past few
> months, there have been nine occasions when the last rsnapshot has not
> been able to delete due to a dirty filesystem.
> 
> This has occurred a few times over the years (that array is about five
> years old), but rebooting and repairing always has fixed the problem, but,
> unfortunately, after some time it occurrs again.
> 
> The weird part is that on this 7T volume, **ALL** of the issues have
> occurred with ONE snapshot set, which seems very unlikely to me with 20
> active sets.
> 
> So, .. my question is:
> 
> How likely are nine different file system problem in the SAME snapshot
> directory with 20 active snapshots?
> 
>        Thanks!
> -- 
> grlug mailing list
> grlug at grlug.org
> https://shinobu.grlug.org/mailman/listinfo/grlug


From grlug at grlug.org  Fri Mar 20 12:20:43 2020
From: grlug at grlug.org (Grand Rapids Linux Users Group)
Date: Fri, 20 Mar 2020 11:20:43 -0500 (CDT)
Subject: [GRLUG] XFS on Softraid
In-Reply-To: <mailman.31673.1584720323.100365.grlug@grlug.org>
References: <mailman.31669.1584719528.100365.grlug@grlug.org>
 <mailman.31673.1584720323.100365.grlug@grlug.org>
Message-ID: <mailman.31677.1584721251.100365.grlug@grlug.org>

On Fri, 20 Mar 2020, Grand Rapids Linux Users Group wrote:

> I am not familiar with rsnapshot. But if this an `md` of mdadm software
> raid, you might be able to isolate and track down the problem with
> ./mdadm --monitor —mail=you at yourhost --syslog --delay=300 /dev/md0
> --daemonise where md0 is your container
>
Thanks for the idea, .. but nothing in journalctl anywhere near the dates
of the problem.

Don't suppose there might be a different way to look for problems with
systemd?

	Lee

From grlug at grlug.org  Fri Mar 20 12:32:12 2020
From: grlug at grlug.org (Grand Rapids Linux Users Group)
Date: Fri, 20 Mar 2020 12:32:12 -0400
Subject: [GRLUG] XFS on Softraid
In-Reply-To: <mailman.31677.1584721251.100365.grlug@grlug.org>
References: <mailman.31669.1584719528.100365.grlug@grlug.org>
 <mailman.31673.1584720323.100365.grlug@grlug.org>
 <mailman.31677.1584721251.100365.grlug@grlug.org>
Message-ID: <mailman.31678.1584721936.100365.grlug@grlug.org>

presumably dmesg would show some hieroglyphics if this were an underlying hardware thing.

Is there any chance you’re reading files in /proc|/run|/dev or any which may have funky locks? if so, maybe those file attributes are causing it. If it was related to a specific set, so maybe start with where it diffs from the good ones.


-j





> On Mar 20, 2020, at 12:20 PM, Grand Rapids Linux Users Group <grlug at grlug.org> wrote:
> 
> On Fri, 20 Mar 2020, Grand Rapids Linux Users Group wrote:
> 
>> I am not familiar with rsnapshot. But if this an `md` of mdadm software
>> raid, you might be able to isolate and track down the problem with
>> ./mdadm --monitor —mail=you at yourhost --syslog --delay=300 /dev/md0
>> --daemonise where md0 is your container
>> 
> Thanks for the idea, .. but nothing in journalctl anywhere near the dates
> of the problem.
> 
> Don't suppose there might be a different way to look for problems with
> systemd?
> 
> 	Lee
> -- 
> grlug mailing list
> grlug at grlug.org
> https://shinobu.grlug.org/mailman/listinfo/grlug

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://shinobu.grlug.org/pipermail/grlug/attachments/20200320/b187964c/attachment.html>