engineer3000 - Mon, 2018/08/13 - 14:38

Hello- I have TKL 14.1 running in hardware and its worked well for several years. When I did my monthly image backup, it took over 1 hour when normally it takes 5 minutes. Turns out there is about 220 GB extra on the disk somewhere. A normal file system search does not show where its coming from, except it looks like its from a virtual disk at /proc/kcore which as I understand is swap space and shouldn't be taking up physical space, but it does. The result of df -h is below and you can see 249 GB in /dev/dm-0 (not sure where that is). I did many forum searches and there does not to be any clear answer, except that perhaps the root partition needs to be enlarged (with GParted) but that seems drastic and risky. Any insight would be appreciated.

 

df -h

Filesystem      Size  Used Avail Use% Mounted on

/dev/dm-0       1.6T  249G  1.3T  17% /

udev             10M     0   10M   0% /dev

tmpfs           6.3G  8.5M  6.3G   1% /run

tmpfs            16G     0   16G   0% /dev/shm

tmpfs           5.0M     0  5.0M   0% /run/lock

tmpfs            16G     0   16G   0% /sys/fs/cgroup

/dev/sda1       236M   66M  158M  30% /boot

/dev/sdb1       1.8T  269G  1.5T  16% /mnt/backup01

 

 

 

Forum: 
Support
Jeremy Davis's picture

Jeremy Davis - Tue, 2018/08/14 - 02:15

FWIW /proc/kcore (and all the files in /proc) are not actually files at all. /proc/kcore is essentially a representation of your virtual memory contents. Have a look here for a bit more info. Also, you shouldn't be including /proc (or /sys or /dev) in your backups anyway!

I'd suggest that you install ncdu. IMO it's the best way to drill down and find what is taking up all your space. Install like this:

apt-get update && apt-get install ncdu

Then launch it like this:

ncdu /

It will take a while to scan your disk, but once it's done, it will display the filesyste in directory size order. You can enter the directories and see the sub directories (again ordered by size), and so on...

I would suggest that you have a recent backup before doing anything (just in case).

Anything in /usr (with the exception of sub directories of /usr/local) should NOT be manually deleted (unless you are really sure what you are doing). Those files are managed by package management. So instead of manually deleting, work out which package the files come from, and remove that via apt-get remove PACKAGE_NAME.

You should also be careful with files in /var/lib, as they are local to your system and likely contain important/useful data. As the name suggests, /var/log are log files and assuming that logrotate is configured right, shouldn't be too bad (and should be being auto cleaned). As a general rule, anything in /var/cache can be safely deleted, but YMMV (hence the backup). Also it will possibly be recreated, so may not be a long term fix.

/etc generally contains configuration files, although sometimes cruft does accumulate there. Be careful deleting files from there. One place that may accumulate cruft particularly is the /etc/.git (hidden) directory. That is where all the versioned data from /etc is kept (etckeeper uses a git repo to store config changes).

Anyway, have a poke around and see how you go. If you want any further advice, please ask and I'll do my best to head you in the right direction.

engineer3000 - Tue, 2018/08/14 - 10:48

Thank you Jeremy. I have previously scanned my disk, but installed ncdu as you suggested to double check (very nice program, thanks for that). The scan showed only 13 GB total for the whole disk, but my backup was over 220 GB. Not sure where that extra data is coming from (/dev/dm-0 perhaps?). My backup program (Clonezilla) is a block for block image backup, so there is no way to exclude anything. There is something on the disk which is the source of all this extra data. 

Steve

Jeremy Davis's picture

Jeremy Davis - Fri, 2018/08/17 - 00:19

First up, apologies on slow response. I thought I'd already replied, but obviously not...

Ah ok, Clonezilla... TBH, I haven't used it for years, but I always found it quite slow. I have just had a quick read and it seems that it's progressed quite a bit since I last used it. I forget what filesystem I used, but it did a "sector-by-sector" copy, so would create backups the same size as the drive capacity. It seems it's a bit smarter now and only copies used blocks. Assuming that the system isn't running when it does that, you won't be getting /dev, /proc or /sys (they're "virtual filesystems" and are generated at boot) so that's something...

Re-reading your thread again, I think I originally missed your point. I think Clonezilla was the missing piece for me! Basically it looks like both df and Clonezilla, think that there is about ~220GB extra space being used in /dev/dm-0 than what there really is.

So what sort of filesystem is /dev/dm-0? If it's LVM, then perhaps you have some snapshot data which you are unaware of? That could perhaps explain it, although TBH, I wouldn't have expected df to be able to see snapshot data, only the data that is visible within the filesystem (i.e. same as ncdu)?!

So I'm a bit perplexed.

engineer3000 - Fri, 2018/08/17 - 12:10

Hi Jeremy,

Appreciate your response. Clonezilla doesn't allow you to explore the cloned image directly, except with a very complicated workaround. Meanwhile, I found the Clonezilla log and compared it to another server running TKL where the same backup worked normally. The relevant lines in the log are compared below. The only difference I see is that /dev/turnkey/root in the problem system is 294.6 GB compared to 48.7 GB in the normal system. But ncdu shows only 14 GB of use in the problem system, so I'm perplexed as well! Is /dev/turnkey/root a partition under LVM? Can it have something hidden in it? That is my main question.

Normal Backup:

Starting to clone device (/dev/turnkey/root) to image (-)
Reading Super Block
memory needed: 47912324 bytes
bitmap 26940800 bytes, blocks 2*10485760 bytes, checksum 4 bytes
Calculating bitmap... Please wait... 
done!
File system:  EXTFS
Device size:  882.8 GB = 215526400 Blocks
Space in use:  48.7 GB = 11880147 Blocks
Free Space:   834.1 GB = 203646253 Blocks
Block size:   4096 Byte
Total block 215526400
Syncing... OK!
Partclone successfully cloned the device (/dev/turnkey/root) to the image (-)
>>> Time elapsed: 815.62 secs (~ 13.593 mins)

Problem backup:

Starting to clone device (/dev/turnkey/root) to image (-)
Reading Super Block
memory needed: 74859652 bytes
bitmap 53888128 bytes, blocks 2*10485760 bytes, checksum 4 bytes
Calculating bitmap... Please wait... 
done!
File system:  EXTFS
Device size:    1.8 TB = 431105024 Blocks
Space in use: 294.6 GB = 71932944 Blocks
Free Space:     1.5 TB = 359172080 Blocks
Block size:   4096 Byte
Total block 431105024
Syncing... OK!
Partclone successfully cloned the device (/dev/turnkey/root) to the image (-)
>>> Time elapsed: 4151.05 secs (~ 69.184 mins)

 

 

Jeremy Davis's picture

Jeremy Davis - Tue, 2018/08/21 - 02:45

I just had a quick read and yeah it sounds like a bit of a pain to mount Clonezilla images. In theory it's relatively easy, at least on Linux (as mounting most images are on Linux). However, because it's stored as a split archive, the steps required to get it to a point of being able to mount make it a bit of a pain...

Other than confirming that Clonezilla is copying "data" which isn't required (and may or may not be real data) the logs don't really give much additional insight...

engineer3000 - Fri, 2018/08/17 - 14:37

Not sure how to interpret this, but on the problem system turnkey-root and turnkey-swap have become dm-0 and dm-1 respectively, whereas on the normal system they are just normal.

Running "lsblk" on the Normal system shows this:

NAME               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                  8:0    0  1.8T  0 disk 
|-sda1               8:1    0  243M  0 part /boot
|-sda2               8:2    0    1K  0 part 
`-sda5               8:5    0  1.8T  0 part 
  |-turnkey-root   254:0    0  1.6T  0 lvm  /
  `-turnkey-swap_1 254:1    0   32G  0 lvm  [SWAP]
sdb                  8:16   0  1.8T  0 disk 
`-sdb1               8:17   0  1.8T  0 part /mnt/backup01
sr0                 11:0    1 1024M  0 rom  

lsblk on the Problem system shows this:

NAME                      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdb                         8:16   0   1.8T  0 disk 
`-sdb1                      8:17   0   1.8T  0 part /mnt/backup
sda                         8:0    0   1.8T  0 disk 
|-sda1                      8:1    0   243M  0 part /boot
|-sda2                      8:2    0     1K  0 part 
`-sda5                      8:5    0 931.3G  0 part 
  |-turnkey-root (dm-0)   254:0    0 822.2G  0 lvm  /
  `-turnkey-swap_1 (dm-1) 254:1    0    16G  0 lvm  [SWAP]
sr0                        11:0    1  1024M  0 rom  

 

Jeremy Davis's picture

Jeremy Davis - Tue, 2018/08/21 - 02:47

But I'm not sure whether this is really related, or not. A bit of googling suggests that /dev/dm-0 etc are just the underlaying device names which LVM uses. Apparently they're all /dev/dm-0 etc under the hood. I suspect that the fact that one system displays them obviously and another doesn't, isn't of any real significance (perhaps slightly different versions of lsblk, or slightly different kernel version, etc). Although I don't know enough about this to be 100% sure...

I'd probably try exploring the LVM snapshot angle I mentioned earlier (although I'm not sure if that's of any value as I don't know enough about Clonezilla). Perhaps it might become clear using tools such as lvs and/or lvdisplay? Otherwise, I guess you could try zeroing out the free space? Perhaps it's caused by deleted data, although TBH, that seems unlikely to me...

Personally, I'd try reaching out to the Clonezilla community. My guess is that they'd be much more likely to be able to help you work out what's going on. I suspect that this would be something they've come across before. Under the hood, TurnKey is a customized Debian which uses LVM, so not particularly unique.

If the LVM tools demonstrate my guess is wrong, then I'd suggest either posting on one of the Clonezilla mailing lists (probably clonezilla-user?) or perhaps you might get some useful suggestions on 'Nix StackExchange?

engineer3000 - Tue, 2018/08/21 - 10:52

Yes, already posted on Clonezilla forum a week ago, got 96 views but no replies. Will look into lvs and lvdisplay. Thanks for your help

Jeremy Davis's picture

Jeremy Davis - Wed, 2018/08/22 - 02:36

The SourceForge forums aren't the same as the mailing lists. It doesn't seem like there is much traffic on the mailing lists, but comparing the SourceForge forums and the mailing list traffic, my guess is that there aren't any developers posting on the forums, just users. I'm not sure, but perhaps it's worth cross posting on one of the mailing lists? Perhaps it might also be worth cross posting on Unix StackExchange too?

Good luck with lvs/lvdisplay - hopefully they give some insight?

Failing all that (or while you wait) untaring the (oversized) archive and mounting it so you can explore it might be worth a try. At least then you may be able to see what the additional size is being taken up with.

On a bit of a tangent, some further research suggests that under the covers, the software that Clonezilla uses to create the backups is called partclone. Perhaps it's worth trying to use that software directly. If you can recreate the issue under those circumstance, perhaps the developer would be able to assist troubleshoot?

Add new comment