dump.exfat should do more than just disk/volume information

[hgkamath]

dump.exfat tool should do more than just disk/volume information.

The advantages of exfat filesystem are that (A) it is simple, (B) has huge maxsize limit and (C) has a bit for indicating contiguous-ness allowing for read optimization (D) absence of journaling is better for lifespan of SSD.
The simplicity is a trade off against not having a journal and disk failure tolerance, robustness.

A use case of exfat filesystem is to store large static container files like virtual-disk images, that are seldom resized/moved. They may also be defragmented in the case of hdd (not ssd) as contiguous files.
More importantly, knowing where the used file blocks are helps in case of disaster recovery.

dump.exfat /dev/sdx|/mnt/exfat <arguments> . Following arguments are being suggested
-s in formatted text show only information from superblock(s)
-u in formatted text show comma separated (or line separated) offset ranges of used sectors in increasing order and the file to which the sector range belongs to specified in bytes/blocks/clusters.
-f <filepath> in formatted text show comma separated (or line separated) offset ranges of bytes/blocks/clusters associated with a file
-R <dirpath> in formatted text recursively descend a directory show pathname and comma separated (or line separated) offset ranges of bytes/blocks/clusters associated with contained files (this functionality may also be simulated by using a shell script to loop over output of find command and use -f argument on each file)
-o <bkup_tableflname> in a binary stream output/dump the FAT sectors to a offline backup file

The dump information saved as a periodical (nightly) backup ahead of any disaster, so that in the event disaster happens, it may be used for disaster recovery.
The formatted text dump information will be helpful while using a command like dd during disaster recovery .

dd if=/bad/partition_blockdev of=/good/fs/opath/frag1 bs=512 offset=off1 count=len1 
dd if=/bad/partition_blockdev of=/good/fs/opath/frag2 bs=512 offset=off2 count=len2  
:
dd if=/bad/partition_blockdev of=/good/fs/opath/fragN bs=512 offset=offN count=lenN
dd if=/bad/partition_blockdev of=/good/fs/opath/fragNb bs=1 offset=$((512*(offN+lenN))) count=lenbytes  
cd /good/fs/opath/
cat frag1 frag2 ... fragN fragNb > reconstructed_file

One may then append the fragments / concatenate them together to recover the original file on a new filesystem
Such metadata can be saved on a different disk/partition and will be helpful during disaster recovery.

The following also requires a option parameter feature addition to the mount -t exfat command
Saving a binary dump of the enitire FAT would be used the following way.
dump.exfat /dev/sdx|/mnt/exfat -o <bkup_tableflname>
This could then be used in conjunction with mount.exfat
mount.exfat /dev/sdx /mnt/tmpexfat -o ro,fat=<bkup_tableflname>
This allows using regular cp/dd/ddrescue/filecopy commands to copy data off the corrupted FAT/partition using the offline backup table.

[namjaejeon]

Okay, I can not understand fully what you want to do. Can you implement it and show the execution log and usage at real ?

[hgkamath]

Essentially, dump tool for kernel-driver based exfatprogs should give information similar to the fuse-driver based exfat-utils
Please compare
https://manpages.debian.org/experimental/exfatprogs/dump.exfat.8.en.html
https://manpages.debian.org/unstable/exfat-utils/dumpexfat.8.en.html
Unfortunately, I do not presently code.
Currently, on my system with 2Tb drives, I have many small 100G partitions. I am fearful of file-system crashes. Small partitions can be repaired using fsck in reasonable, lesser amount of time. Corruption in one partition does not affect filesystem in other partitions. The problem with this partitioning approach is the disk space is badly utilized. As the partition fills up, the total free-space of the disk is fragmented within a paritition and between partitions. The rigid space-limits of each partition makes it difficult create large virtual-disks and move them around.
I was investigating the idea of combining the partitions back to one large exfat partition, and put Virtual-disk-images that are dynamic sized and have max size 100G. The virtual disks have their own file-systems on them, which can be mounted. Usually the disks are much smaller than the max size. Hence, the combined free-space is larger and allows for easier manipulation of large virtual-disks.
In this regard, I considered how I would do disaster recovery. If the large exfat filesystem experienced degradation/corruption, then dump information saved previously from tools like dump.exfat may help.

One should be able recover a file by knowing the every file's FAT-CHAIN of bytes/sectors/clusters.

misc:

• hdparm --fibmap </path/to/file> slow; brute forces its way through the file to determine the file's absolute sectors on disk.
• nfi.exe - Windows NT File System (NTFS) File Sector Information Utility
• ntfscluster /dev/sda17 -F /test1.img
• fsutil file layout path\to\file
• contig -a
• filefrag -v /opt/tmp/test1.bin notice that filefrag for btrfs is faster and more informative than exfat

[namjaejeon]

fsck is ready on exfat-next branch. Please check whether it is covered for your request.

git clone --branch=exfat-next https://github.com/exfatprogs/exfatprogs

[hgkamath]

check whether it is covered for your request.

Not really, the RFE/issue was regarding for precaution-sake being able to get the file-fragment info dumps, in case of needing to do manual-scraping/recovery before even attempting fs-repair. Online fs-repair tools irreversibly modify on-disk data. As a user, one can only say "Y" to whatever confirmation fsck asks. The fsck may attempt to rebuild FAT as well find/reallocate/relocate sectors. When data in volume is too important, one dd-s the whole corrupted volume into a volume-image on larger fresh disk, makes a duplicate copy, and attempts fs-repair on the copy.
Pre-saved FAT file-fragment info dumps is one contingency option. ExFAT being a fs that is descendant of FAT technology, does not have the same corruption robustness as other large filesystems (ntfs, btrfs, bcachefs, etc). This contingency option is useful for an ExFAT volume, which has a few large changing virtual-disk-files that rarely every grow or change-size, and so for long periods of time, the virtual-disk-files can be expected to always occupy the same sector-locations on host-exfat-volume, unless defragged/moved/copied etc. When a virtual machine writes to the virtual-disk-file, it is expected to only overwrite already-allocated sectors. This option allows for manual sector by sector disk-scraping as was done on a bad harddisk in the old days.

Nevertheless it is good to know exfat fsck is ready. Good-work and thankyou for your contributions!

As of now, my filesystems/hdds are fine, and I don't have any corruption to test against, and I hope they remain that way.

[root@fedora gana]# rpm -qa | grep exfat
exfatprogs-1.1.3-2.fc36.x86_64

koji builds for fc37 are at exfatprogs-1.1.3.
https://koji.fedoraproject.org/koji/packageinfo?packageID=32895

[namjaejeon]

@hgkamath We have released exfat 1.2.0 version. Please check it.

[namjaejeon]

@hgkamath Ping?

[hgkamath]

• switched back to linux now, am checking now
• info

  root@localhost:~# uname -a
  Linux sirius 6.13.5-200.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Feb 27 15:07:31 UTC 2025 x86_64 GNU/Linux
  root@localhost:~# cat /etc/os-release | grep PRETTY
  PRETTY_NAME="Fedora Linux 41 (Workstation Edition)"
  root@localhost:~# rpm -qa | grep exfatprogs
  exfatprogs-1.2.8-1.fc41.x86_64
  

• dump.exfat

  root@localhost:~# dump.exfat -h
  Usage: dump.exfat
          -V | --version                        Show version
          -h | --help                           Show help
  

• exfat partition

  root@localhost:~# sgdisk /dev/sda -p | grep WINPAGE
     3       138217472       171902975   16.1 GiB    0700  WINPAGE
  

• dump.exfat output

  root@sirius:~# dump.exfat /dev/sda3
  exfatprogs version : 1.2.8
  -------------- Dump Boot sector region --------------
  Volume Length(sectors):                 33685504
  FAT Offset(sector offset):              2048
  FAT Length(sectors):                    4160
  Cluster Heap Offset (sector offset):    8192
  Cluster Count:                          526208
  Root Cluster (cluster offset):          6
  Volume Serial:                          0x7f97f433
  Bytes per Sector:                       512
  Sectors per Cluster:                    64
  
  ----------------- Dump Root entries -----------------
  Volume label entry position:            0x420000
  Volume label character count:           7
  Volume label:                           WINPAGE
  Upcase table entry position:            0x420040
  Upcase table start cluster:             5
  Upcase table size:                      5836
  Bitmap entry position:                  0x420020
  Bitmap start cluster:                   2
  Bitmap size:                            65776
  
  ---------------- Show the statistics ----------------
  Cluster size:                           32768
  Total Clusters:                         526208
  Free Clusters:                          1907
  

• mount exfat and list files

  root@localhost:~# mount -t exfat /dev/sda3 /mnt/t1
  root@localhost:~# find /mnt/t1 -ls
          1     32 drwxr-xr-x   4 root     root        32768 Mar  6 19:41 /mnt/t1
       7056     32 drwxr-xr-x   3 root     root        32768 Jul 28  2021 /mnt/t1/System\ Volume\ Information
       7060     32 -rwxr-xr-x   1 root     root           76 Jul 28  2021 /mnt/t1/System\ Volume\ Information/IndexerVolumeGuid
       7061     32 -rwxr-xr-x   1 root     root           12 Jul 28  2021 /mnt/t1/System\ Volume\ Information/WPSettings.dat
       7062     32 drwxr-xr-x   3 root     root        32768 Aug  8  2021 /mnt/t1/System\ Volume\ Information/EDP
       7064     32 drwxr-xr-x   2 root     root        32768 Aug  8  2021 /mnt/t1/System\ Volume\ Information/EDP/Recovery
       7065     32 drwxr-xr-x   2 root     root        32768 Jul 28  2021 /mnt/t1/$RECYCLE.BIN
       7067     32 -rwxr-xr-x   1 root     root          129 Jul 28  2021 /mnt/t1/$RECYCLE.BIN/desktop.ini
       7068 16777216 -rwxr-xr-x   1 root     root     17179869184 Mar  6 11:09 /mnt/t1/pagefile.sys
       7069       32 -rwxr-xr-x   1 root     root           12288 Mar  6 11:08 /mnt/t1/DumpStack.log.tmp
  

• filefrag on exfat file

  root@localhost:~# filefrag -v /mnt/t1/pagefile.sys
  Filesystem type is: 2011bab0
  File size of /mnt/t1/pagefile.sys is 17179869184 (33554432 blocks of 512 bytes)
  Discontinuity: Block 16777216 is at 33563456 (was 33562624)
  /mnt/t1/pagefile.sys: 3 extents found
  

• hdparm fibmap

  root@localhost:~# hdparm --fibmap /mnt/t1/pagefile.sys
  
  /mnt/t1/pagefile.sys:
   filesystem blocksize 512, begins at LBA 138217472; assuming 512 byte sectors.
   byte_offset  begin_LBA    end_LBA    sectors
             0  155002880  171780095   16777216
    8589934592  171780928  171902975     122048
    8652423168  138226304  154881471   16655168
  

• fsck.exfat

  root@localhost:~# umount /mnt/t1
  root@sirius:~# fsck.exfat --version
  exfatprogs version : 1.2.8
  Usage: fsck.exfat
          -r | --repair        Repair interactively
          -y | --repair-yes    Repair without ask
          -n | --repair-no     No repair
          -p | --repair-auto   Repair automatically
          -a                   Repair automatically
          -b | --ignore-bad-fs Try to recover even if exfat is not found
          -s | --rescue        Assign orphaned clusters to files
          -V | --version       Show version
          -v | --verbose       Print debug
          -h | --help          Show help
  root@localhost:~# fsck.exfat -n -v /dev/sda3
  exfatprogs version : 1.2.8
  label: WINPAGE
  sector size:  512.00 B
  cluster size: 32.00 KB
  volume size:  16.06 GB
  /dev/sda3: clean. directories 5, files 5
  

[hgkamath]

• I understand that fsck.exfat might help with the recovery of files in case of corrupted fs/disk.
• The reason I think its also a good idea to have a pre-disaster tool that dumps the FAT/fragment-info/extent-info is because one can then use a file's last known locations and then scrape the extents off and stitch the extents together much like fsck.exfat but with reliable uncorrupted backed up FAT-info.
• fsck is a in-place repair tool. Sometimes, in order to prevent further deterioration of the corrupted image, one does not want to repair the FS, just copy important files off from it and save them elsewhere without modifying the image.
• Perhaps the FAT dump may be allowed to be not human readable
  • One way to achieve the above would be to just dump the extents on disk that correspond to the FAT.
  • Perhaps, another tool can be written to then parse the FAT-dump
  • In my previous experimentations, it felt like dd-ing out the first 100Mb of the ExFAT FS contained the FAT, and would serve the same purpose
• The dump file hence serves as a backup of the FAT , which can then be used as an input file to fsck.exfat in the event the FAT on disk-image is corrupted.
  • ex (hypothetical)

  ~# # backup of exfat dump as daily chron task at 12:00am midnight
  ~# exfat.dump -o /home/bkup/sda3_exfat_good_dump.bin
  ~#
  
  ~# # later, if in case the FS has troubles
  ~# losetup -f /home/bkup/corrupted_exfat_fs.img
  ~# fsck.exfat -n -v -i /home/bkup/sda3_exfat_good_dump.bin /dev/loop0 
  

  in the above, fsck.exfat uses the dump file made some time earlier to disaster in order to fix /dev/loop0
• This can be useful when one knows the FAT itself undergoes modification less frequently, and the extents/fragments of the files in interest do not change/grow/shrink. This is usually true for large fixed-size VHDX,VMDK,VDI,raw.img, .iso and other virtual-disk, VM disk images.