Skip to content

feat(handler): add erofs filesystem & decompression handler #1163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 9, 2025

Conversation

rxpha3l
Copy link
Contributor

@rxpha3l rxpha3l commented Apr 4, 2025

feat(handler): add erofs filesystem & decompression handler

EROFS is a high-performance read-only filesystem originally developed by Huawei and now used extensively in AOSP. It offers features like compression (LZ4, LZMA), inline data, and reduced
storage overhead, making it ideal for read-only system images.

Create erofs file :

mkfs.erofs -z[compressor, level (optional)] foo.erofs.img foo/

Extract erofs file:

fsck.erofs --extract=foo_extracted/ foo.erofs.img

Key notes:

  • First 1024 bytes are reserved for the boot loader
  • Header starts at 0x400 with E0F5E1E2
  • Header can be more than 128 bytes if extra slots provided
  • File format in little endian
  • There is no standarized file extension. The official documentation
    uses ".erofs.img"
  • Checksum is differ form verion to version

Header format :

4 bytes : Magic
4 bytes : Checksum, crc32c
4 byets : feature compact
1 byte : Block size in bit shift
1 byte : Super block extra slots (128 + sb_extslots * 16)
2 bytes : Root ID number
8 bytes : Total valid inodes
8 bytes : Built time
4 bytes : Built time nsec
4 bytes : Block count
4 bytes : Start block addr of metadata area
4 bytes : Start block of shared xattr area
16 bytes : UUID
16 bytes : Volume name
4 bytes : Feature incompact
2 bytes : Union of Compresion algorithm & LZ4 max distance
2 bytes : Extra device
2 bytes : Devt slotoff
1 byte : Directory block size in bit shit
1 byte : Number of ling xattr prefixes
8 bytes : NID of special packed inode
1 byte : Reserved fir xattr name filter
23 bytes : Reserved (Most likely NULL bytes)
Unblob struct the header until the feature_incompact. The rest is labeled
as "reserved". Unblob parser the end offset by multiplying the block
size in bit shift with the block count. Since the checksum is not
consistent troughout the version, unblob matched on block count (at
least 1), printable volume name and if there is a build time.

[Sources]
https://elixir.bootlin.com/linux/v6.14-rc6/source/fs/erofs/erofs_fs.h
https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/erofs/erofs_fs.h?h=v6.6
https://erofs.docs.kernel.org/en/latest/core_ondisk.html
https://www.kernel.org/doc/html/latest/filesystems/erofs.html

@qkaiser qkaiser force-pushed the erofs branch 2 times, most recently from 11bb317 to 27366dc Compare April 4, 2025 16:17
@qkaiser qkaiser requested review from qkaiser and Copilot April 4, 2025 16:17
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 3 out of 9 changed files in this pull request and generated 2 comments.

Files not reviewed (6)
  • install-deps.sh: Language not supported
  • overlay.nix: Language not supported
  • package.nix: Language not supported
  • shell.nix: Language not supported
  • tests/integration/filesystem/android/erofs/input/foo.erofs.img: Language not supported
  • tests/integration/filesystem/android/erofs/output/foo.erofs.img_extract/bar.txt: Language not supported

@qkaiser
Copy link
Contributor

qkaiser commented Apr 4, 2025

@rxpha3l make sure you pull from your remote with git pull origin erofs before doing anything else. I cleaned up your commits.

Copy link
Contributor

@qkaiser qkaiser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code is clean. Nice catch on fsck.erofs being able to extract. Copilot's comments are relevant, you can take care of it.

@qkaiser qkaiser linked an issue Apr 4, 2025 that may be closed by this pull request
@qkaiser qkaiser added this to the Internship 2025 milestone Apr 4, 2025
@qkaiser qkaiser added enhancement New feature or request format:filesystem labels Apr 4, 2025
@qkaiser
Copy link
Contributor

qkaiser commented Apr 4, 2025

@rxpha3l please make sure to also update the documentation with information about EROFS.

Instead of adding patches on top of unlob package available on nixpkgs,
we keep our own package.nix version here so that it's easy to add new
third party dependencies that new handlers may require.

It also makes it easier to upstream it since we just have to submit the
package.nix file to nixpkgs.
@rxpha3l rxpha3l force-pushed the erofs branch 2 times, most recently from ee660fe to 70b29d6 Compare April 8, 2025 10:57
EROFS is a high-performance read-only filesystem originally developed by
Huawei and now used extensively in AOSP. It offers features like
compression (LZ4, LZ4HC, LZMA, ZLIB, ZSTD), inline data, and reduced storage overhead,
making it ideal for read-only system images.

Create erofs file : > mkfs.erofs -z[compressor, level (optional)]
foo.erofs.img foo/

Extract erofs file: > fsck.erofs --extract=foo_extracted/ foo.erofs.img

Key notes:
- First 1024 bytes are reserved for the boot loader
- Header starts at 0x400 with E0F5E1E2
- Header can be more than 128 bytes if extra slots provided
- File format in little endian
- There is no standarized file extension. The official documentation
  uses ".erofs.img"
- Checksum changes from version to version

HEADER / SUPERBLOCK:
>  4 bytes : Magic
>  4 bytes : Checksum
>  4 bytes : Feature
>  1 byte  : Block size in bit shift
>  1 byte  : Superblock extension
>  2 bytes : Root NID
>  8 bytes : Total valid i-node
>  8 bytes : Built time in timestamp
>  4 bytes : Nanoseconds (ns) component of timestamp
>  4 bytes : Total block count
>  4 bytes : Start block addr of metadata area
>  4 bytes : Start block of shared xattr area
> 16 bytes : UUID for volume
> 16 bytes : Filesystem label
>  4 bytes : Incompatible feature flag for kernel
>  2 bytes : Available compression algorithm
>  2 bytes : LZ4 max distance
>  2 bytes : Extra devices
>  2 bytes : Start address of external device table
>  1 byte  : Directory block size 2^blkszbits+dirblkbits
>  1 byte  : Total number of long xattr name prefixes
>  4 bytes : Start address of long xattr prefixes
>  8 bytes : NID of the special packed inode.
>  1 byte  : xattr filter reserved
> 23 bytes : Reserved

Unblob structs the header until feature_incompact. The rest is labeled
as "reserved". Unblob parses the end offset by multiplying the block
size in bit shift with the block count.

Since the checksum is not consistent troughout versions, unblob matches
on block count (min 1), build time, build
time in ns, bit shift (min 9) and printable volume name.

[Sources]
https://elixir.bootlin.com/linux/v6.14-rc6/source/fs/erofs/erofs_fs.h
https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/erofs/erofs_fs.h?h=v6.6
https://erofs.docs.kernel.org/en/latest/core_ondisk.html
https://www.kernel.org/doc/html/latest/filesystems/erofs.html
@qkaiser qkaiser added this pull request to the merge queue Apr 9, 2025
Merged via the queue into onekey-sec:main with commit 32a9a46 Apr 9, 2025
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request format:filesystem
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for Android EROFS (Enhanced Read-Only File System)
3 participants