Upstream status

[Fantu]

@SergeiShtepa Hi, thanks for your works on blksnap kernel module, I think will be very useful for improve backup on linux.
I'm curious about the status of its possible upstream integration, from 85bad8f seems you prepared a patches serie for upstream (for send a mail with it) but I not found it posted searching on lkml.org and google. I saw also other series of months ago on other branches but I not found them on upstream mailing list and found only older your works of 1 year ago or more.
Thanks for any reply and sorry for my bad english.

[SergeiShtepa]

Thank you for creating the issue. So it's important to someone.

Yes, I haven't tried to offer blksnap in the upstream yet.
The problem is that I can't find anyone who could do a code review. And I don 't have enough experience with upstream.
In addition, the module is not a regular driver for some device. I suggest adding a filter to the block layer. Mantainers may not like it.
It would be great if I could attract other developers of backup tools to work on the module. But I think that while the module is not in the upstream, no one will want to invest their time in this project.
Therefore, I doubt the success of this venture.

But I intend to try to do it soon. It remains to double-check something.
I hope that someone will give good advice.
I will add a link to the patch discussion in upstream to this issue.

Any feedback is welcome.

[Fantu]

Thanks for reply, I did/do small contributions on some open source projects but I never did on linux kernel, if I remember good I gave only an help in testing one xen part preparing to upstream many years ago.
To know if can be acceptable your work and receive review of expert kernel developers (who can give you their opinions on it and their advices to improve where needed) post the patches is really important, from a fast look I found this doc that explain how to do it: https://www.kernel.org/doc/html/latest/process/submitting-patches.html (probably you already saw it)
and from a fast look to your latest patches serie ( 85bad8f ) seems you already did a good work on major requirements.
I found this project in a search done after I found one blk-snap patches serie you posted 2 year ago so I think posting patches is also a way let know it to any other interested developers and users (as well as receiving upstream developers review)

[SergeiShtepa]

seems you already did a good work on major requirements.

Thanks. Indeed, it is the good quality of the code that is an important component of success.

a way let know it to any other interested developers and users

Of course, that's why I'll be offering a patch.

[andymend]

I agree that your code represents a potentially important improvement for backup of linux systems, especially as new file systems proliferate. Blksnap is a distinct improvement over earlier such attempts like dattobd and elastio-snap. I have been lightly testing blksnap on a computer I routinely use where there are multiple file systems being used. Yours is a worthwhile project that will hopefully get merged into the mainstream kernel at some point.

[SergeiShtepa]

Thank you @andymend .
Thank you all for your support.
I just sent a patch.
If it passes the robot check, we will see it in the list soon.

[SergeiShtepa]

patchwork.kernel.org
lore.kernel.org

[Fantu]

for what I see here 2 kernel developers reviewed and replied and there is also 2 replied from an automatic test bot (reporting some warnings), for now, no one has objected or criticized the implementation method but only recommended improvements for a v2.

from a quick search I did not find other proposals for implementing "temporary" snapshots for better backup integrity and that supports many filesystems

unfortunately I have not yet had time to try blksnap but from what I have seen from experience this functionality integrated into the kernel is very useful and as I have already seen several backup solutions have made their own modules for similar things but no one who has proposed its implementation in the kernel

I have used several backup solutions over the years, most with file level backup using rsync, however even in such cases I think it would be useful because either the services stop during the whole backup or you risk not having it integrated and the larger the part to back up and the more time it takes, the greater the risk of integrity being compromised.
For example, if you have to back up a server of hundreds of gb or over 1 tb (in most cases via the network the times increase), it takes hours and during the operations to numerous files in various interconnected parts and rsync buckups of the parts before such them, others during and others after would be problematic for its integrity on restore while with a snapshot and rsync backup on it you would have a backup with minimal integrity risk (reduced to the fraction of snapshot creation time FWIK) and restore you would know that you have a fairly healthy situation at the time of the snapshot and not a high risk of integrity problems for the time it took to make the backup. And without such a snapshot, the risks can only be mitigated by stopping the services (as mentioned) or doing it outside working hours.

Another thing I tried at works in latest years is to use btrfs with integrated snapshot for backups (I also tried zfs years ago) but I use only in some cases because for example on virtualization systems, the COW it has a significant impact on performance and there are also some space issues keeping snapshots on the same production filesystem. this could be a solution to make a backup with minimum downtime stopping services to take the snapshot (with blksnap) and maximum integrity probability, or at least making the snapshot to active services like what a COW filesystem like btrfs and zfs can do but without having to use such filesystems.

I hope these examples based on my experience give an idea of how useful a system like blksnap is
When there is a calm period where I am not too busy I will start testing it
in the meantime thanks again for your work SergeiShtepa

sorry for my bad english

[SergeiShtepa]

Thanks @Fantu.

I expected a very calm reaction to the patch.
I plan to:

• correct a flaws, indicated by the reviewers
• move some of the logic into the kernel module, simplifying the main kernel code even more
• fix bugs (QA found something).

So, the next patch will be better.

Alas, the performance of the blksnap module should not be better than DM snapshots (LVM2).
When handling a write request, it is required to read a chunk of data, and then write this chunk of data to a new location. There are no magic here. When writing, the load on the drive increases threefold.

Theoretically, snapshots on Btrfs should be better. But for BTRFS, we cannot read the entire block device. We have to synchronize the source and target file systems. The Veeam Agent for Linux implements btrfs backup support by its native means. The performance was not the best.

The presence of a change tracker and the ease of allocating space for the difference storage are the main advantages of blksnap, compared to "classic" snapshots.

[Fantu]

thanks for reply and your works
about btrfs I know that is better where there is no "limit" or performance issue (for example I use on some backup storage and fileservers), but for example in "low cost" virtualization servers there was significant performance issue, disabling COW partially solves the issue because using snapshot will require it and ignore the previous options to disable cow, so still using ext4 and do temp. snapshot only on backup time, avoid the downtime or having it shortly, having high integrity probability and lower performance only when backup with snapshot is running still seems to me a good idea.
however I can only be helpful enough when I start using it

[SergeiShtepa]

Hello everyone.
I am very glad that the first patch was noticed and I received quite useful feedback.
A lot will have to be rewritten. And that's great.

[Fantu]

hi, is there any news about v2 patch serie for upstream?
I suppose anyway it would not be reviewed in time for the kernel 6.0 :(
edit: merge window for 6.0 is already closed, I hope for the 6.1

[SergeiShtepa]

Hi. That's what I really want to do.
But I had to switch to more urgent tasks. I hope to be able to reduce my task list soon.

[SergeiShtepa]

Hi!

This looks like a workable version of the patch for linux kernel 6.1-rc1 .
It contains corrections in accordance with the comments that were made by the maintainers to the first version of the patch.
I plan to test this for two or three days.
I invite everyone to join the testing.

---

Feedback is welcome.

[SergeiShtepa]

This patch has been sent.

[Fantu]

good, I not checked it in detail but I noted a small mistake on the version of the patch serie posted, v1 instead of v2, but if you have sent it by now there is nothing else to do (or they would receive it double), I saw there is at least a list of changes from v1 in the cover so it should be "ok" anyway

edit: found the link of the serie posted, I put here if can be useful:
https://lore.kernel.org/lkml/[email protected]/

[SergeiShtepa]

Link to patchwork.

[Fantu]

@SergeiShtepa with a search I found this: https://lwn.net/Articles/914852/ and seems that wrote about a lack of documentation to be accepted
in addition to the documentation in code that probably need improvements I suppose they meaning to add another "generic" in Documentation/, I suppose in Documentation/block creating Documentation/block/blksnap.rst and adding in the list of Documentation/block/index.rst
I don't saw reply to lack of documentation in the replies of the patch serie posted, you received other reply directly and not in mailing list?

[SergeiShtepa]

Thanks.
I don't know anything about it.
I received one email about an extra space and 3 emails from the robot.
I'll try to add a couple of comments to the article.

[Fantu]

@SergeiShtepa I saw your replies on lwn, unfortunately even as you explained you want to avoid additional time being wasted reviewing documentation before it is accepted upstream I suppose without more documentation some people won't start reviewing it. For now on mailing list of patch posted I didn't see new replies :(
I did a fast check with kernel-doc script of the kernel-doc comments actually present and spotted some warning:
one I suppose solved with Fantu/linux-blksnap@4f7c236
others spotted:

drivers/block/blksnap/tracker.h:47: warning: Function parameter or member 'flt' not described in 'tracker'
drivers/block/blksnap/tracker.h:47: warning: Function parameter or member 'submit_lock' not described in 'tracker'

drivers/block/blksnap/diff_storage.c:21: warning: Function parameter or member 'link' not described in 'storage_bdev'
drivers/block/blksnap/diff_storage.c:21: warning: Function parameter or member 'dev_id' not described in 'storage_bdev'
drivers/block/blksnap/diff_storage.c:21: warning: Function parameter or member 'bdev' not described in 'storage_bdev'

drivers/block/blksnap/diff_storage.c:33: warning: Function parameter or member 'link' not described in 'storage_block'
drivers/block/blksnap/diff_storage.c:33: warning: Function parameter or member 'bdev' not described in 'storage_block'
drivers/block/blksnap/diff_storage.c:33: warning: Function parameter or member 'sector' not described in 'storage_block'
drivers/block/blksnap/diff_storage.c:33: warning: Function parameter or member 'count' not described in 'storage_block'
drivers/block/blksnap/diff_storage.c:33: warning: Function parameter or member 'used' not described in 'storage_block'

[SergeiShtepa]

To read the entire article https://lwn.net/Articles/914031/ and comments before December 1, you need to be a registered user. After December 1st, access will be unlimited. I tried to answer all the questions.
I plan to work on the documentation and try to write an article for LWN. Maybe it will help.

spotted some warning:

Thanks, I'll have to check it out too for next patch.

Perhaps the next v2 patch with documentation and accompanied by an interesting article will attract more attention.

[Fantu]

@SergeiShtepa thanks for replies and your work
in practice v2 would be the last one posted and v3 the next even if it was written v1, I wouldn't want that perhaps having written v1 had led some people to suppose in a resend without changes, for example it seems strange to me that Christoph Hellwig who had participated a lot in the the first did not give any answer to the second (probably can be good add it in cc of the next)
looking also on other sites the article with write about "the lack of doc." seems to have left its mark :(
https://www.reddit.com/r/linux/comments/yvd14a/blockdevice_snapshots_with_blksnap_lwnnet_good/
probably a v3 with more documentation and even just with the bot's warning fixes and Randy's reporting will push sites to update about it and attract more people to review

[Fantu]

@SergeiShtepa in the patch serie is missed the add of entry in MAINTAINERS file

[SergeiShtepa]

@SergeiShtepa in the patch serie is missed the add of entry in MAINTAINERS file

Maybe...
I'll add this in the next patch.

[SergeiShtepa]

Add docs for /linux/Documentation/block/ : blkfilter and blksnap .
I really hope that I managed to find the right words.

Any feedback is welcome.

P.S.: Since yesterday article is available without registration.

[Fantu]

@SergeiShtepa thanks for the creation of documentation to include in the patch for upstream
my english isn't great but i read them, blkfilter it seems to be ok, about blksnap there is a non-english sentence in "Snapshot overflow resistance" and the part of defects of the alternatives and merits of blksnap leaves me doubtful, I'm afraid that someone who will review or want to watch/try blksnap will get lost in counterproductive discussions.

about btrfs "Obviously cannot be applied to other file systems.", I suppose you mean the fact that the main advantages of transferring to another local or remote destination are lost from or to a different filesystem, and it is, but written like this is wrong, in practice it is possible to use it even if most of the advantages are lost. You can use it both for "temporary" snapshots during which you make a backup (for example with rsync) to another destination; and from a source without btrfs, you make backups with another (for example rsync) and on the destination you use the btrfs snapshots to manage the "history" with some advantages (I've been using it on some systems for years).

the rest of the disadvantages I saw in practice during a long testing some years ago thinking about using btrfs instead of ext4 on several virtualization systems to manage better and faster backups but the performance impact was huge (partly due to the use of hdd) and even if only the last corresponding snapshot was kept for send/receive with the destination there were disadvantages, reduced in part with a single destination and daily backup but considerable with multiple destinations (in my case 2) in "rotation" and having to keep a week or even more the snapshots.
I've seen what the arrival of the btrfs fs to full space caused by snapshot, the damage it causes especially with the virtual machine (which do not have direct access to the fs and "amplify" the problem) and also the fact that it is not "automatically managed/avoided" and the workaround of scripting to see if it's about to fill up and deleting the snapshot(s) caused most of the "PROs" to decay when that happens, which is why we didn't use it in production for that use case

however without listing all the possibilities I think it is wrong to write things like "the blksnap module is the best way to create snapshots for backup tools", it is in good part of cases but not all.
I'm not good at writing and I don't quite know how to describe such parts properly, objectively and correctly but I think that the documentation it should be done that way people should just understand the need for this method, its importance for a good chunk of cases and the need to get it good and integrated upstream

sorry if I didn't explain myself well and for my bad english

[SergeiShtepa]

@Fantu - Thanks for the feedback!

I will try to improve those sections that you have noticed.
After the weekend, I will review the documentation again.
Maybe my colleagues from the technical documentation department will help me.

My English is also not as good as I would like. So I'm trying to do what I'm capable of. :)
Creating documentation is hard work, and now it looks really important.

I can suggest you try to make the document better. Why not try?
Even if I don't accept your merge request, it will allow me to take a fresh look at some sections.

[SergeiShtepa]

Hi!
Thanks!
OK, I will make a full log of changes, as in the previous version of the patch.
Be healthy! We all need to be able to listen to our body when it needs help.

[SergeiShtepa]

Hi.
patch v4 was sent.
But due to the "unwarmed" graylist on the mail server @vger.kernel.org, the first mail was torn off from the rest of the patchset.
Header: https://lore.kernel.org/linux-block/[email protected]/T/#u
Patchset: https://lore.kernel.org/linux-block/[email protected]/T/#t
Whether this is a problem or not, I do not know. I will be waiting for comments on this.

[SergeiShtepa]

patch v5.
For "kernel/git/axboe/linux-block.git" branch "for-6.5/block". Link.
lore.
patchwork.

[Fantu]

@SergeiShtepa the latest mail shipments seem to have been a bit chaotic^^'' v4 splitted and double v5 I saw and replies "scattered"
there is good news that Christoph Hellwig https://lore.kernel.org/linux-block/ZIcsijGWeyk%[email protected]/T/#mb373f939657cfe8c2c2439ef4f5886be21bb061c seems consider blksnap ready for inclusion (there are still some things to do from the replies
I hope that mail send issues don't discourage reviewers

[SergeiShtepa]

Yeah, mail...
For the server vger.kernel.org server veeam.com is unknown. His IP is usually not on the gray list, because I rarely send emails. And if I send the entire patchset at once, then it perceives it as an attack on the server and blocks the senders IP.
Therefore, the last unsuccessful time I sent emails with a timeout per minute.
However, I did not take into account that the session for git send-mail cannot be longer than 10 minutes...

I hope that in the future I will be able to organize work with mail so that such problems do not arise.

[Fantu]

looking https://github.com/torvalds/linux after the latest "block merge" of 2 days ago seems that "for-next" of block was fully merged (https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=for-next) so I think can blksnap can be rebased on latest https://github.com/torvalds/linux (now that include all the block changes) for working on next version of the patch

[SergeiShtepa]

Yes, thanks Fabio.
We already have 6.5-rc2 .
I apologize for the late response. I was on vacation.
There's still a lot of work to do on redesigning the ioctl extension of the difference storage. And blk-crypto...

[SergeiShtepa]

Hi!
Little update.

In the 'stable-v2.0' branch of this repository and in the branch 'blksnap-master' based on vanilla 6.6-rc2 of linux repository , the concept of storing snapshot differences in a file was implemented.
The ability to create a difference storage on a block device under the snapshot has been removed.
Storing differences on a block device remains possible, but only in the case of exclusive ownership of it.

An interesting consequence of these changes:

• Now it is possible to create a difference storage file with the O_TMPFILE flag. The file will be automatically deleted when its descriptor is closed when the snapshot is released.
• Now it is possible to use files in memory and in the 'swap' partition thanks to files on tmpfs.

I think that it is possible to return the ability to create a difference storage on a block device under snapshots. But this will require support from file systems. It can be quite a difficult job.
I plan to start it when I realize that the blksnap module is in demand by users and they really need this feature.

In the near future, I will be engaged in testing, improve documentation and preparing a patch.
I invite everyone to take part in testing the current version.

[SergeiShtepa]

Hi.
Patch lk6.7-rc1-v2 has been prepared.
I plan to send it next week.

[Fantu]

Tested build on amd64 (both builtin and module), armhf, arm64 and various ppc config.
I don't have time to deep check and testing it in these days but seems to me achieved the main requirements for integration, and that with the latest changes reviewer had recommended I assume is approaching.
@SergeiShtepa before send, I suggest taking a look at #79 relating to the patches cover

[SergeiShtepa]

V6:
lore.kernel.org/
patchwork.kernel.org

[SergeiShtepa]

Hi.
Patch lk6.8-rc1 has been prepared.
It also available as a branch in my fork.

[Fantu]

Hi, I don't have enough time to test its use, I just launched some amd64 builds (builtin and module), armhf, arm64 and some ppc. (from automatic tests I prepared in past).
One ppc build failed for an issue not related to blksnap.
On armhf build (https://github.com/Fantu/linux-blksnap/actions/runs/7629304370/job/20782415299) I spotted this warning:

2024-01-23T17:55:07.5898286Z drivers/block/blksnap/main.c: In function ‘ioctl_snapshot_create’:
2024-01-23T17:55:07.5901218Z ##[warning]drivers/block/blksnap/main.c:196:30: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
2024-01-23T17:55:07.5903698Z   196 |         fname = strndup_user((const char __user *)karg.diff_storage_filename,
2024-01-23T17:55:07.5904914Z       |                              ^

[SergeiShtepa]

v7
https://patchwork.kernel.org/project/linux-block/list/?series=824711
https://lore.kernel.org/all/[email protected]/

[Fantu]

This time the send of patch serie seems complete and without duplicate 👍

[SergeiShtepa]

Yep.
I have reduce the number of patches in set and greatly reduced the list of recipients.
The script get_maintainer.pl returns such a long list that it can no longer be used to generate a list of recipients.
The test bot should respond soon. He will not be able to apply patches to the master branch and build.

[Fantu]

Looking kernel bot message of older version seems it try also with linux-block for-next: https://lore.kernel.org/lkml/[email protected]/

url: https://github.com/intel-lab-lkp/linux/commits/Sergei-Shtepa/blksnap-creating-non-persistent-snapshots-for-backup/20221103-004434
base: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git for-next

So I think that if will apply correctly with master or linux-block don't report error, unless it detect other errors and/or warnings

[SergeiShtepa]

I checked it out.
The patch can be successfully applied to Jens branch block/for-next and can be successfully build.
Jens create this branch at the end of last week.

[Fantu]

Of v7 I didn't pay attention to the patches subjects, from patch 3/8 to patch 8/8 I think it should be "blksnap:" or "block/blksnap:" instead of "block:" at the beginning of the subjects

[Fantu]

@SergeiShtepa hi, Christoph did write to you to find out if the latest version was good enough for the upstream application, or need other improvements, or did he not have time to review it?
I haven't seen any public responses to the latest version, I suspect that someone who was interested missed it due to some patches with incorrect titles.
Perhaps even with the fewest mail recipients, but that seems necessary to avoid it being blocked and being sent back double (or more), which would annoy the recipients :(

[SergeiShtepa]

Hi, @Fantu !

I haven't seen any public responses to the latest version, I suspect that someone who was interested missed it due to some patches with incorrect titles.

Yep. Indeed, there is no reaction. I think that everyone who wanted to see the latest patchset did it, regardless of whether he is the recipient of the letter or not.

I am currently preparing an updated standalone version of the module for one of the next releases of our product.
I want to move a some ideas from the upstream version to it. This will improve performance.

Even if the blksnap module is never accepted into the kernel, the work has not been done in vain.
The code of the blksnap module only gets better from this.

I will definitely return to work on the upstream version later.
In working on the next patch for upstream, I think I will focus on more detailed testing and performance measurement.

[SergeiShtepa]

Hi!
The patchset for LK v6.9 is ready to view.
The blksnap-lk6.9 branch is available too.

[Fantu]

Hi, this patchset is only for testing on 6.9 but not for submit, right?
A v8 will be prepared for 6.10 to submit?

[SergeiShtepa]

Hi, I don't know yet when I will write e-mails for maintainers.
I would like to test more.

[fsupper]

Hi, any update on this Topic?
What is missing to get the blksnap module into the linux kernel?
Regards
Flo

[SergeiShtepa]

Hi.

What is missing to get the blksnap module into the linux kernel?

And that's a good question.

As usual, the most valuable resource is missing - time.
But this is a matter of priorities. Well, the most important of them is: "Why?".
I know what Veeam needs it for, but it's completely unclear to me if the upstream needs it.
I have formed the opinion that this is not interesting to the Linux community.
Other backup vendors are not involved in the patchset discussion.
Everyone seems to be quite satisfied with dm-snap.

Nevertheless, I plan to make at least one more attempt. It is possible that this will happen this autumn.

[fsupper]

Hi.

What is missing to get the blksnap module into the linux kernel?

And that's a good question.

As usual, the most valuable resource is missing - time. But this is a matter of priorities. Well, the most important of them is: "Why?". I know what Veeam needs it for, but it's completely unclear to me if the upstream needs it. I have formed the opinion that this is not interesting to the Linux community. Other backup vendors are not involved in the patchset discussion. Everyone seems to be quite satisfied with dm-snap.

Nevertheless, I plan to make at least one more attempt. It is possible that this will happen this autumn.

Thanks for your honest answer!
regards

[Fantu]

I would like to contribute more, but I lack the time and I lack sufficient knowledge for the parts of the kernel involved (and the time to learn them).
Regarding the interest I think there is more people than think (I mean mainly users), but it is difficult for them to find blksnap and those who find it do not know how it could be useful, and therefore you find mostly messages from users asking for help for new kernel version support (for external module), trying the patches for upstream without enough knowledge to use them, or asking for the status for upstream (there are also in other forums, blogs etc...).
Regarding other backup solutions, they mainly concern commercial software and companies in most cases only look at their own interest, calculate very narrowly, stay with something of poor quality rather than contribute even a little to something that also benefits others. Basically, this reasoning (which I consider stupid) could be summarized as: it is better for everyone to stay with worse rather than have better if others also benefit from it.
It seems to me that there are still few companies that understand that having more open and collaborative projects can lead to greater quality and potentially also with lower costs, but there are some that are making progress, for example Microsoft has done much more than I would have ever imagined in the last 10 years (compared to the previous opposite thought they had).
Regarding the developers involved in the discussions, it seems to me that most of them have a limited vision, calculating only or almost their own uses and skills/knowledge and underestimating the segment of users to whom it would be useful, both companies that need a single, centralized and easily manageable backup system for several devices and smaller situations that also need more complete and simple backups. There is something with other solutions, for example some that are slowly being integrated into various distros (but they are mainly things that require more work and limitations to use them massively), but even there they are going slowly and therefore I think blksnap could still have a significant slice of use, even if not as large as it would have been 5-10 years ago.

[SergeiShtepa]

Yes, that's right. I agree with every word @Fantu.

Plus, I and the company I work for have pretty little experience working with open source.
Perhaps , this is our point of growth. I suppose that more efforts should be made in this direction.
This is just my personal opinion.