remove: partial downloads (remove creation of the stub messages) #7373

Simon-Laux · 2025-10-30T16:40:22Z

remove code for partial downloads
remove tests for partial downloads and make a list of them so we can discuss if we need to re-add some in the new system or not
~~make download full message methods fail for now with not implemented error and comment what should happen later~~ doesn't need to change at this time (and possibly not at all)

I kept partial_download_msg_body stock string for now, as it may be reused in the new solution. because I made the linter ignore it, I added a to do item to #7367 to check if it was used in the end.

removed partial download tests:

test_reaction_to_partially_fetched_msg -

core/deltachat-rpc-client/tests/test_something.py

Line 623 in 0d46626

def test_reaction_to_partially_fetched_msg(acfactory, tmp_path):
test_partial_receive_imf -

core/src/download.rs

Line 346 in 8b4c718

async fn test_partial_receive_imf() -> Result<()> {
test_partial_download_and_ephemeral -

core/src/download.rs

Line 391 in 8b4c718

async fn test_partial_download_and_ephemeral() -> Result<()> {
test_status_update_expands_to_nothing -

core/src/download.rs

Line 425 in 8b4c718

async fn test_status_update_expands_to_nothing() -> Result<()> {
test_partial_download_trashed -

core/src/download.rs

Line 536 in 8b4c718

async fn test_partial_download_trashed() -> Result<()> {
test_partial_download_and_reaction -

core/src/reaction.rs

Line 929 in 8b4c718

async fn test_partial_download_and_reaction() -> Result<()> {
test_partial_group_consistency -

core/src/receive_imf/receive_imf_tests.rs

Line 4473 in 8b4c718

async fn test_partial_group_consistency() -> Result<()> {
test_partial_download_key_contact_lookup -

core/src/receive_imf/receive_imf_tests.rs

Line 5288 in 8b4c718

async fn test_partial_download_key_contact_lookup() -> Result<()> {
⭐test_webxdc_update_for_not_downloaded_instance -

core/src/webxdc/webxdc_tests.rs

Line 333 in 8b4c718

async fn test_webxdc_update_for_not_downloaded_instance() -> Result<()> {
test_no_partial_calls -

core/src/calls/calls_tests.rs

Line 617 in 8b4c718

async fn test_no_partial_calls() -> Result<()> {
test_markseen_not_downloaded_msg -

core/src/message/message_tests.rs

Line 331 in 8b4c718

async fn test_markseen_not_downloaded_msg() -> Result<()> {
test_msg_seen_on_imap_when_downloaded -

core/src/message/message_tests.rs

Line 404 in 8b4c718

async fn test_msg_seen_on_imap_when_downloaded() -> Result<()> {
test_create_group_with_big_msg -

core/src/receive_imf/receive_imf_tests.rs

Line 4398 in 8b4c718

async fn test_create_group_with_big_msg() -> Result<()> {
test_download_later -

core/src/receive_imf/receive_imf_tests.rs

Line 4333 in 8b4c718

async fn test_download_later() -> Result<()> {
test_prefer_references_to_downloaded_msgs -

core/src/receive_imf/receive_imf_tests.rs

Line 4792 in 8b4c718

async fn test_prefer_references_to_downloaded_msgs() -> Result<()> {
test_download_on_demand -

core/deltachat-rpc-client/tests/test_chatlist_events.py

Line 114 in 56370c2

def test_download_on_demand(acfactory: ACFactory) -> None:
test_download_limit_chat_assignment -

core/deltachat-rpc-client/tests/test_something.py

Line 723 in 56370c2

def test_download_limit_chat_assignment(acfactory, tmp_path, n_accounts):
test_webxdc_download_on_demand -

core/python/tests/test_1_online.py

Line 225 in 9bc2aee

def test_webxdc_download_on_demand(acfactory, data, lp):

part of #7367

Simon-Laux · 2025-10-30T20:13:35Z

run and fix python tests
- jsonrpc
- cffi
solve that it stops receiving messages (when used in desktop)

src/mimeparser.rs

Simon-Laux · 2025-10-31T18:52:22Z

src/receive_imf.rs

            info!(context, "Message already partly in DB, replacing.");
            Some(msg.chat_id)
+
+            // TODO: look at this place


todo comment

Note that there may be existing message-stubs that are not downloaded yet. After updating, the "Download" button will probably become non-functional; this is fine. We should make sure that DC doesn't crash when clicking "Download".

currently I'm still asking myself whether we have to to make this sacrifice or if we can still have a last version that has an option to download old message stubs and remove it in the version after...

As of the time of writing the download logic still works, because I have only removed the partial download logic that creates the stubs so far.

my current feeling/idea is to remove it in a dedicated pr as soon as it makes problems / the implementation of the new method harder - that way this pr would also be focused on the removal of just the stub creation part of partial download.

It's not that big of a sacrifice:

download-on-demand is just an option (i.e. many people don't have it activated, since they don't change the settings)

this option never worked really well

It can already happen that you can't fully download a message, because it has been deleted on the server already.

The user sees that there is a failure, and can ask the sender to send the message again

It does make sense to remove only the partial-download logic in the PR here, and then remove the replace_msg_id logic in a later PR. But if there is any kind of effort in it, it's not worth it, esp. as we need pre-messages rather sooner than later for calls etc.

src/imap.rs

iequidoo · 2025-11-10T22:31:07Z

src/download.rs

    }
-
-    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-    async fn test_partial_receive_imf() -> Result<()> {


I suggest to replace this with a test which adds a partially downloaded message to the db in some hacky way (probably by adding a full message as usual and then modifying it so that it becomes a partial download) and then receives a full message (probably the same message again). This would be a backward compat test for messages received before the upgrade.

I general other tests can be changed the same way, but i suggest not to do much work here, maybe only cover the most important scenarios which may happen after upgrade.

Quoting #7367:

Note that there may be existing message-stubs that are not downloaded yet. After updating, the "Download" button will probably become non-functional; this is fine. We should make sure that DC doesn't crash when clicking "Download".

I.e. there is not a lot to test here, only that clicking "Download" doesn't result in a crash (the download state can just become "failed"). This could get a test if it's not too much effort.

A test can be added rather easily in this way, probably:

Remove everything after the first call to receive_imf_from_inbox in this test here (on main, not on this branch here)

DELTACHAT_SAVE_TMP_DB=1 cargo test test_partial_receive_imf

sqlite3 test-account-alice.db

.dump

Copy the SQL statement inserting the partially-downloaded message. This SQL statement can be used to create an old message stub.

...but this is just to share some knowledge about what's possible. This can anyways not be done in the branch here, but in #7410

src/mimeparser.rs

iequidoo · 2025-11-10T22:31:15Z

src/webxdc/webxdc_tests.rs

 }

-#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-async fn test_webxdc_update_for_not_downloaded_instance() -> Result<()> {


This can be modified to employ message reordering instead of a partially downloaded instance

iequidoo · 2025-11-10T22:31:26Z

src/download.rs

-    use crate::test_utils::{E2EE_INFO_MSGS, TestContext, TestContextManager};
-
-    #[test]
-    fn test_downloadstate_values() {


Why remove the test if DownloadState doesn't change yet?

right this test needs to be restored

src/download.rs

Hocuri

Nice! LGTM once all comments are addressed (i.e. either resolved them, or commented why it's not worth it / doesn't make sense)

BTW, in general, it's good to request a review from some people using the Github UI once you've finished your PR, in order to get speedier reviews

deltachat-rpc-client/tests/test_chatlist_events.py

Hocuri · 2025-11-11T16:56:03Z

src/download.rs

    }
-
-    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-    async fn test_partial_receive_imf() -> Result<()> {


Quoting #7367:

Note that there may be existing message-stubs that are not downloaded yet. After updating, the "Download" button will probably become non-functional; this is fine. We should make sure that DC doesn't crash when clicking "Download".

I.e. there is not a lot to test here, only that clicking "Download" doesn't result in a crash (the download state can just become "failed"). This could get a test if it's not too much effort.

A test can be added rather easily in this way, probably:

Remove everything after the first call to receive_imf_from_inbox in this test here (on main, not on this branch here)

DELTACHAT_SAVE_TMP_DB=1 cargo test test_partial_receive_imf

sqlite3 test-account-alice.db

.dump

Copy the SQL statement inserting the partially-downloaded message. This SQL statement can be used to create an old message stub.

...but this is just to share some knowledge about what's possible. This can anyways not be done in the branch here, but in #7410

src/mimeparser.rs

Hocuri · 2025-11-11T17:50:33Z

src/receive_imf.rs

            info!(context, "Message already partly in DB, replacing.");
            Some(msg.chat_id)
+
+            // TODO: look at this place


It's not that big of a sacrifice:

download-on-demand is just an option (i.e. many people don't have it activated, since they don't change the settings)

this option never worked really well

It can already happen that you can't fully download a message, because it has been deleted on the server already.

The user sees that there is a failure, and can ask the sender to send the message again

It does make sense to remove only the partial-download logic in the PR here, and then remove the replace_msg_id logic in a later PR. But if there is any kind of effort in it, it's not worth it, esp. as we need pre-messages rather sooner than later for calls etc.

src/stock_str.rs

system as it is only ever applied to messages that have a pre-message.

`Config::SimulateReceiveImfError`

Simon-Laux force-pushed the simon/remove-partial-download-of-messages branch from 0d46626 to 81ca8c2 Compare October 30, 2025 17:30

Simon-Laux commented Oct 31, 2025

View reviewed changes

src/mimeparser.rs Outdated Show resolved Hide resolved

Simon-Laux mentioned this pull request Oct 31, 2025

[Tracking Issue] Pre-Messages #7367

Open

7 tasks

Simon-Laux commented Oct 31, 2025

View reviewed changes

Simon-Laux marked this pull request as ready for review October 31, 2025 23:16

Simon-Laux marked this pull request as draft October 31, 2025 23:29

Simon-Laux changed the title ~~remove: partial downloads~~ remove: partial downloads - creation of the stub messages Oct 31, 2025

Simon-Laux marked this pull request as ready for review November 1, 2025 12:41

iequidoo reviewed Nov 3, 2025

View reviewed changes

src/imap.rs Outdated Show resolved Hide resolved

Simon-Laux force-pushed the pre-messages branch from 6fe6723 to 8140ebc Compare November 6, 2025 13:13

Simon-Laux force-pushed the simon/remove-partial-download-of-messages branch from 6c84e01 to 938c890 Compare November 6, 2025 13:22

Simon-Laux assigned Simon-Laux and unassigned Simon-Laux Nov 6, 2025

Simon-Laux mentioned this pull request Nov 8, 2025

[Branch] pre-messages / next version of download on demand #7371

Draft

Simon-Laux changed the title ~~remove: partial downloads - creation of the stub messages~~ remove: partial downloads (remove creation of the stub messages) Nov 8, 2025

iequidoo mentioned this pull request Nov 10, 2025

Don't prefetch Chat-Version header and don't copy it into outer part #7405

Draft

iequidoo reviewed Nov 10, 2025

View reviewed changes

Hocuri reviewed Nov 11, 2025

View reviewed changes

Simon-Laux mentioned this pull request Nov 12, 2025

feat: receive pre-messages and adapt download on demand #7431

Draft

15 tasks

Simon-Laux force-pushed the simon/remove-partial-download-of-messages branch from 506ecc8 to e564a9b Compare November 12, 2025 19:36

Simon-Laux and others added 9 commits November 12, 2025 20:37

remove: partial downloads

cca4174

remove partial download tests and fix issues in other tests

fdcb222

remove remaining partial messages tests

7f3eabc

remove download limit validation, it does not really matter in the new

2e2bc12

system as it is only ever applied to messages that have a pre-message.

fix clippy

902a0d7

ignore that partial_download_msg_body is dead code

70d7e04

remove test for download limit validation

2284d36

remove test_download_on_demand

53163ab

remove test test_download_limit_chat_assignment

cc9fe5d

Simon-Laux and others added 12 commits November 12, 2025 20:37

Remove superfluous parentheses.

d1dc322

fix the issue that broke the imap fetching

58ad664

fix python lint errors

014d80b

remove cffi python testtest_webxdc_download_on_demand

a044436

fix python lint

28cca8b

fix rebase issue

560e86b

fix another rebase issue

5a2ca7a

fix lint of python test

10c2016

reintroduce Config::FailOnReceivingFullMsg but renamed it to

37b027a

`Config::SimulateReceiveImfError`

allow exclusion of simulate_receive_imf_error from get_info

afa46c8

adapt comment

6d11e08

remove StockMessage::PartialDownloadMsgBody

05a1967

Simon-Laux force-pushed the simon/remove-partial-download-of-messages branch from e564a9b to 05a1967 Compare November 12, 2025 19:38

Uh oh!

remove: partial downloads (remove creation of the stub messages) #7373

Are you sure you want to change the base?

remove: partial downloads (remove creation of the stub messages) #7373

Conversation

Simon-Laux commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

removed partial download tests:

Uh oh!

Simon-Laux commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Simon-Laux Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Hocuri left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Simon-Laux commented Oct 30, 2025 •

edited

Loading

Simon-Laux commented Oct 30, 2025 •

edited

Loading

Simon-Laux Oct 31, 2025 •

edited

Loading

Hocuri left a comment •

edited

Loading