feat: ignore files not in remote when push is false

[Northo]

Fixes #10317

Enables opt-in to remove push: false stage outputs from not_in_remote data status results.

Notable changes:

• Add outs_no_push to dvc.stage.utils.fill_stage_outputs keys, to facilitate making outputs with push: false.
• In status, when flag enabled, filter through files reported as not_in_remote, and remove them if not can_push.
• Add corresponding flag --respect-no-push flag to CLI

Open to suggestions on how to make the flag names more intuitive!

Corresponding PR for the docs: iterative/dvc.org#5373

• ❗ I have followed the Contributing to DVC checklist.

• 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

[codecov]

Codecov Report

Attention: Patch coverage is 93.10345% with 2 lines in your changes missing coverage. Please review.

Project coverage is 91.10%. Comparing base (2431ec6) to head (b5a6a58).
Report is 51 commits behind head on main.

Files with missing lines	Patch %	Lines
dvc/repo/data.py	86.66%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10749      +/-   ##
==========================================
+ Coverage   90.68%   91.10%   +0.41%     
==========================================
  Files         504      504              
  Lines       39795    39997     +202     
  Branches     3141     3160      +19     
==========================================
+ Hits        36087    36438     +351     
+ Misses       3042     2934     -108     
+ Partials      666      625      -41     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Northo]

@skshetry, have you had time to look at this? This feature would be really great for our team!

[skshetry]

I think we can enable this by default. dvc status does the same.

See:

• import-url: disable push by default for cloud-versioned imports #8578 (comment)

[skshetry]

I think we have to extract out the "not_in_remote" checks from above _diff(), as it has nothing to do with diff between worktree and committed changes (index). We need to calculate only for the given index (commited changes).

That is what repo.index is. It's an index of committed changes. You can filter that index to a view, using worktree_view:

dvc/dvc/repo/worktree.py

Line 63 in 549821e

def worktree_view(

if not_in_remote:
    view = worktree_view(repo.index, push=True)
    # ... existing logic

push=True gives us

dvc/dvc/repo/worktree.py

Line 73 in 549821e

push: Whether the view should be restricted to pushable data only.

.

You can get access to DataIndex using view.data["repo"]. And then use index.iteritems(shallow=not granular) on it.

Let me know if you need help. I can take over the PR if you prefer that way.

[skshetry]

You can extract the above not_in_remote stuff. It is using Change type, but you can make minor modification to it to instead use Entry type. change.old and change.new are just Entry type.

Something like follows, maybe:

data_index = view.data["repo"]
for key, entry in data_index.iteritems(shallow=not granular):
    if not (entry and entry.hash_info):
        continue

    k = (*key, "") if entry.meta and entry.meta.isdir else key
    try:
        if not index.storage_map.remote_exists(entry, refresh=remote_refresh):
            yield os.path.sep.join(k)
    except StorageError:
        pass

[skshetry]

Let's not add these please. These are also used in CLIs.

I'll suggest a different way to create a stage in tests below with push=False.

[skshetry]

This is a bit verbose, but it's better than adding outs_no_push arg to stage.add() function.

Suggested change

	dvc.stage.add(
	name="create-foo", cmd="echo foo > foo", deps=["fixed"], outs_no_push=["foo"]
	)
	stage = dvc.stage.create(
	name="create-foo", cmd="echo foo > foo", deps=["fixed"], outs=["foo"]
	)
	stage.outs[0].can_push = False
	stage.dump()

[skshetry]

Btw, you don't have to create a pipeline stage, which can be slower.

Suggested change

	dvc.stage.add(
	name="create-foo", cmd="echo foo > foo", deps=["fixed"], outs_no_push=["foo"]
	)
	stage = dvc.stage.create(single_stage=True, outs=["foo"])
	stage.outs[0].can_push = False
	stage.dump()

[skshetry]

Please move all of these steps into the test itself. If we get rid of --respect-no-push, we'd only need one single test.