sphobjinv-textconv #307

msftcangoblowm · 2024-12-31T12:30:12Z

closes #295

The rebase was challenging. Previous PR was automatically withdrawn

- feat: add sphobjinv-textconv entrypoint (bskinn#295)

- docs: adjustments to pass linkcheck

- ci(asure-pipelines): reduce coverage fail under percentage

- docs(README.md): doctest ELLIPSIS wildcard avoid hardcode inventory count

- feat: entrypoint sphobjinv-textconv ([bskinn#295]) - test: add sphobjinv-textconv unittests offline and online - test: add integration test. Demonstrate git diff objects.inv - test: add pytest module with class to interact with git - docs: add step by step guide to configure git. Examples for bash and python - docs: added sphobjinv-textconv API docs

- fix: py38 with statements multiple --> nested - fix: py38 standard types typing isms e.g. List - fix: use Windows friendly line seperators - test: print diagnostic info on Windows bin and site folders

- test: print entire os.environ rather than a single key

- test: for Windows, attempt add SCRIPTS folder to sys.path - test: for Windows, walk SCRIPTS folder print files and folders

- test: wrong params for list.append instead use list.insert

- test: for subprocess calls use relative path not absolute path

- test: carefully escape regex metacharacters - test: print source .inv file and diff. On windows, issue with regex - test: remove fixture windows_paths

- test: .git/config textconv executable resolve path - test: shutil.which to resolve path to executable

- test: resolve both soi and soi-textconv executables path

- test: regression when to use resolved or unresolved executable path

- test: read inventory on disk - test: print diagnostic before assertions

- test: consistantly use sphobjinv.cli.load:import_infile so compare apple with apples

- test: add resources objects_attrs_plus_one_entry.{txt|inv} - test: compare existing resources. Rather modify then .txt --> .inv - test: add fixtures res_cmp_plus_one_line is_linux gitconfig gitattributes

- refactor .git/config append algo

- refactor: print git diff err message

- fix: specify encoding and linesep

- test: fix use git config to set textconv executable - test: add to WorkDir git config list/get/set support

- test: fix inventory path must surround by single quotes

- test: attempt to diagnose WindowsPath getting garbled

- test: file not found create a .gitattributes

- ci: azure-pipelines coverage report omit setup.py - ci: azure-pipelines need --testall to get 100% coverage - ci: README.md affected by version. Update version - test: WorkDir - test: conftest fixture ensure_doc_scratch

- test: restore fixture ensure_doc_scratch

- chore: fix format and lint - docs: fix README doctest

msftcangoblowm · 2025-01-02T09:27:50Z

During python setup, gh runner failings to download python 3.13 for linux and windows.
Checked the versions manifest and 3.13 for x64 architecture are available.

The runner says could be caused by an authentication issue.

versions-manifest json

##[error]Failed to download Python from the Github Actions python registry python-versions. Error: Error: Failed to download Python versions manifest from the Github Actions python registry python-versions. Invalid GitHub token could cause this.

##[error]Version spec 3.13 for architecture x64 did not match any version in Agent.ToolsDirectory.

This is the relevent entries from the versions manifest. (to save time searching thru it)

{
        "filename": "python-3.13.1-linux-20.04-x64.tar.gz",
        "arch": "x64",
        "platform": "linux",
        "platform_version": "20.04",
        "download_url": "https://github.com/actions/python-versions/releases/download/3.13.1-12154081405/python-3.13.1-linux-20.04-x64.tar.gz"
},
{
        "filename": "python-3.13.1-linux-22.04-x64.tar.gz",
        "arch": "x64",
        "platform": "linux",
        "platform_version": "22.04",
        "download_url": "https://github.com/actions/python-versions/releases/download/3.13.1-12154081405/python-3.13.1-linux-22.04-x64.tar.gz"
      },
{
        "filename": "python-3.13.1-linux-24.04-x64.tar.gz",
        "arch": "x64",
        "platform": "linux",
        "platform_version": "24.04",
        "download_url": "https://github.com/actions/python-versions/releases/download/3.13.1-12154081405/python-3.13.1-linux-24.04-x64.tar.gz"
      },

{
        "filename": "python-3.13.1-win32-x64.zip",
        "arch": "x64",
        "platform": "win32",
        "download_url": "https://github.com/actions/python-versions/releases/download/3.13.1-12154081405/python-3.13.1-win32-x64.zip"
      },

bskinn · 2025-01-02T16:04:49Z

That same config worked recently for a different pipeline, so I don't think it's a config or token problem. More likely, there was just a transient issue with the GitHub Python repository. I'll trigger a retry; if it still fails, then let's just remove 3.13 from azure-pipelines.yml and not worry about it.

I'm working on an overhaul of the CI (#306) that will migrate everything over to GitHub Actions anyways, which has 3.13 natively available.

bskinn · 2025-01-02T17:01:15Z

No, still fails with the same errors. Just remove 3.13 from the Azure DevOps matrix.

msftcangoblowm · 2025-01-05T04:33:42Z

Yes please retrigger the pipeline. If needed and plan to remove of 3.13 from azure-pipelines.yml, please have me make a commit rather than causing more rebases to be necessary. Rebases are scary. Know the final code state, but not intern code states; what the code should look between `git rebase --continue` calls

…

On Thursday, January 2nd, 2025 at 4:05 PM, Brian Skinn ***@***.***> wrote: That same config [worked recently](https://dev.azure.com/brianskinn/sphobjinv/_build/results?buildId=573&view=logs&jobId=337cb850-2e3b-5931-1662-6e2a73a4cc60&j=337cb850-2e3b-5931-1662-6e2a73a4cc60&t=cdcb6bd4-70a6-5618-7395-9b57861e9608) for a different pipeline, so I don't think it's a config or token problem. More likely, there was just a transient issue with the GitHub Python repository. I'll trigger a retry; if it still fails, then let's just remove 3.13 from azure-pipelines.yml and not worry about it. I'm working on an overhaul of the CI ([#306](#306)) that will migrate everything over to GitHub Actions anyways, which has 3.13 natively available. — Reply to this email directly, [view it on GitHub](#307 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AVSZJ2ID6GG2CGARM7PENIL2IVPTRAVCNFSM6AAAAABUNNRT3KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRYGAYDONRTHE). You are receiving this because you authored the thread.Message ID: ***@***.***>

- chore: remove py313 from pipeline

msftcangoblowm · 2025-01-05T05:00:28Z

With py313 removed, the pipeline passes

…

On Sunday, January 5th, 2025 at 4:33 AM, msftcangoblowme ***@***.***> wrote: Yes please retrigger the pipeline. If needed and plan to remove of 3.13 from azure-pipelines.yml, please have me make a commit rather than causing more rebases to be necessary. Rebases are scary. Know the final code state, but not intern code states; what the code should look between `git rebase --continue` calls On Thursday, January 2nd, 2025 at 4:05 PM, Brian Skinn ***@***.***> wrote: > That same config [worked recently](https://dev.azure.com/brianskinn/sphobjinv/_build/results?buildId=573&view=logs&jobId=337cb850-2e3b-5931-1662-6e2a73a4cc60&j=337cb850-2e3b-5931-1662-6e2a73a4cc60&t=cdcb6bd4-70a6-5618-7395-9b57861e9608) for a different pipeline, so I don't think it's a config or token problem. More likely, there was just a transient issue with the GitHub Python repository. I'll trigger a retry; if it still fails, then let's just remove 3.13 from azure-pipelines.yml and not worry about it. > > I'm working on an overhaul of the CI ([#306](#306)) that will migrate everything over to GitHub Actions anyways, which has 3.13 natively available. > > — > Reply to this email directly, [view it on GitHub](#307 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AVSZJ2ID6GG2CGARM7PENIL2IVPTRAVCNFSM6AAAAABUNNRT3KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRYGAYDONRTHE). > You are receiving this because you authored the thread.Message ID: ***@***.***>

bskinn · 2025-01-05T15:28:44Z

Mm, I didn't think about the rebase being such a problem... merging main into your working branch would have probably worked equally well and been a lot less awful. Sorry about that!!

bskinn

Duplicating the review comments from #301

conftest.py

tests/resource/objects_attrs_plus_one_entry.txt

tests/resource/objects_attrs_plus_one_entry.inv

- test: remove inventory objects_attrs_plus_one_entry - test(conftest): remove res_cmp_plus_one_line - test(conftest): consolidate run_cmdline_textconv and run_cmdline_no_checks

msftcangoblowm · 2025-01-07T07:07:13Z

After a quick sync, the latest commit addresses the issues you brought up, besides # pragma: no cover comments.

conftest.py

bskinn

Several more comments across the PR. Still haven't had a chance to dig deeply into the new code itself, or the docs. Will get there.

conftest.py

.gitattributes

README.md

src/sphobjinv/version.py

tests/test_api_good.py

tests/test_cli.py

tests/test_fixture.py

- fix(version): revert do not change semantic version number - ci(azure-pipelines): remove pytest option --testall - test(conftest): remove fixture gitattributes - test(conftest): remove coverage pragma comments - test: revert to original pytest.skip conditional block

msftcangoblowm · 2025-01-12T11:02:31Z

Please rerun azure pipelines. Getting remotedisconnect and ssl errors

- test(conftest.py): remove run_cmdline_textconv by consolidate into run_cmdline_test - test(enum.py): add module for enum.Enum declarations

bskinn

Thanks for your patience as I nibble-review on this!

bskinn · 2025-01-15T05:20:06Z

tests/enum.py

Just realized, enum.py is a bad name, since it shadows stdlib enum.

Should rename this to enums.py, I think.

bskinn · 2025-01-15T05:20:46Z

tox.ini

+# cd .tox; PYTHONPATH=../ tox --root=.. -c ../tox.ini \
+# -e py39-sphx_latest-attrs_latest-jsch_latest --workdir=.; cd - &>/dev/null


Are these commented-out lines still needed?

Instantly forget complicated commands.

For collaborators, somewhere these commands should be documented. I prefer in context, so zero time is wasted recreating them.

non-tox commands should be in a Makefile.

pytest commands within the respective test module.

Have always written, just the complicated, tox commands within the tox file.

This particular command is how to invoke for pyenv users. The active .python-version file differs for: project base (one py version) and the .tox folder (many py versions)

bskinn · 2025-01-15T05:22:00Z

tox.ini

Let's revise the current last line to read:

application-import-names = sphobjinv, tests

That way, the linting will group the (to become) tests.enums imports with the sphobjinv imports.

bskinn · 2025-01-15T05:23:58Z

conftest.py

+@pytest.fixture(scope="session")
+def is_linux():
+    """Report boolean of whether the current system is Linux."""
+    return platform.system() in ("Linux",)
+
+


I don't see anywhere this fixture is currently used, do we need it?

Slated for removal

bskinn · 2025-01-15T05:27:36Z

.coveragerc

@@ -5,6 +5,7 @@ source =
 omit =
    # Don't worry about covering vendored libraries
    src/sphobjinv/_vendored/*
+    setup.py


Noticed this -- I'd be fine with adding conftest.py in here too, actually. It would both be a direct signal that setup.py and conftest.py are meant to not be covered, and I believe would result in them being excluded entirely from any coverage reports.

Oh nice! That's much better

msftcangoblowm added 27 commits December 31, 2024 12:19

sphobjinv-textconv

55e0d00

- feat: add sphobjinv-textconv entrypoint (bskinn#295)

adjustments to pass linkcheck

27153cd

- docs: adjustments to pass linkcheck

reduce fail under percentage

9d0c847

- ci(asure-pipelines): reduce coverage fail under percentage

fix README.md doctest

04bd1d3

- docs(README.md): doctest ELLIPSIS wildcard avoid hardcode inventory count

fix py38

e2cbd6a

- fix: py38 with statements multiple --> nested - fix: py38 standard types typing isms e.g. List - fix: use Windows friendly line seperators - test: print diagnostic info on Windows bin and site folders

kiss principle strikes back

2b8f4a9

- test: print entire os.environ rather than a single key

when need add SCRIPTS folder to sys.path

dab90f2

- test: for Windows, attempt add SCRIPTS folder to sys.path - test: for Windows, walk SCRIPTS folder print files and folders

meant list.insert

325697b

- test: wrong params for list.append instead use list.insert

use relative not absolute path

6f2a6b2

- test: for subprocess calls use relative path not absolute path

escape regex metacharacters

8bf128d

- test: carefully escape regex metacharacters - test: print source .inv file and diff. On windows, issue with regex - test: remove fixture windows_paths

.git/config textconv executable resolve path

2de2492

- test: .git/config textconv executable resolve path - test: shutil.which to resolve path to executable

resolve both executables path

3a1fa88

- test: resolve both soi and soi-textconv executables path

To be young resolved or unresolved

f898793

- test: regression when to use resolved or unresolved executable path

track down object count discrepency

a7c760b

- test: read inventory on disk - test: print diagnostic before assertions

consistantly use import_infile

920b0fb

- test: consistantly use sphobjinv.cli.load:import_infile so compare apple with apples

compare existing resources

bc36bc9

- test: add resources objects_attrs_plus_one_entry.{txt|inv} - test: compare existing resources. Rather modify then .txt --> .inv - test: add fixtures res_cmp_plus_one_line is_linux gitconfig gitattributes

.git/config algo refactor

9b6d7cf

- refactor .git/config append algo

git diff err message

a1450fa

- refactor: print git diff err message

specify encoding and linesep

88b188f

- fix: specify encoding and linesep

git config to the rescue

3578516

- test: fix use git config to set textconv executable - test: add to WorkDir git config list/get/set support

inventory path need single quotes

617b132

- test: fix inventory path must surround by single quotes

what garble WindowsPath

6011edd

- test: attempt to diagnose WindowsPath getting garbled

sandbox mystery no file .gitattributes

097aeb8

- test: file not found create a .gitattributes

ci adjustments need

17ae632

- ci: azure-pipelines coverage report omit setup.py - ci: azure-pipelines need --testall to get 100% coverage - ci: README.md affected by version. Update version - test: WorkDir - test: conftest fixture ensure_doc_scratch

restore fixture

cd36666

- test: restore fixture ensure_doc_scratch

fix format and lint

c28a4a5

- chore: fix format and lint - docs: fix README doctest

remove py313 from pipeline

695aa55

- chore: remove py313 from pipeline

bskinn reviewed Jan 5, 2025

View reviewed changes

msftcangoblowm and others added 2 commits January 7, 2025 05:00

Merge branch 'bskinn:main' into textconv

9b313d4

cleanup conftest

71b914e

- test: remove inventory objects_attrs_plus_one_entry - test(conftest): remove res_cmp_plus_one_line - test(conftest): consolidate run_cmdline_textconv and run_cmdline_no_checks

bskinn added type: enhancement ✨ Something to add area: cli 💻 function: convert 🧪 bump: minor 🐒 Backward-compatible feature adds (x.y.0) labels Jan 9, 2025

bskinn reviewed Jan 9, 2025

View reviewed changes

conftest.py Outdated Show resolved Hide resolved

bskinn reviewed Jan 12, 2025

View reviewed changes

one run_cmdline fixture

0a30a83

- test(conftest.py): remove run_cmdline_textconv by consolidate into run_cmdline_test - test(enum.py): add module for enum.Enum declarations

bskinn reviewed Jan 15, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sphobjinv-textconv #307

sphobjinv-textconv #307

msftcangoblowm commented Dec 31, 2024

msftcangoblowm commented Jan 2, 2025

bskinn commented Jan 2, 2025

bskinn commented Jan 2, 2025

msftcangoblowm commented Jan 5, 2025 via email

msftcangoblowm commented Jan 5, 2025 via email

bskinn commented Jan 5, 2025

bskinn left a comment

msftcangoblowm commented Jan 7, 2025

bskinn left a comment

msftcangoblowm commented Jan 12, 2025

bskinn left a comment

bskinn Jan 15, 2025

bskinn Jan 15, 2025

msftcangoblowm Jan 16, 2025

bskinn Jan 15, 2025

bskinn Jan 15, 2025

msftcangoblowm Jan 16, 2025

bskinn Jan 15, 2025

msftcangoblowm Jan 16, 2025

		# cd .tox; PYTHONPATH=../ tox --root=.. -c ../tox.ini \
		# -e py39-sphx_latest-attrs_latest-jsch_latest --workdir=.; cd - &>/dev/null

sphobjinv-textconv #307

Are you sure you want to change the base?

sphobjinv-textconv #307

Conversation

msftcangoblowm commented Dec 31, 2024

msftcangoblowm commented Jan 2, 2025

bskinn commented Jan 2, 2025

bskinn commented Jan 2, 2025

msftcangoblowm commented Jan 5, 2025 via email

msftcangoblowm commented Jan 5, 2025 via email

bskinn commented Jan 5, 2025

bskinn left a comment

Choose a reason for hiding this comment

msftcangoblowm commented Jan 7, 2025

bskinn left a comment

Choose a reason for hiding this comment

msftcangoblowm commented Jan 12, 2025

bskinn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment