[Bug]: Minor issue: Running just the filter program with image_set=all produces plotting directory structure erroneously. #1076

SylviaWhittle · 2025-01-21T16:40:55Z

Checklist

Find the offending file in the output. If processing halts, re-run analysis with topostats --core 1 process.
Describe the bug.
Include the configuration file.
Copy of the log-file from running with topostats --log-level debug <command>.
The exact command that failed. This is what you typed at the command line, including any options.
TopoStats version, this is reported by topostats --version
Operating System and Python Version

Describe the bug

Just to emphasize, this is a really niche and minor bug.

When running the filter program on an image using a config file that has image_set=all, it produces the plotting output directory structure, despite this program not plotting anything.

See here the output folders for the different image sets:

image_set=core:

❯ tree output
output
├── config.yaml
└── processed
    └── minicircle_small_processed.topostats

2 directories, 2 files

image_set=all:

❯ tree output
output
├── config.yaml
└── processed
    ├── minicircle_small_processed
    │   ├── dnatracing
    │   │   ├── above
    │   │   │   ├── curvature
    │   │   │   └── nodes
    │   │   └── below
    │   │       ├── curvature
    │   │       └── nodes
    │   ├── filters
    │   └── grains
    │       ├── above
    │       └── below
    └── minicircle_small_processed.topostats

14 directories, 2 files

Copy of the log-file from running with `topostats --log-level debug <command>`

image_set=core:

❯ topostats -c config.yaml filter
[Tue, 21 Jan 2025 16:25:46] [INFO    ] [topostats] The YAML configuration file is valid.
[Tue, 21 Jan 2025 16:25:46] [INFO    ] [topostats] The YAML plotting configuration file is valid.
[Tue, 21 Jan 2025 16:25:46] [ERROR   ] [topostats] Splining enabled but Filters disabled. Please check your configuration file.
[Tue, 21 Jan 2025 16:25:46] [ERROR   ] [topostats] [processing.py] [1384] Splining enabled but Filters disabled. Please check your configuration file.
[Tue, 21 Jan 2025 16:25:46] [INFO    ] [topostats] Configuration file loaded from      : config.yaml
[Tue, 21 Jan 2025 16:25:46] [INFO    ] [topostats] Scanning for images in              : data
[Tue, 21 Jan 2025 16:25:46] [INFO    ] [topostats] Output directory                    : output
[Tue, 21 Jan 2025 16:25:46] [INFO    ] [topostats] Looking for images with extension   : .topostats
[Tue, 21 Jan 2025 16:25:46] [INFO    ] [topostats] Images with extension .topostats in data : 1
[Tue, 21 Jan 2025 16:25:46] [INFO    ] [topostats] Thresholding method (Filtering)     : std_dev
[Tue, 21 Jan 2025 16:25:46] [INFO    ] [topostats] Thresholding method (Grains)        : std_dev
[Tue, 21 Jan 2025 16:25:46] [INFO    ] [topostats] Extracting image from data/minicircle_small_processed.topostats
16:25:46 | INFO |topostats.py:topostats:load_topostats:38 | Loading image from : data/minicircle_small_processed.topostats
16:25:47 | INFO |topostats.py:topostats:load_topostats:46 | [minicircle_small_processed] TopoStats file version : 0.2
Processing images from data, results are under output:   0%|                                                      | 0/1 [00:00<?, ?it/s][Tue, 21 Jan 2025 16:25:50] [INFO    ] [topostats] Processing : minicircle_small_processed
[Tue, 21 Jan 2025 16:25:50] [ERROR   ] [topostats] You have not included running the initial filter stage. This is required for all subsequent stages of processing. Please check your configuration file.
[Tue, 21 Jan 2025 16:25:50] [ERROR   ] [topostats] [processing.py] [127] You have not included running the initial filter stage. This is required for all subsequent stages of processing. Please check your configuration file.
[Tue, 21 Jan 2025 16:25:50] [INFO    ] [topostats] [minicircle_small_processed] : Saving image to .topostats file
Processing images from data, results are under output: 100%|██████████████████████████████████████████████| 1/1 [00:03<00:00,  3.20s/it][Tue, 21 Jan 2025 16:25:50] [INFO    ] [topostats] [minicircle_small_processed] Filtering completed.
Processing images from data, results are under output: 100%|██████████████████████████████████████████████| 1/1 [00:03<00:00,  3.20s/it]


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


  _______      _____      __ __       _____     ______    _______      _____      _______    ______
/\_______)\   ) ___ (    /_/\__/\    ) ___ (   / ____/\ /\_______)\   /\___/\   /\_______)\ / ____/\
\(___  __\/  / /\_/\ \   ) ) ) ) )  / /\_/\ \  ) ) __\/ \(___  __\/  / / _ \ \  \(___  __\/ ) ) __\/
  / / /     / /_/ (_\ \ /_/ /_/ /  / /_/ (_\ \  \ \ \     / / /      \ \(_)/ /    / / /      \ \ \
 ( ( (      \ \ )_/ / / \ \ \_\/   \ \ )_/ / /  _\ \ \   ( ( (       / / _ \ \   ( ( (       _\ \ \
  \ \ \      \ \/_\/ /   )_) )      \ \/_\/ /  )____) )   \ \ \     ( (_( )_) )   \ \ \     )____) )
  /_/_/       )_____(    \_\/        )_____(   \____\/    /_/_/      \/_/ \_\/    /_/_/     \____\/


[Tue, 21 Jan 2025 16:25:50] [INFO    ] [topostats]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ COMPLETE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  TopoStats Version           : 2.3.1.dev84+g83aaed27d.d20250120
  Base Directory              : data
  File Extension              : .topostats
  Files Found                 : 1
  Successfully Processed^1    : 1 (100.0%)
  All statistics              : output/all_statistics.csv
  Distribution Plots          : Disabled. Enable in config 'summary_stats/run' if needed.

  Configuration               : output/config.yaml

  Email                       : topostats@sheffield.ac.uk
  Documentation               : https://afm-spm.github.io/topostats/
  Source Code                 : https://github.com/AFM-SPM/TopoStats/
  Bug Reports/Feature Request : https://github.com/AFM-SPM/TopoStats/issues/new/choose
  Citation File Format        : https://github.com/AFM-SPM/TopoStats/blob/main/CITATION.cff

  ^1 Successful processing of an image is detection of grains and calculation of at least
     grain statistics. If these have been disabled the percentage will be 0.

  If you encounter bugs/issues or have feature requests please report them at the above URL
  or email us.

  If you have found TopoStats useful please consider citing it. A Citation File Format is
  linked above and available from the Source Code page.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

image_set=all:

❯ topostats -c config.yaml filter
[Tue, 21 Jan 2025 16:14:34] [INFO    ] [topostats] The YAML configuration file is valid.
[Tue, 21 Jan 2025 16:14:34] [INFO    ] [topostats] The YAML plotting configuration file is valid.
[Tue, 21 Jan 2025 16:14:34] [ERROR   ] [topostats] Splining enabled but Filters disabled. Please check your configuration file.
[Tue, 21 Jan 2025 16:14:34] [ERROR   ] [topostats] [processing.py] [1384] Splining enabled but Filters disabled. Please check your configuration file.
[Tue, 21 Jan 2025 16:14:34] [INFO    ] [topostats] Configuration file loaded from      : config.yaml
[Tue, 21 Jan 2025 16:14:34] [INFO    ] [topostats] Scanning for images in              : data
[Tue, 21 Jan 2025 16:14:34] [INFO    ] [topostats] Output directory                    : output
[Tue, 21 Jan 2025 16:14:34] [INFO    ] [topostats] Looking for images with extension   : .topostats
[Tue, 21 Jan 2025 16:14:34] [INFO    ] [topostats] Images with extension .topostats in data : 1
[Tue, 21 Jan 2025 16:14:34] [INFO    ] [topostats] Thresholding method (Filtering)     : std_dev
[Tue, 21 Jan 2025 16:14:34] [INFO    ] [topostats] Thresholding method (Grains)        : std_dev
[Tue, 21 Jan 2025 16:14:34] [INFO    ] [topostats] Extracting image from data/minicircle_small_processed.topostats
16:14:34 | INFO |topostats.py:topostats:load_topostats:38 | Loading image from : data/minicircle_small_processed.topostats
16:14:34 | INFO |topostats.py:topostats:load_topostats:46 | [minicircle_small_processed] TopoStats file version : 0.2
Processing images from data, results are under output:   0%|                                                      | 0/1 [00:00<?, ?it/s][Tue, 21 Jan 2025 16:14:37] [INFO    ] [topostats] Processing : minicircle_small_processed
[Tue, 21 Jan 2025 16:14:37] [ERROR   ] [topostats] You have not included running the initial filter stage. This is required for all subsequent stages of processing. Please check your configuration file.
[Tue, 21 Jan 2025 16:14:37] [ERROR   ] [topostats] [processing.py] [127] You have not included running the initial filter stage. This is required for all subsequent stages of processing. Please check your configuration file.
[Tue, 21 Jan 2025 16:14:37] [INFO    ] [topostats] [minicircle_small_processed] : Saving image to .topostats file
Processing images from data, results are under output: 100%|██████████████████████████████████████████████| 1/1 [00:03<00:00,  3.29s/it][Tue, 21 Jan 2025 16:14:37] [INFO    ] [topostats] [minicircle_small_processed] Filtering completed.
Processing images from data, results are under output: 100%|██████████████████████████████████████████████| 1/1 [00:03<00:00,  3.29s/it]


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


  _______      _____      __ __       _____     ______    _______      _____      _______    ______
/\_______)\   ) ___ (    /_/\__/\    ) ___ (   / ____/\ /\_______)\   /\___/\   /\_______)\ / ____/\
\(___  __\/  / /\_/\ \   ) ) ) ) )  / /\_/\ \  ) ) __\/ \(___  __\/  / / _ \ \  \(___  __\/ ) ) __\/
  / / /     / /_/ (_\ \ /_/ /_/ /  / /_/ (_\ \  \ \ \     / / /      \ \(_)/ /    / / /      \ \ \
 ( ( (      \ \ )_/ / / \ \ \_\/   \ \ )_/ / /  _\ \ \   ( ( (       / / _ \ \   ( ( (       _\ \ \
  \ \ \      \ \/_\/ /   )_) )      \ \/_\/ /  )____) )   \ \ \     ( (_( )_) )   \ \ \     )____) )
  /_/_/       )_____(    \_\/        )_____(   \____\/    /_/_/      \/_/ \_\/    /_/_/     \____\/


[Tue, 21 Jan 2025 16:14:37] [INFO    ] [topostats]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ COMPLETE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  TopoStats Version           : 2.3.1.dev84+g83aaed27d.d20250120
  Base Directory              : data
  File Extension              : .topostats
  Files Found                 : 1
  Successfully Processed^1    : 1 (100.0%)
  All statistics              : output/all_statistics.csv
  Distribution Plots          : Disabled. Enable in config 'summary_stats/run' if needed.

  Configuration               : output/config.yaml

  Email                       : topostats@sheffield.ac.uk
  Documentation               : https://afm-spm.github.io/topostats/
  Source Code                 : https://github.com/AFM-SPM/TopoStats/
  Bug Reports/Feature Request : https://github.com/AFM-SPM/TopoStats/issues/new/choose
  Citation File Format        : https://github.com/AFM-SPM/TopoStats/blob/main/CITATION.cff

  ^1 Successful processing of an image is detection of grains and calculation of at least
     grain statistics. If these have been disabled the percentage will be 0.

  If you encounter bugs/issues or have feature requests please report them at the above URL
  or email us.

  If you have found TopoStats useful please consider citing it. A Citation File Format is
  linked above and available from the Source Code page.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Include the configuration file

Using partial config files (🎉)

image_set=core:

base_dir: ./data # Directory in which to search for data files
output_dir: ./output # Directory to output results to
log_level: info # Verbosity of output. Options: warning, error, info, debug
cores: 1 # Number of CPU cores to utilise for processing multiple files simultaneously.
file_ext: .topostats # File extension of the data files.
loading:
  channel: Height # Channel to pull data from in the data files.
  extract: all # Array to extract when loading .topostats files.
filter:
  run: false # Options : true, false
plotting:
  run: true # Options : true, false
  image_set: core # Options : all, core

image_set=all:

base_dir: ./data # Directory in which to search for data files
output_dir: ./output # Directory to output results to
log_level: info # Verbosity of output. Options: warning, error, info, debug
cores: 1 # Number of CPU cores to utilise for processing multiple files simultaneously.
file_ext: .topostats # File extension of the data files.
loading:
  channel: Height # Channel to pull data from in the data files.
  extract: all # Array to extract when loading .topostats files.
filter:
  run: false # Options : true, false
plotting:
  run: true # Options : true, false
  image_set: all # Options : all, core

To Reproduce

topostats -c config.yaml filter using the above image_set=all config. It'll produce the plot directory output structure.

TopoStats Version

Git main branch

Python Version

3.11

Operating System

GNU/Linux

Python Packages

N/A IMO

The text was updated successfully, but these errors were encountered:

ns-rse · 2025-01-21T20:55:01Z

I know why this is, and it's because we setup the directory structure up-front based on the loaded configuration which includes everything that is to be run, even if invocation is via the first of many new entry points in the #517 work.

Will need some refactoring of how and when to do this, shouldn't be too big a job.

Closes #743 Adds an entry point for running GrainStats `process grainstats --help`. You can test it with... ``` ❱ topostats -b tests/resources/test_image -f .topostats grainstats ❱ tree output [4.0K Feb 24 14:53] output ├── [2.7K Feb 24 14:53] output/config.yaml ├── [2.7K Feb 24 14:53] output/height_profiles.json ├── [1.7K Feb 24 14:53] output/image_stats.csv └── [4.0K Feb 24 14:53] output/processed └── [109K Feb 24 14:53] output/processed/minicircle_small.topostats 2 directories, 4 files ``` Unlike with #1076 only the required directory structure is created.

Closes #1076

SylviaWhittle added the bug label Jan 21, 2025

ns-rse mentioned this issue Feb 24, 2025

feature(grainstats): Adds entry point for grainstats #1094

Merged

5 tasks

ns-rse added a commit that referenced this issue Feb 24, 2025

fix(processing): create grains/dnatracing directories if needed

a9f115e

Closes #1076

ns-rse mentioned this issue Feb 24, 2025

fix(processing): create grains/dnatracing directories if needed #1095

Merged

2 tasks

ns-rse closed this as completed in #1095 Mar 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Minor issue: Running just the filter program with image_set=all produces plotting directory structure erroneously. #1076

[Bug]: Minor issue: Running just the filter program with image_set=all produces plotting directory structure erroneously. #1076

SylviaWhittle commented Jan 21, 2025

ns-rse commented Jan 21, 2025

[Bug]: Minor issue: Running just the filter program with image_set=all produces plotting directory structure erroneously. #1076

[Bug]: Minor issue: Running just the filter program with image_set=all produces plotting directory structure erroneously. #1076

Comments

SylviaWhittle commented Jan 21, 2025

Checklist

Describe the bug

Copy of the log-file from running with topostats --log-level debug <command>

Include the configuration file

To Reproduce

TopoStats Version

Python Version

Operating System

Python Packages

ns-rse commented Jan 21, 2025

Copy of the log-file from running with `topostats --log-level debug <command>`