Skip to content

Commit 828049e

Browse files
claynerobisondependabot[bot]sramakintelMahathi-Vatsaljitendra42
committed
Crobiso1 poetry2 (#2656)
* Bump github/codeql-action from 3.28.5 to 3.28.8 in /.github/workflows (#2611) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.28.5 to 3.28.8. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v3.28.5...v3.28.8) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Bump github/codeql-action from 3.28.8 to 3.28.9 in /.github/workflows (#2616) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.28.8 to 3.28.9. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v3.28.8...v3.28.9) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump transformers in /models_v2/pytorch/stable_diffusion/inference/gpu (#2619) Bumps [transformers](https://github.com/huggingface/transformers) from 4.38.1 to 4.48.0. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](huggingface/transformers@v4.38.1...v4.48.0) --- updated-dependencies: - dependency-name: transformers dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Srikanth Ramakrishna <[email protected]> * Bump transformers in /models_v2/pytorch/bert_large/inference/gpu (#2629) Bumps [transformers](https://github.com/huggingface/transformers) from 4.38.0 to 4.48.0. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](huggingface/transformers@v4.38.0...v4.48.0) --- updated-dependencies: - dependency-name: transformers dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Srikanth Ramakrishna <[email protected]> Co-authored-by: mahathis <[email protected]> * Bump ossf/scorecard-action from 2.4.0 to 2.4.1 in /.github/workflows (#2632) Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 2.4.0 to 2.4.1. - [Release notes](https://github.com/ossf/scorecard-action/releases) - [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md) - [Commits](ossf/scorecard-action@v2.4.0...v2.4.1) --- updated-dependencies: - dependency-name: ossf/scorecard-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Bump github/codeql-action from 3.28.9 to 3.28.10 in /.github/workflows (#2633) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.28.9 to 3.28.10. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v3.28.9...v3.28.10) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Address Coverity Issues (#2644) * add net drop * add net_drop to all dockerfiles * add for rn50 * address lint errors Signed-off-by: Srikanth Ramakrishna <[email protected]> * remove cloud data connector. no longer needed & supported (#2622) * add license to gpt-j mlperf files (#2648) * add license to gpt-j mlperf files * replace w/ apache * poetry conversion stage 1 * take 1 * take 2 * still wont build * tf rn50v1.5 trainging gpu poetry support complete. Proxy env vars now need to be passed as in docker-compose --env-file parameter * fix unit-test.yaml * fix script path * add poetry activate * more Poetry support * adding poetry path to Makefile * Get runner OS version and update python to 3.10 * fix Makefile * setting path to poetry in Makefile * Fix syntax * accomodate sh * fix * I hate makefiles * take 15 * ugh * progress * fix tox.ini * move dependency managment back to workflow; expand flake8 tests * remove linting for old models directory * remove models_v2 folder for now * fix linting issues * linting errors in notebooks folder * fix formatting and update Intel® oneAPI Base Toolkit link in profile_utils.py * fix indentation in hardware selection logic in profile_utils.py * update tox.ini to rename pytest environment to py3-py.test * add mock dependency to pytest environment in tox.ini * update pytest environment in tox.ini to include poetry installation steps * refactor pytest environment in tox.ini to separate dependencies for clarity * removing test_launch_benchmark because it is obsolete. * simplify omit patterns in tox.ini to exclude all benchmarks * update coverage threshold and refine omit patterns in tox.ini * add entries for ai.reference.models in ipldt_results-resolve-false-hits.db --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Srikanth Ramakrishna <[email protected]> Co-authored-by: mahathis <[email protected]> Co-authored-by: Jitendra Patil <[email protected]>
1 parent 5455e70 commit 828049e

File tree

12 files changed

+2867
-444
lines changed

12 files changed

+2867
-444
lines changed

Makefile

Lines changed: 32 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -13,45 +13,65 @@
1313
# limitations under the License.
1414
#
1515

16+
# Check if poetry is available
17+
POETRY_EXISTS := $(shell command -v poetry >/dev/null 2>&1 && echo yes || echo no)
1618

17-
PY_VERSION := 2
1819
# if py_version is 2, use virtualenv, else python3 venv
1920
VIRTUALENV_EXE=$(if $(subst 2,,$(PY_VERSION)),python3 -m venv,virtualenv)
20-
VIRTUALENV_DIR=$(if $(subst 2,,$(PY_VERSION)),.venv3,.venv)
21-
ACTIVATE="$(VIRTUALENV_DIR)/bin/activate"
21+
#VIRTUALENV_DIR=$(if $(subst 2,,$(PY_VERSION)),.venv3,.venv)
22+
VIRTUALENV_DIR=$(shell poetry env info --path)
23+
VIRTUAL_ENV_ACTIVATE="$(VIRTUALENV_DIR)/bin/activate"
2224

2325
.PHONY: venv
2426

25-
all: venv lint unit_test
27+
all: check-poetry venv lint unit_test
28+
29+
check-poetry:
30+
ifeq ($(POETRY_EXISTS),yes)
31+
@echo "Poetry is installed. Ready to go!"
32+
else
33+
@echo "Poetry is not installed. Please install it first."
34+
# Optionally, you can add commands to handle the absence of poetry
35+
endif
36+
37+
2638

2739
# we need to update pip and setuptools because venv versions aren't latest
2840
# need to prepend $(ACTIVATE) everywhere because all make calls are in subshells
2941
# otherwise we won't be installing anything in the venv itself
3042
$(ACTIVATE):
31-
@echo "Updating virtualenv dependencies in: $(VIRTUALENV_DIR)..."
32-
@test -d $(VIRTUALENV_DIR) || $(VIRTUALENV_EXE) $(VIRTUALENV_DIR)
33-
@. $(ACTIVATE) && python -m pip install -r requirements.txt
34-
@touch $(ACTIVATE)
43+
. ~/.bashrc && echo "Updating virtualenv dependencies in: $(VIRTUALENV_DIR)..."
44+
ifeq ($(POETRY_EXISTS),yes)
45+
@echo "Poetry is installed. Ready to go!"
46+
VIRTUALENV_DIR=$(shell poetry env info --path)
47+
else
48+
@echo "Poetry is not installed. Please install it first."
49+
VIRTUALENV_DIR=$(if $(subst 2,,$(PY_VERSION)),.venv3,.venv)
50+
python3 -m venv $(VIRTUALENV_DIR)
51+
endif
52+
@echo "Activating virtualenv..."
53+
@. $(VIRTUAL_ENV_ACTIVATE)
54+
@touch $(VIRTUAL_ENV_ACTIVATE)
3555

3656
venv: $(ACTIVATE)
3757
@echo -n "Using "
38-
@. $(ACTIVATE) && python --version
58+
@. $(VIRTUAL_ENV_ACTIVATE) && python --version
3959

4060
venv3: PY_VERSION=3
4161
venv3: $(ACTIVATE)
4262
@echo -n "Using "
43-
@. $(ACTIVATE) && python3 --version
63+
@. $(VIRTUAL_ENV_ACTIVATE) && python3 --version
4464

4565
tox:
4666
tox
4767

4868
lint:
4969
@echo "Running style check..."
50-
tox -e py3-flake8
70+
@. $(VIRTUAL_ENV_ACTIVATE) && tox -e py3-flake8
5171

5272
unit_test:
5373
@echo "Running unit tests..."
54-
tox -e py3-py.test
74+
@. $(VIRTUAL_ENV_ACTIVATE) && tox -e py3-py.test
5575

5676
test_tl_tf_notebook: venv
5777
@. $(ACTIVATE) && poetry install --file docs/notebooks/transfer_learning/pyproject.toml && \

docker/tensorflow/docker-compose.yml

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,6 @@
1515
# See the License for the specific language governing permissions and
1616
# limitations under the License.
1717
#
18-
19-
#
20-
21-
version: '3'
2218
services:
2319
resnet50v1_5-inference-gpu:
2420
build:
@@ -28,6 +24,10 @@ services:
2824
TF_BASE_TAG: ${TF_BASE_TAG:-2.15.0.1-xpu}
2925
no_proxy: ''
3026
NO_PROXY: ''
27+
https_proxy: '${https_proxy}'
28+
http_proxy: '${http_proxy}'
29+
HTTP_PROXY: '${HTTP_PROXY}'
30+
HTTPS_PROXY: '${HTTPS_PROXY}'
3131
INT8_MODEL_URL: ${INT8_MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/resnet50_v1_int8.pb}
3232
MODEL_URL: ${MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/3_1/resnet50_v1.pb}
3333
dockerfile: docker/tensorflow/resnet50v1_5/inference/gpu/tf-gpu-resnet50v1-5-inference.Dockerfile
@@ -69,6 +69,12 @@ services:
6969
args:
7070
MPI_VER: ${MPI_VER:-2021.13.1-767}
7171
CCL_VER: ${CCL_VER:-2021.13.1-31}
72+
no_proxy: ''
73+
NO_PROXY: ''
74+
https_proxy: '${https_proxy}'
75+
http_proxy: '${http_proxy}'
76+
HTTP_PROXY: '${HTTP_PROXY}'
77+
HTTPS_PROXY: '${HTTPS_PROXY}'
7278
dockerfile: docker/tensorflow/resnet50v1_5/training/gpu/tf-max-series-resnet50v1-5-training.Dockerfile
7379
extends: resnet50v1_5-inference-gpu
7480
image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-max-gpu-resnet50v1-5-training

docker/tensorflow/resnet50v1_5/training/gpu/tf-max-series-resnet50v1-5-training.Dockerfile

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,13 +31,14 @@ RUN apt-get update && \
3131
intel-oneapi-ccl=${CCL_VER} && \
3232
rm -rf /var/lib/apt/lists/*
3333

34+
RUN curl -sSL https://install.python-poetry.org | python3 -
3435
WORKDIR /workspace/tf-max-series-resnet50v1-5-training/models
3536

3637
COPY models_v2/tensorflow/resnet50v1_5/training/gpu .
3738

38-
RUN pip install -r requirements.txt
39+
RUN /root/.local/bin/poetry install
3940

40-
RUN python -m pip install --no-cache-dir intel-optimization-for-horovod
41+
RUN /root/.local/bin/poetry add intel-optimization-for-horovod
4142

4243
RUN mkdir -p resnet50 && \
4344
cd resnet50 && \

models_v2/tensorflow/resnet50v1_5/training/gpu/poetry.lock

Lines changed: 1252 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
[project]
2+
name = "models-v2-tensorflow-resnet50v1-5-training-gpu"
3+
version = "3.4"
4+
description = "AIRM - models_v2-tensorflow-resnet50v1_5-training-gpu"
5+
authors = [
6+
{name = "Intel(R) AI Reference Models",email = "[email protected]"}
7+
]
8+
readme = "README.md"
9+
requires-python = ">=3.10,<=3.11"
10+
dependencies = [
11+
"gin (>=0.1.6,<0.2.0)",
12+
"gin-config (>=0.5.0,<0.6.0)",
13+
"intel-optimization-for-horovod (==0.28.1.4)",
14+
"protobuf (==3.20.3)",
15+
"pyyaml (>=6.0.2,<7.0.0)",
16+
"tensorflow-datasets (>=4.9.7,<5.0.0)",
17+
"tensorflow-model-optimization (>=0.8.0,<0.9.0)",
18+
"tf-keras (==2.15.0)",
19+
"tfa-nightly (>=0.23.0.dev20240415222534,<0.24.0)"
20+
]
21+
22+
[tool.poetry]
23+
package-mode = false
24+
25+
[build-system]
26+
requires = ["poetry-core>=2.0.0,<3.0.0"]
27+
build-backend = "poetry.core.masonry.api"

notebooks/profiling/profile_utils.py

Lines changed: 33 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,6 @@
1919
#
2020
import sys
2121
import configparser
22-
import json
2322
import os
2423
import fnmatch
2524
import psutil
@@ -42,14 +41,15 @@
4241
has_git = False
4342
pass
4443

44+
4545
class GitOps:
4646

4747
def __init__(self, repopath='./'):
4848
self.repopath = repopath
4949
if has_git is True:
5050
try:
5151
self.repo = Repo(repopath)
52-
except:
52+
except Exception:
5353
print('no repo, init one')
5454
self.repo = Repo.init(repopath)
5555
self.repo.git.add(A=True)
@@ -70,7 +70,7 @@ def find_commit_with_keyword(self, keyword, search_commits_depth=5):
7070
return ret
7171
try:
7272
repo = Repo(self.repopath)
73-
except:
73+
except Exception:
7474
print('no repo, init one')
7575
repo = Repo.init(self.repopath)
7676
repo.git.add(A=True)
@@ -126,6 +126,7 @@ def dump_platform_info(self):
126126
self.cpu_socket_count = cpu_info.sockets
127127
self.svmem = svmem
128128

129+
129130
class CommonUtils:
130131

131132
def __init__(self):
@@ -150,9 +151,10 @@ def found_files_in_folders(self, pattern, paths):
150151
foundpaths += paths
151152
return foundfiles, foundpaths
152153

154+
153155
class GeneralConfigFile:
154156
def __init__(self, ai_root):
155-
self.root=ai_root
157+
self.root = ai_root
156158
self.framework = ''
157159
self.device = ''
158160
# Empty for CPU
@@ -196,7 +198,6 @@ def __init__(self, ai_root):
196198
disabled=False,
197199
)
198200

199-
200201
def toggle_events(self):
201202
def on_hardware_change(change):
202203
if change.new == 'GPU':
@@ -238,16 +239,22 @@ def on_intel_oneapi_change(change):
238239
if change.new == 'No':
239240
# Intel® oneAPI Base Toolkit is not installed
240241
print("Intel® oneAPI Base Toolkit is not installed.")
241-
print("Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.")
242+
print(
243+
"Follow instructions at [Intel® oneAPI Base Toolkit Download page]"
244+
"(https://www.intel.com/content/www/us/en/developer/tools/oneapi"
245+
"/base-toolkit-download.html?operatingsystem=linux) "
246+
"to setup the package manager repository."
247+
)
242248
print("Once Intel® oneAPI Base Toolkit is installed on the machine, please re-run this cell")
243249
return
244250

245-
246251
self.hardware_dropdown.observe(on_hardware_change, names='value')
247252
self.framework_dropdown.observe(on_framework_change, names='value')
248253
self.intel_oneapi_dropdown.observe(on_intel_oneapi_change, names='value')
249254

250-
if (self.hardware_dropdown.value and self.framework_dropdown.value and (self.hardware_dropdown.value == 'CPU' or (self.hardware_dropdown.value == 'GPU' and self.gpu_series_dropdown.value))):
255+
if (self.hardware_dropdown.value and self.framework_dropdown.value and
256+
(self.hardware_dropdown.value == 'CPU' or
257+
(self.hardware_dropdown.value == 'GPU' and self.gpu_series_dropdown.value))):
251258
self.success = True
252259
display(self.hardware_dropdown)
253260
display(self.framework_dropdown)
@@ -267,7 +274,7 @@ def __init__(self, confpath, AIpath):
267274
self.device = ''
268275
self.precision = ''
269276
self.test_mode = ''
270-
#self.ai_type = ''
277+
# self.ai_type = ''
271278
self.custom_args = ''
272279
self.json_fname = ''
273280
self.json_fname = 'stock_'
@@ -372,7 +379,7 @@ def read_config(self, model_name):
372379
for key, value in model_data.items():
373380
if key == 'name':
374381
configs['name'] = value
375-
#elif key == 'ai-type':
382+
# elif key == 'ai-type':
376383
# configs['ai-type'] = value
377384
elif key == 'model-name':
378385
configs['model-name'] = value
@@ -398,7 +405,10 @@ def read_config(self, model_name):
398405
wget = sub_entry['wget']
399406
if model_name.split()[0] not in configs:
400407
configs[model_name.split()[0]] = []
401-
configs[model_name.split()[0]].append({'precision': precision, 'test_mode': test_mode, 'script': scripts, 'wget': wget})
408+
configs[model_name.split()[0]].append({'precision': precision,
409+
'test_mode': test_mode,
410+
'script': scripts,
411+
'wget': wget})
402412

403413
return configs
404414

@@ -489,7 +499,7 @@ def model_selection(self):
489499

490500
for index, section in enumerate(sections):
491501
model_name = section.get('model-name', 'Unknown')
492-
#ai_type = section.get('ai-type', 'Unknown')
502+
# ai_type = section.get('ai-type', 'Unknown')
493503
framework = section.get('framework', 'Unknown')
494504
mode = section.get('mode', 'Unknown')
495505
device = section.get('device', 'Unknown')
@@ -498,7 +508,8 @@ def model_selection(self):
498508
print("Supported Models: ")
499509
display(models_table)
500510

501-
model_index = int(input('Input an index number of a model: ')) if 'MODEL_1_INDEX' not in os.environ else int(os.environ['MODEL_1_INDEX'])
511+
model_index = int(input('Input an index number of a model: ')) \
512+
if 'MODEL_1_INDEX' not in os.environ else int(os.environ['MODEL_1_INDEX'])
502513

503514
if not 0 <= model_index < len(sections):
504515
raise Exception("Invalid choice for model index")
@@ -513,14 +524,16 @@ def device_specific(self, model_section, device, yaml_file, framework):
513524
# Set initial properties
514525
self.model_name = model_section['name']
515526
self.framework = model_section['framework']
516-
self.mode = model_section['mode'][int(input('0 for training and 1 for inference: '))] if len(model_section['mode']) > 1 else model_section['mode'][0]
527+
self.mode = model_section['mode'][int(input('0 for training and 1 for inference: '))] \
528+
if len(model_section['mode']) > 1 else model_section['mode'][0]
517529

518530
# Load model-specific configuration
519531
model_specific_config = AIReferenceConfigFile(yaml_file, self.ai_root)
520532
model_specific_section = model_specific_config.read_supported_section()[0]
521533

522534
# Determine the precision options key dynamically
523-
precision_key = model_section.get('model-name', self.model_name.split()[0]) # Use 'model-name' if available, else fallback
535+
# Use 'model-name' if available, else fallback
536+
precision_key = model_section.get('model-name', self.model_name.split()[0])
524537

525538
# Get precision options
526539
model_precisions = model_specific_section.get(precision_key)
@@ -564,7 +577,9 @@ def device_specific(self, model_section, device, yaml_file, framework):
564577
# Non-TensorFlow-specific handling
565578
model_precision_test_mode = model_precision_section.get('test_mode', [])
566579
if not model_precision_test_mode:
567-
raise ValueError(f"No test modes found for model '{self.model_name}' with precision '{self.precision}'.")
580+
raise ValueError(
581+
f"No test modes found for model '{self.model_name}' with precision '{self.precision}'."
582+
)
568583
print(f"Available Test Modes for {self.precision}:")
569584
model_test_mode_table = PrettyTable(["Index", "Test Mode"])
570585
for index, test_mode in enumerate(model_precision_test_mode):
@@ -578,7 +593,7 @@ def device_specific(self, model_section, device, yaml_file, framework):
578593
self.test_mode = model_precision_test_mode[model_precision_test_mode_index]
579594

580595
# Print selected configuration
581-
print(f"Selected {self.model_name} {self.framework} {self.precision} {self.script if framework == 'TensorFlow' else self.test_mode}")
596+
print(f"Selected {self.model_name} {self.framework} {self.precision} "
597+
"{self.script if framework == 'TensorFlow' else self.test_mode}")
582598

583599
return model_specific_section
584-

0 commit comments

Comments
 (0)