Update 2022 05 03 #317

kba · 2022-05-03T13:24:51Z

contains #314 (but with ocrd_cis in the main venv), #315, #316 plus the regular updates to the processor projects.

Move ocrd_detectron2 to headless-tf1 to avoid conflicts in the main virtual environment. Signed-off-by: Stefan Weil <[email protected]>

Move ocrd_cis to headless-tf1 to avoid a conflict with ocrd_calamari. Signed-off-by: Stefan Weil <[email protected]>

Signed-off-by: Stefan Weil <[email protected]>

Makefile uses GNU parallel semaphores not only for git but also for pip, but the old rule only cleaned the former ones. Signed-off-by: Stefan Weil <[email protected]>

Signed-off-by: Stefan Weil <[email protected]>

…22-05-03

stweil

Thank you.

stweil · 2022-05-03T13:33:43Z

CircleCI fails again because https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 is broken. That's unrelated to the changes here, and all we can do is wait and hope that it will work again soon. Maybe repeating the check is sufficient.

kba · 2022-05-03T13:57:57Z

CircleCI fails again because https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 is broken. That's unrelated to the changes here, and all we can do is wait and hope that it will work again soon. Maybe repeating the check is sufficient.

Yeah, I am debugging this right now. The docker image for the CUDA docker images have been updated yesterday, I assume with a fix to the issue with the signatures. However, our ocrd/core-cuda image needs rebuilding, which currently fails

apt-get -y install --no-install-recommends cuda-runtime-10-0 cuda-runtime-10-1 cuda-runtime-10-2 cuda-runtime-11-0 cuda-runtime-11-1 cuda-runtime-11-3 libcudnn7
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package libcudnn7

stweil · 2022-05-03T14:40:13Z

@kba, it is fixed by running curl https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub | sudo apt-key add -.

kba · 2022-05-03T14:46:06Z

curl https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub | sudo apt-key add -.

The problem is that we need to fix it in the ocrd/core-cuda image, which I am trying to rebuild. That is in turn based on the updated nvidia/cuda:11.3.1-cudnn8-runtime-ubuntu18.04, which does not have the GPG key problem but a new one:

apt-get -y install --no-install-recommends cuda-runtime-10-0 cuda-runtime-10-1 cuda-runtime-10-2 cuda-runtime-11-0 cuda-runtime-11-1 cuda-runtime-11-3 libcudnn7             
Reading package lists...                                                                                                                                    
Building dependency tree...                                                                                                                                                  
Reading state information...                                                                                                                                                 
E: Unable to locate package libcudnn7

stweil · 2022-05-03T15:20:02Z

That problem is gone as soon as you have installed the key from https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub and run apt update.

kba · 2022-05-03T15:25:01Z

That problem is gone as soon as you have installed the key from https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/3bf863cc.pub and run apt update.

As I said, that GPG key problem does not occur with the docker image, that has been fixed upstream AFAICT. However, this updated nvidia/cuda:11.3.1-cudnn8-runtime-ubuntu18.04 does not contain libcudnn7 anymore, only libcudnn8. Neither @bertsky nor myself were sure why we even had that in. So I removed it, rebuilt and uploaded ocrd/core-cuda and restarted the build nvidia/cuda:11.3.1-cudnn8-runtime-ubuntu18.04 and let's all cross fingers that this fixes the build and does not cause new problems 🤞

kba · 2022-05-03T16:33:13Z

It failed again, but this time at the make check stage: https://app.circleci.com/pipelines/github/OCR-D/ocrd_all/794/workflows/a912a236-0452-43c5-b3e0-d67fa69c25c0/jobs/975 If anybody sees the problem, I'd appreciate any hints. Why is tensorflow 2.4.4 installed into the tf1 sub-venv?

bertsky · 2022-05-03T16:39:39Z

By updating sbb_binarization to its current head, you dragged in TF2 into headless-tf1.

bertsky · 2022-05-03T16:42:21Z

@kba I therefore suggest moving sbb_binarization and eynollah into top level venv (and removing their tf1nvidia recipe line).

bertsky · 2022-05-03T18:10:55Z

Alas,

tensorflow 2.5.0 has requirement tensorflow-estimator<2.6.0,>=2.5.0rc0, but you have tensorflow-estimator 2.4.0.

The problem again comes from the fixed (freezed / pip-tool generated) requirements.txt in ocrd_pc_segmentation. TF 2.6 requires tensorflow-estimator 2.6, but ocr4all-pixel-classifier insists on TF 2.5.

We have a few choices here:

asking @crater2150 to update his ocr4all-pixel-classifier/requirements.in once again – in this case removing the upper end in

tensorflow >= 2.0.0, <= 2.5.0

moving ocrd_pc_segmentation to its own dedicated venv (which will cost at least 2 GB extra, all the benefits of the recent changes go to waste)
disabling ocrd_pc_segmentation for the time being, since AFAICT no-one has actually been using it in OCR-D (please correct me, @chreul @maxnth)

cneud · 2022-05-03T18:36:39Z

My prefered solution would be (3) for the benefit of being able to flexibly make a new ocrd_all release quickly without depending on ocrd_pc_segmentation. We can re-activate it later if we see the need or a fix has been implemented. The drawback is that we then also have to make this clear in the documentation (it seems to have only this mention though - and we have to remove sbb-textline-detector there also anyway).

bertsky · 2022-05-03T18:43:07Z

Okay, so just to see how we would fare with that choice in CI, I added a commit disabling ocrd_pc_segmentation for now. (We can drop it from the PR if it does not help or there is no consensus.)

Besides the WF guide, there's also a markdown checkbox to be unchecked in the README here.

bertsky · 2022-05-03T19:35:49Z

Ah, limitless joy – here comes another surprise from the TF trolls for us:

ImportError: cannot import name 'LayerNormalization'

(failing in ocrd-calamari-recognize, ocrd-anybaseocr-layout-analysis, ocrd-anybaseocr-tiseg, ocrd-eynollah-segment, ocrd-sbb-binarize)
I'll investigate ways out of this via SSH...

bertsky · 2022-05-03T20:23:55Z

I can immediately see multiple problems here:

eynollah and sbb_binarization depend on tensorflow-gpu (newest: v2.4.4), while the other TF2 modules use tensorflow (newest: v2.6.2). pip is still dumb-ass and allows both but cripples the variant that gets installed first.
sbb_binarization still constraints h5py < 3 which conflicts with h5py requirements for TF2 (and should only be relevant for TF1 IIRC)
ocrd_anybaseocr still depends on keras (v2.6.0), which should not be necessary in TF2

stweil · 2022-05-03T20:28:13Z

sbb_binarization still constraints h5py < 3 which conflicts with h5py requirements for TF2 (and should only be relevant for TF1 IIRC)

That's the current show stopper, see my comment there.

cneud · 2022-05-03T20:28:16Z

eynollah and sbb_binarization depend on tensorflow-gpu (newest: v2.4.4), while the other TF2 modules use tensorflow (newest: v2.6.2). pip is still dumb-ass and allows both but cripples the variant that gets installed first.

sbb_binarization still constraints h5py < 3 which conflicts with h5py requirements for TF2 (and should only be relevant for TF1 IIRC)

Oops. We'll look into those asap.

stweil · 2022-05-03T20:33:44Z

eynollah and sbb_binarization depend on tensorflow-gpu, while the other TF2 modules use tensorflow

It seems that meanwhile it is possible to install both tensorflow and tensorflow-gpu (with compatible version 2.x.y) from PyPI at the same time without any conflict, but I am not sure whether that is reasonable or makes a difference.

stweil · 2022-05-03T20:41:01Z

Allthough it would be good if ocrd_pc_segmentation could relax the upper limit tensorflow<=2.5.0, we can currently build with tensorflow==2.5.0. So there is no need to remove it. I removed the commit which did that.

bertsky · 2022-05-03T20:49:18Z

eynollah and sbb_binarization depend on tensorflow-gpu, while the other TF2 modules use tensorflow

It seems that meanwhile it is possible to install both tensorflow and tensorflow-gpu (with compatible version 2.x.y) from PyPI at the same time without any conflict, but I am not sure whether that is reasonable or makes a difference.
Allthough it would be good if ocrd_pc_segmentation could relax the upper limit tensorflow<=2.5.0, we can currently build with tensorflow==2.5.0. So there is no need to remove it.

No, it does not – I just explained it above!

I removed the commit which did that.

...and one step backwards, again – I'm out.

bertsky · 2022-05-03T20:51:00Z

force-push removing while others are already at the job – @stweil!!!

chreul · 2022-05-04T06:36:43Z

[...] ocr4all-pixel-classifier [...]
3. disabling ocrd_pc_segmentation for the time being, since AFAICT no-one has actually been using it in OCR-D (please correct me, @chreul @maxnth)

ocr4all-pixel-classifier is not directly related to ocr4all. we do not use ocrd_pc_segmentation and do not intend to do so in the future.

stweil · 2022-05-04T06:49:03Z

I don't object removing OCR-D processors which nobody uses. Technically there is no need to remove ocrd_pc_segmentation.

@kba, I suggest to merge this PR – either with or without that processor.

kba · 2022-05-04T06:57:14Z

I don't object removing OCR-D processors which nobody uses. Technically there is no need to remove ocrd_pc_segmentation.

@kba, I suggest to merge this PR – either with or without that processor.

On it.

bertsky · 2022-05-04T06:57:20Z

Technically there is no need to remove ocrd_pc_segmentation.

There is!

See above.

stweil · 2022-05-04T08:07:12Z

All Docker builds in CircleCI now passed again. The longest run took 52 minutes, so was well below one hour.

stweil and others added 11 commits April 27, 2022 09:48

Remove unneeded virtual environment headless-torch14

babfc8b

Move ocrd_detectron2 to headless-tf1 to avoid conflicts in the main virtual environment. Signed-off-by: Stefan Weil <[email protected]>

Remove unneeded virtual environment headless-tf2

df2b460

Move ocrd_cis to headless-tf1 to avoid a conflict with ocrd_calamari. Signed-off-by: Stefan Weil <[email protected]>

Update clean target to remove base directory of sub venvs too

b706ad5

Signed-off-by: Stefan Weil <[email protected]>

Update clean target to remove all potential OCR-D semaphore files

2457768

Makefile uses GNU parallel semaphores not only for git but also for pip, but the old rule only cleaned the former ones. Signed-off-by: Stefan Weil <[email protected]>

Update ocrd_cis

d53b5b0

Signed-off-by: Stefan Weil <[email protected]>

Merge remote-tracking branch 'stweil/clean' into update-2022-05-03

18a6f21

📝 changelog

7fbee34

Merge remote-tracking branch 'stweil/headless-torch14' into update-20…

4afd5e8

…22-05-03

keep ocrd_cis in main venv

5c60bf6

update 2022-05-03

d7f9c95

📝 changelog (ocrd_cis)

1dd05ae

stweil approved these changes May 3, 2022

View reviewed changes

move eynollah and sbb_binarization into top venv

520f101

stweil mentioned this pull request May 3, 2022

fix tf2 with tf1.compat session qurator-spk/sbb_binarization#35

Merged

stweil force-pushed the update-2022-05-03 branch from 29bb998 to 520f101 Compare May 3, 2022 20:42

Robert Sachunsky added 2 commits May 3, 2022 23:34

disable building ocrd_pc_segmentation by default

b9ba60c

update submodules (fixing TF2 dependency)

b61f8a1

kba merged commit b61f8a1 into master May 4, 2022

stweil deleted the update-2022-05-03 branch May 4, 2022 07:06

Update 2022 05 03 #317

Update 2022 05 03 #317

Uh oh!

Conversation

kba commented May 3, 2022

Uh oh!

stweil left a comment

Choose a reason for hiding this comment

Uh oh!

stweil commented May 3, 2022

Uh oh!

kba commented May 3, 2022

Uh oh!

stweil commented May 3, 2022

Uh oh!

kba commented May 3, 2022

Uh oh!

stweil commented May 3, 2022

Uh oh!

kba commented May 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kba commented May 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bertsky commented May 3, 2022

Uh oh!

bertsky commented May 3, 2022

Uh oh!

bertsky commented May 3, 2022

Uh oh!

cneud commented May 3, 2022

Uh oh!

bertsky commented May 3, 2022

Uh oh!

bertsky commented May 3, 2022

Uh oh!

bertsky commented May 3, 2022

Uh oh!

stweil commented May 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cneud commented May 3, 2022

Uh oh!

stweil commented May 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stweil commented May 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bertsky commented May 3, 2022

Uh oh!

bertsky commented May 3, 2022

Uh oh!

chreul commented May 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stweil commented May 4, 2022

Uh oh!

kba commented May 4, 2022

Uh oh!

bertsky commented May 4, 2022

Uh oh!

stweil commented May 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kba commented May 3, 2022 •

edited

Loading

kba commented May 3, 2022 •

edited

Loading

stweil commented May 3, 2022 •

edited

Loading

stweil commented May 3, 2022 •

edited

Loading

stweil commented May 3, 2022 •

edited

Loading

chreul commented May 4, 2022 •

edited

Loading