Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🍎 Mac Qt6 .dmg #60039

Open
wants to merge 34 commits into
base: master
Choose a base branch
from
Open

🍎 Mac Qt6 .dmg #60039

wants to merge 34 commits into from

Conversation

m-kuhn
Copy link
Member

@m-kuhn m-kuhn commented Jan 2, 2025

Description

Please download the release from the MacOS link in this comment #60039 (comment) for testing-- there are other links in other comments to older broken builds.

Creates macos installers (dmg), based on Qt6/vcpkg for desktop components

  • It provides QGIS, qgis_process, qgis_bench, ogr2ogr and a few executable binaries
  • The package is created for master, pull requests and releases
  • It creates two builds, one for x64 (intel) and one for arm64 (M) processors it creates a "universal" executable that works natively on both x64 and arm64 (socalled "multiarchitecture" or "fat" binaries)
  • It simplifies a lot of cmake code by not building frameworks. If there is a good reason to reintroduce frameworks, I am happy to discuss how to deal with specific use cases for which they are currently used.
  • It separates the Python_SITEARCH directory with an absolute system path (where site-packages is installed on the system) from a new variable QGIS_PYTHON_INSTALL_DIR with relative variable where our python bindings will be installed to (which can be the same path as Python_SITEARCH if the prefix provided through CMAKE_INSTALL_PREFIX and/or DESTDIR matches)
  • It completely removed CMAKE_INSTALL_DIR from cmake code in install directives, to allow for installation on relative paths
  • It does not include QGIS server, if you want to run QGIS server on macos, there are homebrew, conda and other options
  • This removes the current mac bundles (the dependencies in there are outdated)
  • It separates building the application (for development/debugging) from bundling. The macos bundle is only created at bundling time. For debugging, no libraries are copied into the build path.
  • For creating an app and fixing rpath, macdeployqt is used (only minimal usage within cmake) For fixing rpath, a custom script similar to maceployqt is used. Macdeployqt was too slow and didn't handle some cases (like symlinks)
  • For creating a shiny dmg, create-dmg is used
  • No notarisation or code signing is done (yet)

Fixes #46299

@github-actions github-actions bot added this to the 3.42.0 milestone Jan 2, 2025
Copy link

github-actions bot commented Jan 2, 2025

🧮 Vcpkg update report

Added packages:
🍓 qtcharts: 6.8.1 (Features: )

Updated packages:
🍇 py-poetry-core: 1.9.1 -> 2.0.0
🍇 py-pyqt-builder: 1.17.0 -> 1.17.1
🍇 py-tomli: 2.0.1 -> 2.0.1#1
🍇 py-trove-classifiers: 2024.10.21.16 -> 2025.1.7.14

🪟 Windows builds

Download Windows builds of this PR for testing.
Debug symbols for this build are available here.
(Built from commit a912987)

🪟 Windows Qt6 builds

Download Windows Qt6 builds of this PR for testing.
(Built from commit a912987)

🍎 MacOS Qt6 builds

Download MacOS Qt6 builds of this PR for testing.
This installer is not signed, control+click > open the app to avoid the warning
(Built from commit a912987)

cmake/VcpkgInstallDeps.cmake Outdated Show resolved Hide resolved
@m-kuhn m-kuhn changed the title Mac Qt6 .dmg 🍎 Mac Qt6 .dmg Jan 2, 2025
@m-kuhn
Copy link
Member Author

m-kuhn commented Jan 3, 2025

Creating the dmg on x64 github action runners (macos-13) randomly fails because of "resource busy" in hdiutil. There is not much we can do but retry until we succeed -- which can eventually take a very long time.

@nyalldawson
Copy link
Collaborator

@m-kuhn

Creating the dmg on x64 github action runners (macos-13) randomly fails because of "resource busy" in hdiutil. There is not much we can do but retry until we succeed -- which can eventually take a very long time.

Should we just disable these then? (At least it's only the legacy build)

@m-kuhn
Copy link
Member Author

m-kuhn commented Jan 5, 2025

Possibly, I'm trying to create a zip or a universal app. Last resort would be killing x64

@eflaten
Copy link

eflaten commented Jan 6, 2025

I tried a x64 build during the weekend, and that one worked :) The one that worked was when the downloadable .dmg was separated between M prosessor and Intel. I am on macos 15.2

The one that are for download now first dident launch because of permission errors, but launched after I did:
chmod 755 /Applications/QGIS-PR60039.app/Contents/MacOS/QGIS-PR60039
so it is -rwxr-xr-x@
Then it starts, but crashes:

Termination Reason:    Namespace DYLD, Code 1 Library missing
Library not loaded: @rpath/libQt6OpenGLWidgets.6.8.1.dylib
Referenced from: <CF52E830-6AD3-32CD-9072-DA23F27414B0> /Applications/QGIS-PR60039.app/Contents/MacOS/QGIS-PR60039
Reason: tried: '/Applications/QGIS-PR60039.app/Contents/MacOS/../Frameworks/libQt6OpenGLWidgets.6.8.1.dylib' (no such file), '/Applications/QGIS-PR60039.app/Contents/MacOS/../Frameworks/libQt6OpenGLWidgets.6.8.1.dylib' (no such file)

If there is any particular tests I can do. I am happy to help. If I can. Nice to see progress one the macos work on QGIS, Mathias :)

@m-kuhn
Copy link
Member Author

m-kuhn commented Jan 7, 2025

@eflaten if you happen to have an arm64 (M series) machine, a test of https://github.com/qgis/QGIS/actions/runs/12642053020/artifacts/2393456714 would be appreciated.

@eflaten
Copy link

eflaten commented Jan 7, 2025

@eflaten if you happen to have an arm64 (M series) machine, a test of https://github.com/qgis/QGIS/actions/runs/12642053020/artifacts/2393456714 would be appreciated.

Oh sorry. No arm64 machine here. Just a x64.

@NyakudyaA
Copy link
Contributor

@eflaten if you happen to have an arm64 (M series) machine, a test of https://github.com/qgis/QGIS/actions/runs/12642053020/artifacts/2393456714 would be appreciated.

Oh sorry. No arm64 machine here. Just a x64.

I can test this, but the link keeps timing out from downloading the DMG

@m-kuhn
Copy link
Member Author

m-kuhn commented Jan 7, 2025

@eflaten if you happen to have an arm64 (M series) machine, a test of https://github.com/qgis/QGIS/actions/runs/12642053020/artifacts/2393456714 would be appreciated.

Oh sorry. No arm64 machine here. Just a x64.

I can test this, but the link keeps timing out from downloading the DMG

Strange, link works here... Can you download the arm64 zip from here https://github.com/qgis/QGIS/actions/runs/12642053020?

@m-kuhn
Copy link
Member Author

m-kuhn commented Jan 7, 2025

The latest dmg (dmg https://github.com/qgis/QGIS/actions/runs/12656340998/artifacts/2397658648 from run https://github.com/qgis/QGIS/actions/runs/12656340998) should be universal and works here on my x64 machine ...

@eflaten
Copy link

eflaten commented Jan 7, 2025

qgis macos Qt6 universal: ( https://github.com/qgis/QGIS/actions/runs/12656340998/artifacts/2397658648)
in: (https://github.com/qgis/QGIS/actions/runs/12656340998)
started from terminal.
Hurra!

`

QGIS version 3.41.0-Master
QGIS code revision d465ca2
 
Libraries
Qt version 6.8.1
Python version 3.11.10
GDAL/OGR version 3.9.3
PROJ version 9.5.1
EPSG Registry database version v11.022 (2024-11-05)
GEOS version 3.13.0-CAPI-1.19.0
SQLite version 3.47.2
PDAL version 2.8.1
PostgreSQL client version unknown
SpatiaLite version 5.1.0
QWT version 6.3.0
QScintilla2 version 2.14.1
OS version macOS Sequoia (15.2)
 
Active Python plugins
processing 2.12.99
grassprovider 2.12.99
db_manager 0.1.20
MetaSearch 0.3.6

@NyakudyaA
Copy link
Contributor

Also could start the app using the dmg
Screenshot 2025-01-08 at 09 42 47

However, starting QGIS from the terminal I get a bunch of errors i.e
Screenshot 2025-01-08 at 09 44 25

Screenshot 2025-01-08 at 09 45 02 Screenshot 2025-01-08 at 09 45 48 Screenshot 2025-01-08 at 09 46 21

@m-kuhn
Copy link
Member Author

m-kuhn commented Jan 8, 2025

@NyakudyaA that's great news, thanks !
Is it possible that when starting from the console there are conflicting environment variables around (e.g. for python)?

@m-kuhn m-kuhn marked this pull request as ready for review January 8, 2025 10:35
@m-kuhn
Copy link
Member Author

m-kuhn commented Jan 8, 2025

The problems with x64 are "fixed".
Here's what we do now:

  • we create .app's in the matrix run (one for x64 and one for arm64)
  • upload these as assets
  • merge the two apps and their libraries into a universal FAT .app (which runs natively on both architectures)
  • bundle this into a dmg on an arm64 runner (where hdiutil runs stable)

Best of all worlds 🎉

Gdal algorithms are fixed as well now.

@e-kotov
Copy link

e-kotov commented Jan 8, 2025

Tested arm64 on M1 with latest Sonoma. Started the app, opened a simple project. Seems to be working well.

@noirchen
Copy link

Glad to see that a working M1 build, but I got some issues converting a raster geotif to vector using polygonize. It appears that the python path (/Applications/QGIS-PR60039.app/Contents/Frameworks/lib/python3.11) is not correctly recognized
Screenshot 2025-01-20 at 18 27 17

@m-kuhn
Copy link
Member Author

m-kuhn commented Jan 20, 2025

Glad to see that a working M1 build, but I got some issues converting a raster geotif to vector using polygonize. It appears that the python path (/Applications/QGIS-PR60039.app/Contents/Frameworks/lib/python3.11) is not correctly recognized

Thanks, the file seems to be present in python3.11/site-packages/osgeo_utils/gdal_polygonize.py, it needs to be checked if the file is at the wrong place or if the processing algorithm looks for it at the wrong place

@M-Rick
Copy link

M-Rick commented Jan 20, 2025

I got troubles to get Python 3.11 to be regonized, pip3 as well, the Terminal said it didn’t exist…
It’s because the bundle isn’t notarized, so I did ‘sudo xattr -cr /Applications/QGIS-3.40.app’ and got it working,

@oskarlin
Copy link

Another rookie question, where do I find latest builds to download when this branch has merged with the main? It would be nice to be able to continue using the latest versions using QT6.

@m-kuhn
Copy link
Member Author

m-kuhn commented Jan 20, 2025

Another rookie question, where do I find latest builds to download when this branch has merged with the main? It would be nice to be able to continue using the latest versions using QT6.

Once merged this will create a new dmg from the master branch continuously (will keep it on github for 30 days IIRC). This will be downloadable for people logged in to github.
Either by manually navigating to the last github actions run.
Or we can add a https://nightly.link/ shortcut to download the latest build

@justinbb
Copy link
Contributor

Glad to see that a working M1 build, but I got some issues converting a raster geotif to vector using polygonize. It appears that the python path (/Applications/QGIS-PR60039.app/Contents/Frameworks/lib/python3.11) is not correctly recognized

Thanks, the file seems to be present in python3.11/site-packages/osgeo_utils/gdal_polygonize.py, it needs to be checked if the file is at the wrong place or if the processing algorithm looks for it at the wrong place

Two problems: (1) the .py files in osgeo_utils need to have execute permission turned on (2) that directory is not in the PATH. As a test, I symlinked gdal_polygonize.py into the bundle's main executable directory /Applications/QGIS-PR60039.app/Contents/MacOS:/Applications/QGIS-PR60039.app/Contents/MacOS, where the gdal executables are located. This allowed gdal_polygonize to run, though it complained it couldn't find module osgeo.

@M-Rick
Copy link

M-Rick commented Jan 20, 2025

Another rookie question, where do I find latest builds to download when this branch has merged with the main? It would be nice to be able to continue using the latest versions using QT6.

I have created a GDAL Framework if you want. It is independent from QGIS builds. It is GDAL 3.9 and doesn't support ECW currently. I built it using MacPorts and it is based on Python 12, Proj 9, Postgresql 16, ODBC, kea, OPENJPEG, OpenEXR and Xerces.
https://s10.swdrive.fr/s/3P58X9HkbBDHq9Q

Place it in /Library/Frameworks and then sudo xattr -cr /Library/Frameworks/GDAL.framework.

@perrygeo
Copy link

Another +1 for this build. I'm using Sequoia 15.1.1 on an M3 arm64 and this works beautifully so far.

@songololo
Copy link

Trying this PR release to see if it handles geoparquet on Macs.

Encountering this form of error for reading files: .parquet is not a valid or recognized data source and don't see an option to export files as parquet.

Is there a way to enable this functionality?

Thanks.

Screenshot 2025-01-23 at 15 13 53

@m-kuhn
Copy link
Member Author

m-kuhn commented Jan 23, 2025

Trying this PR release to see if it handles geoparquet on Macs.

Encountering this form of error for reading files: .parquet is not a valid or recognized data source and don't see an option to export files as parquet.

Is there a way to enable this functionality?

Thanks.

Screenshot 2025-01-23 at 15 13 53

On the todo list for spring

@m-kuhn
Copy link
Member Author

m-kuhn commented Jan 26, 2025

Glad to see that a working M1 build, but I got some issues converting a raster geotif to vector using polygonize. It appears that the python path (/Applications/QGIS-PR60039.app/Contents/Frameworks/lib/python3.11) is not correctly recognized Screenshot 2025-01-20 at 18 27 17

The latest build (becoming available in the next hours) should support these gdal algorithms

@noirchen
Copy link

Glad to see that a working M1 build, but I got some issues converting a raster geotif to vector using polygonize. It appears that the python path (/Applications/QGIS-PR60039.app/Contents/Frameworks/lib/python3.11) is not correctly recognized Screenshot 2025-01-20 at 18 27 17

The latest build (becoming available in the next hours) should support these gdal algorithms

Tested it with the same operation but there is an unsupported x64 python3.11

Screenshot 2025-01-27 at 12 09 53

@m-kuhn
Copy link
Member Author

m-kuhn commented Jan 27, 2025

Tested it with the same operation but there is an unsupported x64 python3.11

Seems there was still an absolute path to python from the build system inside, update should fix that 🤞

@noirchen
Copy link

Tested it with the same operation but there is an unsupported x64 python3.11

Seems there was still an absolute path to python from the build system inside, update should fix that 🤞

This seems to be a permission issue
Screenshot 2025-01-27 at 20 30 07

@m-kuhn
Copy link
Member Author

m-kuhn commented Jan 27, 2025

This seems to be a permission issue

What makes you think so?

@rouault do you have an educated guess what could be the problem here?

@rouault
Copy link
Contributor

rouault commented Jan 27, 2025

do you have an educated guess what could be the problem here?

no, except the obvious question: does QGIS have write permissions to /private ?

@m-kuhn
Copy link
Member Author

m-kuhn commented Jan 27, 2025

@noirchen I guess we could validate if this the problem by specifying the output path instead of using a temp layer

@noirchen
Copy link

@noirchen I guess we could validate if this the problem by specifying the output path instead of using a temp layer

Well, not a permission issue
Screenshot 2025-01-28 at 18 02 57

@m-kuhn
Copy link
Member Author

m-kuhn commented Jan 28, 2025

@noirchen could you try to run that same command from a terminal to see if there is more information?

@noirchen
Copy link

@noirchen could you try to run that same command from a terminal to see if there is more information?

I ran the same command in a terminal and I got ModuleNotFoundError: No module named 'osgeo'. I am not sure if this is the cause.

@ping13
Copy link

ping13 commented Jan 31, 2025

Trying this PR release to see if it handles geoparquet on Macs.

Encountering this form of error for reading files: .parquet is not a valid or recognized data source and don't see an option to export files as parquet.

Is there a way to enable this functionality?

@songololo This might be a workaround for loading parquet files: the plugin QDuckDB works now with the Qt6 version. You can load your parquet file via a DuckDB connection - I just tested it and it works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Native support for Apple Silicon (ARM64 architecture)