Skip to content

Feature/project search #304

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
grafikrobot merged 3 commits into from
Jun 10, 2023
Merged

Feature/project search #304

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3f6dd2d
Add automatic project searching.
grafikrobot May 3, 2023
87c35a5
Fix incorrect project search env var name.
grafikrobot May 4, 2023
4a7c415
Fix infinite recursion when project path has "..".
grafikrobot May 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 96 additions & 1 deletion doc/src/tasks.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -670,7 +670,7 @@ for details.
[[bbv2.tasks.packagemanagers]]
== Package Managers

B2 support automatic, or manual, loading of generated build files
B2 supports automatic, or manual, loading of generated build files
from package managers. For example using the Conan package manager which
generates `conanbuildinfo.jam` files B2 will load that files automatically
when it loads the project at the same location. The included file can
Expand Down Expand Up @@ -726,3 +726,98 @@ managers. Currently the supported ones are:

* Conan (`conan`): currently supports the
link:https://docs.conan.io/en/latest/reference/generators/b2.html[`b2 generator`].

[[bbv2.tasks.projectsearch]]
== Searching For Projects

B2 supports automatic searching for referenced global projects. For example,
if you have references to `/boost/predef` with some minimal configuration B2
can find the B2 project for it and automatically resolve the reference. The
searching supports two modes: finding regular B2 project directories, and
package/config style loading of single jam files.

[[bbv2.tasks.projectsearch.path]]
=== Search Path

To control which and where projects are found one can use different methods:

* `B2_PROJECT_PATH` environment variable.
* `--project-search` command line argument.
* `rule project-search` project rule.

The search path in both `B2_PROJECT_PATH` and `--project-search` specifies a
key-value list of _project-id_ and _path_. The parts of that key-value list, as
Copy link
Contributor

@Kojoley Kojoley May 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

key-value

Do you really think this complication worth it? I would guess that in the most user cases all the projects under specified paths will be loaded anyway, and for a package manager like vcpkg it is not a big deal to customize/optimize search scope via project-search rule.

Copy link
Member Author

@grafikrobot grafikrobot Jun 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

key-value

Do you really think this complication worth it? I would guess that in the most user cases all the projects under specified paths will be loaded anyway, and

Not sure I follow that. The use case I'm working on ATM is the case of a modular Boost. Where definitely not all projects would be loaded even though they could be under the same specified path. But maybe I'm not understanding what you are thinking of.

for a package manager like vcpkg it is not a big deal to customize/optimize search scope via project-search rule.

Recently I've come to learn that everything is a big deal in vcpkg because it actually does very little for you :-)

Copy link
Contributor

@Kojoley Kojoley Jun 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The project (Boost) that has Jamroot will eventually include everything in its subdirectories

he use case I'm working on ATM is the case of a modular Boost. Where definitely not all projects would be loaded even though they could be under the same specified path.

Can't Boost superproject (Jamroot) have that loader which will load a needed subproject when you request a library? Or your vision is that every Boost library will get its own Jamroot and libraries will no longer will be rooted to the superproject? But then I don't understand who and how will set and update these mappings for every library.

Copy link
Member Author

@grafikrobot grafikrobot Jun 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The project (Boost) that has Jamroot will eventually include everything in its subdirectories

But it doesn't have to.

he use case I'm working on ATM is the case of a modular Boost. Where definitely not all projects would be loaded even though they could be under the same specified path.

Can't Boost superproject (Jamroot) have that loader which will load a needed subproject when you request a library? Or your vision is that every Boost library will get its own Jamroot and libraries will no longer will be rooted to the superproject? But then I don't understand who and how will set and update these mappings for every library.

A project that has just a top-level Jamfile automatically also works as if that Jamfile was the Jamroot. That's been around for some time. The goal is two fold.. 1. support the use case, for Boost, where there is a super-project Jamroot and 2. support the use case, for Boost, where the super-project doesn't exist (or there is no Jamroot/Jamfile at that root.

But perhaps a concrete example is better. Here's the modular Boost I've been working on https://github.com/grafikrobot/boost-b2-modular. As it currently stands there the following are possible:

  1. It works as is where the Jamroot loads the libs projects.
  2. You can replace everything after the project boost .. ; declaration with project-search /boost : libs ; and everything will still work.
  3. You can delete the Jamroot entirely and do cd /*/boosroot/libs/serialization/build && b2 --project-search=/boost:/*/boostroot/libs ... and it will build serialization without problems. (* = absolute or relative path).

A package manager, or equivalent, would be doing (3) if they want modular Boost packages. Although they would probably do a bunch of individual key:value mappings for each. And they could also do (1), or (2), for monolithic builds (probably .. as I'm still working out all the details on that).

Does that help answer the questions? Or is it creating more questions? ;-)

Copy link
Contributor

@Kojoley Kojoley Jun 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. You can delete the Jamroot entirely and do cd /*/boosroot/libs/serialization/build && b2 --project-search=/boost:/*/boostroot/libs ... and it will build serialization without problems. (* = absolute or relative path).

From my understanding of the written here is that this will only work because boost superproject loads serialization and all its dependencies, am I wrong?

A package manager, or equivalent, would be doing (3) if they want modular Boost packages. Although they would probably do a bunch of individual key:value mappings for each.

On the one hand it is expected from package managers to know all the dependencies of the requested library, on the other hand the current solution does not provide a way to discover dependency tree to construct such b2 invocation. vcpkg struggled with that quite a bunch and currently solves the issue with their own header scanning script + boostdep mappings.

Copy link
Member Author

@grafikrobot grafikrobot Jun 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. You can delete the Jamroot entirely and do cd /*/boosroot/libs/serialization/build && b2 --project-search=/boost:/*/boostroot/libs ... and it will build serialization without problems. (* = absolute or relative path).

From my understanding of the written here is that this will only work because boost superproject loads serialization and all its dependencies, am I wrong?

You are wrong :-) The way it will works in modular Boost is that individual libraries will need to specify their dependencies in Jamfiles. The method is the usual one of just having a target reference. For example serialization/build.jam (aka jamfile) would look like this https://github.com/grafikrobot/boost-b2-modular/blob/b2-modular/patch/libs/serialization/build.jam#L8. And so one for dependencies recursively. With this PR B2 when it tries to resolve those targets would split them into the /project//target parts and then use the user specified mappings to find the, for example, array/build.jam and do a use-porject /boost/array : .../array ; And then look up the boost_array target. Hence no need or involvement from a super-project.

A package manager, or equivalent, would be doing (3) if they want modular Boost packages. Although they would probably do a bunch of individual key:value mappings for each.

On the one hand it is expected from package managers to know all the dependencies of the requested library, on the other hand the current solution does not provide a way to discover dependency tree to construct such b2 invocation. vcpkg struggled with that quite a bunch and currently solves the issue with their own header scanning script + boostdep mappings.

But it could provide such information since it's going to be present in the build of the library itself. I.e. we could add an introspection feature to B2 that, given a target, spits out the list of external project dependencies (which would be the other Boost libs in this case).

Copy link
Contributor

@Kojoley Kojoley Jun 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see what I have missed: libs part in --project-search=/boost:/*/boostroot/libs, which looks so strange and alien while still not working for sublibs and stuff in tools folder.

grafikrobot/boost-b2-modular@b2-modular/patch/libs/serialization/build.jam#L8.

I'm sorry to say that but I have to -- it looks ugly, writing the same stuff twice... Is it impossible to make <library>/boost/array work? Even better if it could be just simple /boost/array. It probably could be achieved by considering project-id reference as implying adding every project top-level target as <source>.

Copy link
Member Author

@grafikrobot grafikrobot Jun 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see what I have missed: libs part in --project-search=/boost:/*/boostroot/libs, which looks so strange and alien while still not working for sublibs and stuff in tools folder.

Well.. it's something entirely new. So some strangeness is expected. I thought about a couple different syntax options on the CLI and ENV. But this was the least problematic given all the constraints of the multiple platforms.

grafikrobot/boost-b2-modular@b2-modular/patch/libs/serialization/build.jam#L8.

I'm sorry to say that but I have to -- it looks ugly, writing the same stuff twice... Is it impossible to make <library>/boost/array work?

Maybe. It would mean finding a way for a project declaring some target as the one to add to the requirements and usage-requirements. Which would be used if a bare project reference is asked for.

Even better if it could be just simple /boost/array. It probably could be achieved by considering project-id reference as implying adding every project top-level target as <source>.

Maybe. The easier way would be to add some other init argument to define those project references.

So certainly some things to think about. But those are orthogonal to this PR. As they would be additions to synthesize the references. I.e. things for some future PRs.

the name indicates, is a path delimiter separated list. For example if we had
these two projects we wanted to find: `/zlib` and `/boost/asio` the search paths
could look like:

[horizontal]
Linux::
`/zlib:/usr/local/share/zlib:/boost/asio:/home/user/external/boost-1.81/libs/asio`
Windows, VxWorks::
`/zlib;C:/Dev/zlib;/boost/asio;C:Dev/boost-1.81/libs/asio`
VMS::
`/zlib,X:external.zlib,/boost/asio,X:external.boost181.libs.asio`

The _project-id_ in the search path specification maps that project root to the
indicated _path_. Which B2 will use to search for any projects and sub-projects
with that _project-id_ root.

[[bbv2.tasks.projectsearch.process]]
=== Search Process

Regardless of how the search path is specified, how the search happens is the
same. Searching involves either searching for a B2 project directory, i.e.
a directory containing a jamfile, or searching for a specially named `*.jam`
file to include (similar to how the <<Package Managers>> support includes
jam files).

For a given _project-id_ of the form `/d1/d2/../dn` we search for the following,
in this order:

. The project at `d1/d2/../dn` in any path registered for the `/` root.
. The project at `dn` in any path registered for the `/d1/d2/../dn-1` root.
. The jamfile `dn.jam` in any path registered for the `/d1/d2/../dn-1` root.
. The project at `dn-1_dn` in any path registered for the `/d1/d2/../dn-2` root.
. The jamfile `dn-1_dn.jam` in any path registered for the `/d1/d2/../dn-2`
root.
. And so on until it searches for the project `d1_d2_.._dn` in any path
registered for the `/` root.
. And for the jamfile `d1_d2_.._dn.jam` in any path registered for the `/` root.

For example, with this search paths:

* `/boost`: `/usr/share/boost-1.81.0`, `/home/user/boost-dev/libs`
* `/`: `/usr/share/b2/external`

And given the `/boost/core` _project-id_ to resolve, we search for:

. `/usr/share/b2/external/boost/core/<jamfile>`
. `/usr/share/boost-1.81.0/core/<jamfile>`
. `/home/user/boost-dev/libs/core/<jamfile>`
. `/usr/share/boost-1.81.0/core.jam`
. `/home/user/boost-dev/libs/core.jam`
. `/usr/share/boost-1.81.0/boost_core/<jamfile>`
. `/home/user/boost-dev/libs/boost_core/<jamfile>`
. `/usr/share/boost-1.81.0/boost_core.jam`
. `/home/user/boost-dev/libs/boost_core.jam`
. `/usr/share/b2/external/boost_core.jam`

The first project jamfile will be assigned to the _project-id_. Or the first
`*.jam` file found will be loaded.

[[bbv2.tasks.projectsearch.loading]]
=== Loading Process

Depending on whether a project jamfile or `*.jam` file determines how the
project is loaded.

When loading a project jamfile with a _project-id_ and _path_ it is equivalent
to calling `use-project _project-id_ : _path_ ;` from the context of the project
that has the reference.

When loading a `*.jam` file as the _path_ it is equivalent to calling:
`use-packages _path_ ;` from the context of the project that has the reference.
In this case it means that the file will be loaded as part of the referenced
project and hence any bare targets or information it declares will be part of
the project.
136 changes: 135 additions & 1 deletion src/build/project.jam
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,16 @@

import "class" : new ;
import modules ;
import os ;
import path ;
import print ;
import property-set ;
import regex ;
import sequence ;


.debug-loading = [ MATCH ^(--debug-loading)$ : [ modules.peek : ARGV ] ] ;
.debug-project-search = [ MATCH ^(--debug-project-search)$ : [ modules.peek : ARGV ] ] ;


# Loads the Jamfile at the given location. After loading, project global file
Expand Down Expand Up @@ -141,12 +144,42 @@ rule load-parent ( location )
#
rule find ( name : current-location )
{
name = [ path.normalize-all $(name) ] ;
local project-module ;

# Try interpreting name as project id.
if [ path.is-rooted $(name) ]
{
project-module = $($(name).jamfile-module) ;
if ! $(project-module)
{
# Not immediately found as a project. Try and search for a project
# we can import.
local root-and-jamfile = [ search $(name) ] ;
local root = $(root-and-jamfile[1]) ;
local jamfile = $(root-and-jamfile[2]) ;
if $(root)
{
local caller-project = [ CALLER_MODULE ] ;
local caller-module = [ $(caller-project).project-module ] ;
if $(jamfile)
{
modules.call-in $(caller-module)
: use-packages $(jamfile) ;
}
else
{
if $(.debug-project-search)
{
ECHO Using project '$(name)' from '$(root)' ;
}
modules.call-in $(caller-module)
: use-project $(name) : $(root) ;
load-used-projects $(caller-module) ;
}
}
project-module = $($(name).jamfile-module) ;
}
}

if ! $(project-module)
Expand Down Expand Up @@ -177,6 +210,96 @@ rule find ( name : current-location )
}


B2_PROJECT_PATH = [ modules.peek : B2_PROJECT_PATH ]
[ regex.split-list
[ MATCH ^--project-search=(.*)$ : [ modules.peek : ARGV ] ]
: [ os.path-separator ] ] ;
{
while $(B2_PROJECT_PATH)
{
local root = $(B2_PROJECT_PATH[1]) ;
local path = [ path.make $(B2_PROJECT_PATH[2]) ] ;
B2_PROJECT_PATH = $(B2_PROJECT_PATH[3-]) ;
if $(root) && $(path)
{
path = [ path.root $(path) [ path.pwd ] ] ;
.search-path.$(root) += $(path) ;
}
}
}


rule add-project-search ( root : search-paths + )
{
for local search-path in $(search-paths)
{
if ! [ path.is-rooted $(search-path) ]
{
local prj = [ current ] ;
search-path = [ path.join [ $(prj).location ] $(search-path) ] ;
search-path = [ path.root $(search-path) [ path.pwd ] ] ;
}
.search-path.$(root) += $(search-path) ;
}
}


local rule search ( name )
{
if [ path.is-rooted $(name) ]
{
# Check for a regular B2 project relative to the search path and
# project subdir.
for local dir in "$(.search-path./)"
{
dir = $(dir)$(name) ;
local jamfile = [ path.glob $(dir) : $(JAMROOT) $(JAMFILE) ] ;
if $(jamfile)
{
return $(dir) ;
}
}
# Do searching for various dir names and jamfiles based on the built
# basename <a>_<b>_<c>[.jam].
local base = [ path.basename $(name) ] ;
local root = [ path.parent $(name) ] ;
while $(base)
{
# Check for a regular B2 project in the search path and subdir base.
for local dir in $(.search-path.$(root))
{
dir = [ path.join $(dir) $(base) ] ;
local jamfile = [ path.glob $(dir) : $(JAMROOT) $(JAMFILE) ] ;
if $(jamfile)
{
return $(dir) ;
}
}
# Check for a <base>.jam to include.
for local dir in $(.search-path.$(root))
{
local jamfile = [ path.glob $(dir) : $(base:L).jam ] ;
if $(jamfile)
{
return $(dir) $(jamfile[1]) ;
}
}
if [ path.has-parent $(root) ]
{
base = [ path.basename $(root) ] $(base) ;
base = $(base:L:J=_) ;
root = [ path.parent $(root) ] ;
}
else
{
base = ;
root = ;
}
}
}
}


# Returns the name of the module corresponding to 'jamfile-location'. If no
# module corresponds to that location yet, associates the default module name
# with that location.
Expand Down Expand Up @@ -322,7 +445,7 @@ rule load-package-manager-build-info ( )
if $(pm) && ! ( $(pm-tag) in $(.package-manager-build-info) )
{
.package-manager-build-info += $(pm-tag) ;
# We found a matching builf info to load, but we have to be careful
# We found a matching build info to load, but we have to be careful
# as the loading can affect the current project since it can define
# sub-projects. Hence we save and restore the current project.
local saved-project = $(.current-project) ;
Expand Down Expand Up @@ -619,6 +742,9 @@ rule initialize (
{
.first-project-root = $(module-name) ;
}
# Set the default build-dir so that we don't get build results
# spread out everywhere.
# $(attributes).set build-dir : ".b2/bin" ;
}

local parent ;
Expand Down Expand Up @@ -1418,4 +1544,12 @@ module project-rules
] ;
}
}

# Defines search paths for resolving global, i.e. rooted, project references.
rule project-search ( root : search-paths + )
{
import path ;
import project ;
project.add-project-search $(root) : $(search-paths) ;
}
}