Port to dune

Motivation
==========

With a port to dune, the ctypes library can be embedded in larger dune
projects simply by including it in the directory tree of the bigger
project. This in turn will allow MirageOS unikernels to use Ctypes
seamlessly as part of embedded compilation, using the standard
cross-compilation and library variants support built into Dune. The goal
is to make Ctypes the default FFI interface to MirageOS, and have it
work out of the box on all of the backends. All existing OS platforms
should be supported as well before merge, including Windows.

Porting Approach
================

This PR still installs ocamlfind libraries in the same way as the
previous Makefile infrastructure, so backwards compat is preserved.
However, the findlib schema has been slightly modified to separate out
the foreign library from the core ctypes library. We now install:

- ctypes that contains ctypes.top and ctypes.stubs as before. There are
  aliases to ctypes.foreign that redirect existing uses of those.
- ctypes-foreign contains the foreign library.
  All of the configuration logic for libffi is now in
  src/ctypes-foreign-base, so deleting these directories will remove the
  ffi build logic without touching the core library.

Since dune by default has stricter warnings enabled (the default
--profile=dev mode), the PR currently sprinkles files with [@@@warning
tags. The warnings can be fixed as well if desired, but that would muddy
this PR so not done yet.

Layout
======

The previous packaging would install ctypes and ctypes.stubs in the same
library. Depending on the build system, it means that it's possible to
specify a dependency on ctypes and use ctypes.stubs succesfully. This is
not the case with Dune since it installs these in different directories.
This means that it is necessary to patch these reverse dependencies,
though it can be argued that they were relying on a bug.

Followup work
=============

A test uses OCaml syntax to skip test on windows. Once it's acceptable
to use Dune 3.9, it should be updated to use build_if.

Test and dynamic libraries
==========================

Due to a different linking model, there are some changes in the test
suite regarding dynamic library loading.

Dynamic symbols in ocaml <4.06
------------------------------

The setup for tests is that each test executable tests some properties
using both "stubs" and "foreign" strategies to refer to some symbols
define in `tests/clib/` (the `test_functions` library).

The symbols are accessed through both strategies:
- as a linked symbol (through an `external` via stubs)
- through the dynamic loader (through `dlopen` via foreign)

Dune links the stubs statically, so by default the symbols would not be
visible to the dynamic loader. OCaml sets `-Wl,-E` in `LDFLAGS` to make
these symbols visible at runtime, but until
ocaml/ocaml@edbba02
(between 4.05 and 4.06), this was ignored when linking executables.

So this commit adds these flags when building tests for older versions.
An alternative would be to use `(link_flag)` in an `(env)` stanza but
this requires dune 3.

This issue does not affect the original make-based build because it
assembles the test executables a bit differently: the `test_functions`
library is linked dynamically to the test executable. So the symbols are
already visible to the dynamic loader, even when `-Wl,-E` is not set.

clib loading
------------

This change is specific to windows (but does not change the behavior on
Linux)

It modifies the test setup a bit so that the contents of `clib` are
dynamically loaded at the beginning of each test. This ensures that
later foreign calls (that use the default handle) will succeed.

It was not necessary before because `clib` was linked dynamically. Now
that it is linked statically, dynamic symbols do not have access to the
symbols in the main executable. This is a problem on windows which does
not have the concept of `-Wl,--export-dynamic`. Also, on windows the
stubs DLL can not be loaded directly so we recompile the library using
plain `%{cc}` instead of going through ocamlopt and flexlink.

Instrumentation
===============

This uses dune instrumentation for coverage The instructions are now:

    opam install bisect_ppx
    dune runtest --instrument-with bisect_ppx --force
    bisect-ppx-report html

See <https://github.com/aantron/bisect_ppx#Dune>

Depext handling
===============

This port uses conf-libffi instead of hardcoding depexts in
ctypes-foreign.

Co-authored-by: Anil Madhavapeddy <[email protected]>

Author: 2 people

.depend

@@ -63,10 +63,7 @@ jobs:
           opam install -t --deps-only .
 
       - name: Build
-        run: opam exec -- make
+        run: opam exec -- dune build
 
       - name: Test
-        run: opam exec -- make test
-
-      - name: Test inverted stubs
-        run: opam pin add ctypes-inverted-stubs-example https://github.com/yallop/ocaml-ctypes-inverted-stubs-example.git 
+        run: opam exec -- dune runtest

.github/workflows/test.yml

@@ -1,30 +1,5 @@
+.*.swp
 _build
 _opam
-libffi.config
-asneeded.config
-discover
-gen_c_primitives
-gen_c_primitives.log
-gen_libffi_abi
-gen_libffi_abi.log
-src/ctypes/ctypes_primitives.ml
-src/ctypes_config.h
-src/ctypes_config.ml
-src/ctypes-foreign/dl_stubs.c
-src/ctypes-foreign/dl.ml
-src/discover/commands.cm*
-src/discover/discover.cm*
-src/configure/extract_from_c.cm*
-src/configure/gen_c_primitives.cm*
-src/configure/gen_libffi_abi.cm*
-*~
-generated_stubs.c
-generated_bindings.ml
-generated_struct_bindings.ml
-ncurses_generated.ml
-ncurses_stubs.c
-date_generated.ml
-date_stubs.c
-fts_generated.ml
-fts_stubs.c
-libffi_abi.ml
+*.install
+.merlin

.gitignore

@@ -1,200 +1,10 @@
-.SECONDEXPANSION:
+.PHONY: build clean test
 
-BEST:=$(shell if ocamlopt > /dev/null 2>&1; then echo native; else echo byte; fi)
-OPAQUE:=$(shell if ocamlopt -opaque 2>/dev/null; then echo -opaque; fi) 
-NO_KEEP_LOCS:=$(shell if ocamlopt -no-keep-locs 2>/dev/null; then echo -no-keep-locs; fi) 
-DEBUG=true
-COVERAGE=false
-OCAML=ocaml
-OCAMLFIND=ocamlfind $(OCAMLFINDFLAGS)
-HOSTOCAMLFIND=$(OCAMLFIND)
-OCAMLDEP=$(OCAMLFIND) ocamldep
-OCAMLMKLIB=$(OCAMLFIND) ocamlmklib
-VPATH=src examples
-BUILDDIR=_build
-BASE_PROJECTS=configure libffi-abigen configured ctypes ctypes-top
-FOREIGN_PROJECTS=test-libffi ctypes-foreign
-STUB_PROJECTS=cstubs
-PROJECTS=$(BASE_PROJECTS) $(FOREIGN_PROJECTS) $(STUB_PROJECTS)
-DEP_DIRS=$(foreach project,$(PROJECTS),$($(project).dir))
-GENERATED=src/ctypes/ctypes_primitives.ml	\
-          src/ctypes-foreign/libffi_abi.ml \
-          src/ctypes-foreign/dl.ml		\
-          src/ctypes-foreign/dl_stubs.c	\
-          libffi.config				\
-          asneeded.config                       \
-          discover				\
-          gen_c_primitives			\
-          gen_c_primitives.log			\
-          gen_libffi_abi			\
-	  src/configure/extract_from_c.cmi	\
-	  src/configure/extract_from_c.cmo	\
-	  src/configure/gen_c_primitives.cmi	\
-	  src/configure/gen_c_primitives.cmo	\
-	  src/configure/gen_libffi_abi.cmi	\
-	  src/configure/gen_libffi_abi.cmo	\
-	  src/discover/commands.cmi		\
-	  src/discover/commands.cmo		\
-	  src/discover/discover.cmi		\
-	  src/discover/discover.cmo
+build:
+	dune build
 
-OCAML_FFI_INCOPTS=$(libffi_opt)
-export CFLAGS DEBUG
+test:
+	dune runtest
 
-EXTDLL:=$(shell $(OCAMLFIND) ocamlc -config | awk '/^ext_dll:/{print $$2}')
-OSYSTEM:=$(shell $(OCAMLFIND) ocamlc -config | awk '/^system:/{print $$2}')
-
-ifneq (,$(filter mingw%,$(OSYSTEM)))
-OS_ALT_SUFFIX=.win
-else
-OS_ALT_SUFFIX=.unix
-endif
-
-# public targets
-all: libffi.config $(PROJECTS)
-
-ctypes-base: $(BASE_PROJECTS)
-ctypes-foreign: ctypes-base test-libffi
-ctypes-stubs: ctypes-base $(STUB_PROJECTS)
-
-clean: clean-examples clean-tests
-	rm -fr _build
-	rm -f $(GENERATED)
-
-# ctypes subproject
-ctypes.cmi_only = ctypes_static ctypes_primitive_types ctypes_structs cstubs_internals
-ctypes.public = lDouble complexL ctypes posixTypes ctypes_types
-ctypes.dir = src/ctypes
-ctypes.extra_mls = ctypes_primitives.ml
-ctypes.deps = bigarray-compat integers
-ctypes.linkdeps = integers_stubs
-ctypes.install = yes
-ctypes.install_native_objects = yes
-ifeq ($(XEN),enable)
-ctypes.xen = yes
-endif
-
-ctypes: PROJECT=ctypes
-ctypes: $(ctypes.dir)/$(ctypes.extra_mls) $$(LIB_TARGETS)
-
-# cstubs subproject
-cstubs.public = cstubs_structs cstubs cstubs_inverted
-cstubs.dir = src/cstubs
-cstubs.subproject_deps = ctypes
-cstubs.deps = str integers
-cstubs.install = yes
-cstubs.install_native_objects = yes
-cstubs.extra_hs = $(package_integers_path)/ocaml_integers.h
-
-cstubs: PROJECT=cstubs
-cstubs: $(cstubs.dir)/$(cstubs.extra_mls) $$(LIB_TARGETS)
-
-# ctypes-foreign subproject
-ctypes-foreign.public = dl libffi_abi foreign
-ctypes-foreign.dir = src/ctypes-foreign
-ctypes-foreign.subproject_deps = ctypes
-ctypes-foreign.deps = integers
-ctypes-foreign.install = yes
-ctypes-foreign.install_native_objects = yes
-ctypes-foreign.extra_cs = dl_stubs.c
-ctypes-foreign.extra_mls = libffi_abi.ml dl.ml
-ctypes-foreign.cmi_opts = $(OPAQUE) $(NO_KEEP_LOCS)
-ctypes-foreign.cmo_opts = $(OCAML_FFI_INCOPTS:%=-ccopt %)
-ctypes-foreign.cmx_opts = $(OCAML_FFI_INCOPTS:%=-ccopt %)
-ctypes-foreign.link_flags = $(libffi_lib) $(lib_process)
-ctypes-foreign.threads = yes
-
-ctypes-foreign: PROJECT=ctypes-foreign
-ctypes-foreign: $$(LIB_TARGETS)
-
-# ctypes-top subproject
-ctypes-top.public = ctypes_printers
-ctypes-top.dir = src/ctypes-top
-ctypes-top.install = yes
-ctypes-top.deps = compiler-libs integers
-ctypes-top.subproject_deps = ctypes
-ctypes-top.install_native_objects = yes
-
-ctypes-top: PROJECT=ctypes-top
-ctypes-top: $$(LIB_TARGETS)
-
-# configuration
-configured: src/ctypes/ctypes_primitives.ml src/ctypes-foreign/libffi_abi.ml src/ctypes-foreign/dl.ml src/ctypes-foreign/dl_stubs.c
-
-src/ctypes-foreign/dl.ml: src/ctypes-foreign/dl.ml$(OS_ALT_SUFFIX)
-	cp $< $@
-src/ctypes-foreign/dl_stubs.c: src/ctypes-foreign/dl_stubs.c$(OS_ALT_SUFFIX)
-	cp $< $@
-
-src/ctypes/ctypes_primitives.ml: src/configure/extract_from_c.ml src/configure/gen_c_primitives.ml
-	$(HOSTOCAMLFIND) ocamlc -o gen_c_primitives -package str -strict-sequence -linkpkg $^ -I src/configure
-	./gen_c_primitives > $@ 2> gen_c_primitives.log || (rm $@ && cat gen_c_primitives.log && false)
-
-src/ctypes-foreign/libffi_abi.ml: src/configure/extract_from_c.ml src/configure/gen_libffi_abi.ml
-	$(HOSTOCAMLFIND) ocamlc -o gen_libffi_abi -package str -strict-sequence -linkpkg $^ -I src/configure
-	./gen_libffi_abi > $@ 2> gen_c_primitives.log || (rm $@ && cat gen_c_primitives.log && false)
-
-libffi.config: src/discover/commands.mli src/discover/commands.ml src/discover/discover.ml
-	$(HOSTOCAMLFIND) ocamlc -o discover -package str -strict-sequence -linkpkg $^ -I src/discover
-	./discover -ocamlc "$(OCAMLFIND) ocamlc" > $@ || (rm $@ && false)
-
-asneeded.config:
-	./src/discover/determine_as_needed_flags.sh >> $@
-
-# dependencies
-depend: configure
-	$(OCAMLDEP) -one-line $(foreach dir,$(DEP_DIRS),-I $(dir)) \
-            $(shell find src examples -name '*.mli' -o -name '*.ml') \
-           | sed "s!src/!_build/src/!g; s!examples/!_build/examples/!g" | sort > .depend
-
-#installation
-META-install:
-	$(OCAMLFIND) install ctypes META CHANGES.md
-
-install-%: PROJECT=$*
-install-%:
-	$(if $(filter yes,$($(PROJECT).install)),\
-		$(OCAMLFIND) install -add ctypes -optional $^ \
-                   $(LIB_TARGETS) $(LIB_TARGET_EXTRAS) \
-                   $(INSTALL_MLIS) $(INSTALL_CMIS) \
-                   $(INSTALL_CMTS) $(INSTALL_CMTIS) \
-                   $(INSTALL_HEADERS) \
-                   $(if $(filter yes,$($(PROJECT).install_native_objects)),$(NATIVE_OBJECTS)))
-
-$(PROJECTS:%=install-%): META-install
-
-install: META-install $(PROJECTS:%=install-%)
-
-uninstall:
-	$(OCAMLFIND) remove ctypes
-
-DOCFILES=$(foreach project,$(PROJECTS),\
-           $(foreach mli,$($(project).public),\
-            $($(project).dir)/$(mli).mli))
-DOCFLAGS=-I $(shell ocamlfind query integers) $(foreach project,$(PROJECTS),-I $(BUILDDIR)/$($(project).dir))
-
-doc:
-	ocamldoc -html $(DOCFLAGS) $(DOCFILES)
-
-
-.PHONY: depend clean configure all install doc $(PROJECTS)
-
-include .depend Makefile.rules Makefile.examples Makefile.tests
--include libffi.config
--include asneeded.config
-
-ifeq ($(libffi_available),false)
-test-libffi:
-	@echo "The following required C libraries are missing: libffi."
-	@echo "Please install them and retry. If they are installed in a non-standard location"
-	@echo "or need special flags, set the environment variables <LIB>_CFLAGS and <LIB>_LIBS"
-	@echo "accordingly and retry."
-	@echo
-	@echo " For example, if libffi is installed in /opt/local, you can type:"
-	@echo
-	@echo "   export LIBFFI_CFLAGS=-I/opt/local/include"
-	@echo "   export LIBFFI_LIBS=\"-L/opt/local/lib -lffi\""
-	@exit 1
-else:
-test-libffi:
-endif
+clean:
+	dune clean