Symbol not found when loading Eigen

[nilsbecker]

i have been trying to load the Eigen library on the way to loading Owl into the native toplevel on ocaml 4.10.

#use "omod.nattop";;
Omod.load Eigen;;

gives

[INC] /Users/nbecker/.opam/410nattop/lib/ocaml
[INC] /Users/nbecker/.opam/410nattop/lib/integers
[OBJ] /Users/nbecker/.opam/410nattop/lib/integers/integers.cmxs
[INC] /Users/nbecker/.opam/410nattop/lib/ctypes
[OBJ] /Users/nbecker/.opam/410nattop/lib/ctypes/ctypes.cmxs
[INC] /Users/nbecker/.opam/410nattop/lib/eigen
[OBJ] /Users/nbecker/.opam/410nattop/lib/eigen/eigen.cmxs
[ERROR] load /Users/nbecker/.opam/410nattop/lib/eigen/eigen.cmxs:
Error while loading /Users/nbecker/.opam/410nattop/lib/eigen/eigen.cmxs: error loading shared library: Dynlink.Error (Dynlink.Cannot_open_dll "Failure(\"dlopen(/Users/nbecker/.opam/410nattop/lib/eigen/eigen.cmxs, 10): Symbol not found: _c_eigen_dsmat_c_cols\\n  Referenced from: /Users/nbecker/.opam/410nattop/lib/eigen/eigen.cmxs\\n  Expected in: flat namespace\\n in /Users/nbecker/.opam/410nattop/lib/eigen/eigen.cmxs\")")

Is this a known limitation or is something wrong with the Eigen installation?

For reference, see also owlbarn/eigen#27 and ocaml/dune#3889 . Those threads are probably only interesting at the end; in the beginning there was an unrelated problem of cmxs files not being properly installed on my system probably due to leftovers from a failed installation.

[dbuenzli]

Sometimes this can happen if the external declarations mention cstubs that are defined in another library and there is no explicit OCaml-level dependency between the two (i.e. one the ocamlobjinfo can see). This happens for example in digestif_c.cma whose stubs are in a library called rakia.

The work around is to load that other library before. Is that maybe the case here ?

In any first try to see if any of the following works:

1. Try to load eigen with ocamlfind in ocaml
2. Try to load eigen with Omod in ocaml

[nilsbecker]

ok. loading with ocamlfind seems to work:

# #use "topfind";;
- : unit = ()
Findlib has been successfully loaded. Additional directives:
  #require "package";;      to load a package
  #list;;                   to list the available packages
  #camlp4o;;                to load camlp4 (standard syntax)
  #camlp4r;;                to load camlp4 (revised syntax)
  #predicates "p,q,...";;   to set these predicates
  Topfind.reset();;         to force that packages will be reloaded
  #thread;;                 to enable threads

- : unit = ()
# #require "eigen";;
/Users/nbecker/.opam/410nattop/lib/ocaml/unix.cma: loaded
/Users/nbecker/.opam/410nattop/lib/ocaml/bigarray.cma: loaded
/Users/nbecker/.opam/410nattop/lib/bytes: added to search path
/Users/nbecker/.opam/410nattop/lib/integers: added to search path
/Users/nbecker/.opam/410nattop/lib/integers/integers.cma: loaded
/Users/nbecker/.opam/410nattop/lib/ctypes: added to search path
/Users/nbecker/.opam/410nattop/lib/ctypes/ctypes.cma: loaded
/Users/nbecker/.opam/410nattop/lib/ctypes/ctypes-top.cma: loaded
/Users/nbecker/.opam/410nattop/lib/eigen/cpp: added to search path
/Users/nbecker/.opam/410nattop/lib/eigen/cpp/eigen_cpp_stubs.cma: loaded
/Users/nbecker/.opam/410nattop/lib/eigen: added to search path
/Users/nbecker/.opam/410nattop/lib/eigen/eigen.cma: loaded

loading with omod in ocaml:

# #use "omod.top";;
- : unit = ()
Omod v0.0.2 loaded. Type Omod.help () for more info.
- : unit = ()
# Omod.load "Eigen";;
[INC] /Users/nbecker/.opam/410nattop/lib/ocaml
[INC] /Users/nbecker/.opam/410nattop/lib/integers
[OBJ] /Users/nbecker/.opam/410nattop/lib/integers/integers.cma
[INC] /Users/nbecker/.opam/410nattop/lib/ctypes
[OBJ] /Users/nbecker/.opam/410nattop/lib/ctypes/ctypes.cma
[INC] /Users/nbecker/.opam/410nattop/lib/eigen
[OBJ] /Users/nbecker/.opam/410nattop/lib/eigen/eigen.cma
[ERROR] load /Users/nbecker/.opam/410nattop/lib/eigen/eigen.cma:
Cannot load required shared library dlleigen_stubs.
Reason: /Users/nbecker/.opam/410nattop/lib/stublibs/dlleigen_stubs.so: dlopen(/Users/nbecker/.opam/410nattop/lib/stublibs/dlleigen_stubs.so, 10): Symbol not found: _c_eigen_dsmat_c_cols
  Referenced from: /Users/nbecker/.opam/410nattop/lib/stublibs/dlleigen_stubs.so
  Expected in: flat namespace
 in /Users/nbecker/.opam/410nattop/lib/stublibs/dlleigen_stubs.so.

- : bool = false

fails with apparently the same issue

[dbuenzli]

So where is _c_eigen_dsmat_c_cols defined ? Also looking at the ocamlfind META and comparing with the above trace of what omod loaded should provide a hint of what omod is missing.

[nilsbecker]

the external function that is not found seems to be defined here: https://github.com/owlbarn/eigen/blob/61c47df9b7e42800f1317b5bed131785409c5abe/eigen/ffi_eigen_bindings.ml#L125
that is in a functor which seems to be applied to a generated module.

[dbuenzli]

That's the ctypes stubs generation. It must be in a C library somewhere.

[nilsbecker]

i'll try to dig deeper.

[dbuenzli]

According to the META file eigen has an indirect dependency on eigen.cpp.
The following works here:

> ocaml
        OCaml version 4.09.0

Down v0.0.3 loaded. Type Down.help () for more info.
Omod v0.0.2 loaded. Type Omod.help () for more info.
# Omod.load "eigen_cpp_stubs";;
[INC] /Users/dbuenzli/.opam/4.09.0/lib/ocaml
[INC] /Users/dbuenzli/.opam/4.09.0/lib/eigen/cpp
[OBJ] /Users/dbuenzli/.opam/4.09.0/lib/eigen/cpp/eigen_cpp_stubs.cma
- : bool = true
# Omod.load "eigen";;
[INC] /Users/dbuenzli/.opam/4.09.0/lib/integers
[OBJ] /Users/dbuenzli/.opam/4.09.0/lib/integers/integers.cma
[INC] /Users/dbuenzli/.opam/4.09.0/lib/ctypes
[OBJ] /Users/dbuenzli/.opam/4.09.0/lib/ctypes/ctypes.cma
[INC] /Users/dbuenzli/.opam/4.09.0/lib/eigen
[OBJ] /Users/dbuenzli/.opam/4.09.0/lib/eigen/eigen.cma
- : bool = true
# 

[nilsbecker]

ah cool, i'll try that. i'm a bit out of my depth with the c bindings business.

[nilsbecker]

hmm. for me the first load succeeds but not the second.

#use "omod.nattop";;
File "_none_", line 1:
Warning 58: no cmx file was found in path for module Opttopdirs, and its interface was not compiled with -opaque
- : unit = ()
Omod v0.0.2 loaded. Type Omod.help () for more info.
- : unit = ()
# Omod.load "eigen_cpp_stubs" ;;
[INC] /Users/nbecker/.opam/410nattop/lib/ocaml
[INC] /Users/nbecker/.opam/410nattop/lib/eigen/cpp
[OBJ] /Users/nbecker/.opam/410nattop/lib/eigen/cpp/eigen_cpp_stubs.cmxs
[INC] /Users/nbecker/.opam/410nattop/lib/integers
[OBJ] /Users/nbecker/.opam/410nattop/lib/integers/integers.cmxs
[INC] /Users/nbecker/.opam/410nattop/lib/ctypes
[OBJ] /Users/nbecker/.opam/410nattop/lib/ctypes/ctypes.cmxs
[INC] /Users/nbecker/.opam/410nattop/lib/eigen
[OBJ] /Users/nbecker/.opam/410nattop/lib/eigen/eigen.cmxs
[ERROR] load /Users/nbecker/.opam/410nattop/lib/eigen/eigen.cmxs:
Error while loading /Users/nbecker/.opam/410nattop/lib/eigen/eigen.cmxs: error loading shared library: Dynlink.Error (Dynlink.Cannot_open_dll "Failure(\"dlopen(/Users/nbecker/.opam/410nattop/lib/eigen/eigen.cmxs, 10): Symbol not found: _c_eigen_dsmat_c_cols\\n  Referenced from: /Users/nbecker/.opam/410nattop/lib/eigen/eigen.cmxs\\n  Expected in: flat namespace\\n in /Users/nbecker/.opam/410nattop/lib/eigen/eigen.cmxs\")").

- : bool = false

[dbuenzli]

Can you first try to see if it works in the ocaml toplevel please.

[nilsbecker]

ah sorry i missed that. (disregard the comment that was here) yes, in ocaml, omod can successfully load as you showed.

[dbuenzli]

Well according to what I see here via (nm -gu) the eigen_cpp_stubs.cmxs has not the symbols from libeigen_cpp_stubs_stubs.a so this can't work. It's needs to have the stubs in there.

[nilsbecker]

would this be a bug in the library? if so would it mean that the library is not working properly in general? (sorry for the naive questions)

[dbuenzli]

would this be a bug in the library? if so would it mean that the library is not working properly in general? (sorry for the naive questions)

Yes. In general the .cmxs paths of libraries is not well tested since you only get to use it if you use the Dynlink module or venture into the officially unsupported ocamlnat.

Have a look the build of eigen I suspect the cmxs is generated along the lines of

ocamlopt -shared -o eigen_cpp_stubs.cmxs eigen_cpp_stubs.cmxa 

but it should have something like

ocamlopt -shared -o eigen_cpp_stubs.cmxs eigen_cpp_stubs.cmxa libeigen_cpp_stubs_stubs.a 

[nilsbecker]

indeed! this line is in the build log:

$ (cd _build/default && /Users/nbecker/.opam/410nattop/bin/ocamlopt.opt -w -40 -g -shared -linkall -I eigen_cpp -o eigen_cpp/eigen_cpp_stubs.cmxs eigen_cpp/eigen_cpp_stubs.cmxa)

i'll report this on the eigen bug tracker. thanks!

[nilsbecker]

incidentally, is this just a matter of writing the correct dune file or should dune do something better?

[dbuenzli]

I don't know you should ask the dune people.

[dbuenzli]

That should be solved by ocaml/opam-repository#17613