opam install --deps-only --locked has nothing to do or installs wrong version

[sim642]

I had the following in the opam package files in the current directory:

simmo@goblint-new:/mnt/goblint-svcomp/sv-comp/goblint-clean/goblint$ grep ppxlib goblint.opam
  "ppxlib" {>= "0.30.0"}
simmo@goblint-new:/mnt/goblint-svcomp/sv-comp/goblint-clean/goblint$ grep ppxlib goblint.opam.locked
  "ppxlib" {= "0.32.1"}

Trying to install those dependencies suggests the following:

simmo@goblint-new:/mnt/goblint-svcomp/sv-comp/goblint-clean/goblint$ opam install --deps-only .
[NOTE] It seems you have not updated your repositories for a while. Consider updating them with:
       opam update

The following actions will be performed:
=== recompile 4 packages
  ↻ goblint-cil         2.0.4 (pinned)   [uses ppx_deriving_yojson]  
  ↻ ocaml-monadic       0.5              [uses ppxlib]
  ↻ ppx_deriving_yojson 3.7.0            [uses ppxlib]
  ↻ ppx_distr_guards    0.3              [uses ppxlib]
=== upgrade 4 packages
  ↗ ppx_blob            0.7.2 to 0.8.0   [required by goblint]
  ↗ ppx_deriving        5.2.1 to 6.0.2   [required by goblint]
  ↗ ppx_deriving_hash   0.1.1 to 0.1.2   [required by goblint]
  ↗ ppxlib              0.28.0 to 0.32.1 [required by goblint]

Proceed with ↻ 4 recompilations and ↗ 4 upgrades? [y/n] n

Even though I hadn't done opam update for a while, my local repositories were aware of the existence of ppxlib.0.32.1.

However, with --locked, suddenly there's nothing to do:

simmo@goblint-new:/mnt/goblint-svcomp/sv-comp/goblint-clean/goblint$ opam install --deps-only --locked .
[NOTE] It seems you have not updated your repositories for a while. Consider updating them with:
       opam update

Nothing to do.

Which is clearly wrong: with --locked the required dependency is not installed.

Then I updated my local repositories:

simmo@goblint-new:/mnt/goblint-svcomp/sv-comp/goblint-clean/goblint$ opam update

<><> Updating package repositories ><><><><><><><><><><><><><><><><><><><><><><>
[default] synchronised from git+https://github.com/ocaml/opam-repository.git
default (at git+https://github.com/ocaml/opam-repository.git):
    [WARNING] opam is out-of-date. Please consider updating it (https://opam.ocaml.org/doc/Install.html)


<><> Synchronising development packages <><><><><><><><><><><><><><><><><><><><>
[camlidl.1.12] synchronised (no changes)
[goblint-cil.2.0.4] synchronised (no changes)
[apron.v0.9.15] synchronised (no changes)
Now run 'opam upgrade' to apply any package updates.

(The pinned packages match the opam package files in the directory.)

And tried again:

simmo@goblint-new:/mnt/goblint-svcomp/sv-comp/goblint-clean/goblint$ opam install --deps-only --locked .
The following actions will be performed:
=== recompile 5 packages
  ↻ goblint-cil         2.0.4 (pinned)   [uses ppx_deriving_yojson]  
  ↻ ocaml-monadic       0.5              [uses ppxlib]
  ↻ ppx_blob            0.7.2            [uses ppxlib]
  ↻ ppx_deriving_yojson 3.7.0            [uses ppx_deriving]
  ↻ ppx_distr_guards    0.3              [uses ppxlib]
=== upgrade 3 packages
  ↗ ppx_deriving        5.2.1 to 6.0.3   [required by goblint]
  ↗ ppx_deriving_hash   0.1.1 to 0.1.2   [required by goblint]
  ↗ ppxlib              0.28.0 to 0.33.0 [required by ppx_deriving]  

Proceed with ↻ 5 recompilations and ↗ 3 upgrades? [y/n] n

Which is also wrong: with --locked the wrong version of the required dependency is to be installed.

For reference, I didn't have the latest opam when doing this, so maybe something has been recently fixed:

# opam config report
# opam-version         2.2.0~alpha~dev (061d7305ce1f555c6ead0c81c5d8874504426b39) 
# self-upgrade         no
# system               arch=x86_64 os=linux os-distribution=ubuntu os-version=22.04
# solver               builtin-mccs+glpk
# install-criteria     -removed,-count[avoid-version,changed],-count[version-lag,request],-count[version-lag,changed],-count[missing-depexts,changed],-changed
# upgrade-criteria     -removed,-count[avoid-version,changed],-count[version-lag,solution],-count[missing-depexts,changed],-new
# jobs                 15
# repositories         1 (version-controlled)
# pinned               3 (git)
# current-switch       /mnt/goblint-svcomp/sv-comp/goblint-clean/goblint
# ocaml:native         true
# ocaml:native-tools   true
# ocaml:native-dynlink true
# ocaml:stubsdir       /mnt/goblint-svcomp/sv-comp/goblint-clean/goblint/_opam/lib/ocaml/stublibs:/mnt/goblint-svcomp/sv-comp/goblint-clean/goblint/_opam/lib/ocaml
# ocaml:preinstalled   false
# ocaml:compiler       4.14.0+options+flambda

[kit-ty-kate]

That looks like a duplicate of #5567 which has a fix #6209 currently being reviewed. It will be fixed in opam 2.4.0 due to release around April (first alpha planned for March).

Would you be able to try the PR out to make sure it is indeed a duplicate and the fix correctly fixes it?

[sim642]

Unfortunately I already wiped the opam switch and recreated it from scratch.

That issue/PR seem to be about pinned packages, but neither the ppxlib nor goblint package (whose directory I'm in) were pinned in my switch. Or did the pinned goblint-cil package (to be upgraded in the same operation) also screw up other package installs then?

[kit-ty-kate]

I was able to reproduce with 2.3.0~alpha1. It seems that the interaction between --deps-only (--check also breaks) and packages that exist in opam-repository but where you want the local version, is indeed very broken. Once you pin the local goblint the problem goes away.

That's pretty annoying, i'll have a go at fixing it. Thanks for the report.

[rjbou]

Discussion in dev meeting:
Atoms need to contains the good version of the newly pinned package, but it can't be done too early. It should be done once source_pin has been called as it sets the pinned package version. Once that done, it is possible to set the version in the atoms list and return it.
see #6390 too for more complete fix