Create a custom deploy process to avoid symlink issues

[jkrumbiegel]

(I thought I had posted this before but I can't find a reference to it anymore, so maybe I never sent it or I'm just not seeing it)

We're seeing 404 issues all over the place (e.g. #241, MakieOrg/AlgebraOfGraphics.jl#631) and it looks like the root cause is vuejs/vitepress#4667

This happens because the Documenter deploy process creates a bunch of symlinks like stable which are resolved to full copies by the gh-pages deploy process later. This means that the vitepress pages are displayed from different URLs later than what they were built for, for example stable instead of v1.14.7. But due to the dynamic nature of vitepress this cannot generally be done without errors.

So what I think we need instead is our own deploy function which should simply run only the vitepress build step, which is fairly quick, multiple times, once for each symlink. So if there's a new version v1.14.7 tagged, there should be three builds, one with v1.14.7, one with v1.14 and one with stable.

Currently this is not easy to do because Documenter doesn't assume multi-version builds in its deploy function, so we need our own version. The julia side of the rendering process should also only be done once to not waste resources.

[jkrumbiegel]

@asinghvi17 or @lazarusA any thoughts on this? The current situation where lots of docs 404 is not great

[lazarusA]

if this works, we should do it. Unfortunately, I'm not very familiar with the Documenter build process. But, if you could start/lead with the a custom function to do this, I will be happy to test and iterate over it.

Also, we can use DocumenterVitepress for testing since is also having the same issue.

[asinghvi17]

Agreed, this would be nice to have. It looks like there is simply no chance for a Vitepress site to be relocatable given the response to the issue you posted...

[jkrumbiegel]

Basically the current render function needs to be split in two, first the documenter pages should be rendered and then N vitepress builds be triggered. Each build gets its own folder and then these multiple folders need to get synced into gh-pages. Currently it's assumed there's just one build. The difference is not that complicated but I looked a bit at the code and it was too confusing to quickly refactor it

[lazarusA]

mmm... I must say, that I'm very confused now on why then Rasters docs are working fine, even when v0.xx.x are published.

Docs without hydration error:
https://rafaqz.github.io/Rasters.jl/stable/

See gh-pages structure: https://github.com/rafaqz/Rasters.jl/blob/gh-pages/versions.js

and then the redirects:

note there is always a redirect between a v0.xx and the v0.xx.x versions:

[jkrumbiegel]

Hm yes that's confusing, also when clicking links on that page, the stable in the URL stays intact. It never did that for my docs even when they didn't immediately 404, so there must be some difference. But I didn't see anything in the github workflow or make.jl

[lazarusA]

Maybe the gh-pages workflow? Will need to investigate. DimensionalData.jl also works fine for the stable links. Rasters at some point was given 404 but after fixing all broken links it went away.

EDIT:

DimensionalData is at v0.29.15 , so definitely more than 6 characters.

[lazarusA]

oh, gosh! the main difference is indeed this extra step:

https://github.com/rafaqz/DimensionalData.jl/blob/gh-pages/.github/workflows/DeployDocs.yml

we have the same in Rasters, and Lux.jl also (which also works fine!).

[thofma]

DimensionalData.jl and Rasters.jl do the custom gh-page deployment to work around some (other?) bug with broken links.

Edit: I thought that this was exactly done to work around those 404 issues.

See also #81 and rafaqz/Rasters.jl#709

Edit edit: This "solution" is still not ideal, see #187.

[lazarusA]

yes, I just thought we were already doing it here and also in AoG 😓 .

[jkrumbiegel]

Ah see that workaround is basically what the multiple builds I suggested would do, just without the caveats.

[lazarusA]

ok, pushing to gh-pages fixed the issue for DVpress now: https://luxdl.github.io/DocumenterVitepress.jl/stable/

should we document this work-around as well in the docs?

[jkrumbiegel]

jkrumbiegel#1 this PR is a first attempt at doing possibly multiple vitepress builds off one Documenter render. I think in principle that works, but once we have the multiple folders, the deploydocs function from Documenter is still not right for that, because it again determines the subfolder to deploy to automatically, and we have to decide that instead. @lazarusA is that enough for you to play around with?

[thofma]

Note that the workaround also has a caveat see #187.

This not just changed the symbolic links, but also changed all the strings that has the same pattern v0.12.3 in the entire documentation.

(Emphasis mine.)

[lazarusA]

yeah, is a can full of worms 😢 . Latest release of Rasters stable is again failing :(

[jkrumbiegel]

Yes, I'm pretty sure we have to do what I'm proposing due to the way vitepress works.