F::unlink() error handling fails when server language not English

[twiro]

Description

I'm currently working on a new website that was running just fine on 5.0.0-beta.3, but now has some serious issues in the panel after updating to the latest beta. I could track down the one linked above (failing page creation), but am pretty stuck with another one:

Deleting pages and files fails in all cases and always returns this kind of error:

• unlink(/var/www/clients/clientXX/webXX/web/content/2_about/_changes/ocean.jpg.de.txt): Datei oder Verzeichnis nicht gefunden
• unlink(/var/www/clients/clientXX/webXX/web/content/2_about/testkapitel/_changes/kapitel.de.txt): Datei oder Verzeichnis nicht gefunden

I guess it has to do something with the new versions-feature and the "_changes"-folder not existing, but I'm not sure, as I'm not able to reproduce this behaviour myself.

I tried to track it down by rebuilding the whole site step by step under a different domain up to the moment, where I had a complete clone of the website running there (including the complete content), but everything kept working fine, including the deleting of files and pages.

I then copied my working clone from Domain B and replaced my original code of Domain A (fixing the configs/urls for sure) only to face the error again… no deleting of pages or files possible.

Both domains run on the same server (Timme/Nginx) with exactly the same configuration.

Pretty strange, but even if I can't reproduce it and it might be somehow releated to my specific domain/server-setup I thought I'd post a bug report nonetheless, as this errors are a direct result of updating to the latest beta and we didn't face them before (with the same server/domain-setup).

Screenshot

Your setup

Kirby Version: 5.0.0.beta4

[bastianallgeier]

I'm not able to reproduce this so far. Do you have any plugins installed?

[twiro]

Yes, quite some, but the error remains, even if I remove all plugins.

On the other hand everything works fine again, as soon as I switch back from beta4 to beta3… I wish I could give more helpful information here, but as I said, I wasn't even able to reproduce this on the same server :/

[bastianallgeier]

I'm cc-ing @lukasbestle here. Could this be related to permissions or something else? I'm quite lost. Our code should simply ignore non-existing files when deleting.

[lukasbestle]

To be honest it's super hard to track down such issues without specific pointers to a root cause. You can help us find and fix the issue with additional debugging. It would already help a lot to narrow down either the change in Kirby that caused the regression or the custom configuration in your site code that causes it. So here are two approaches to debug this:

• You could install the kirby folder as a Git repository and use git bisect to find the commit between beta 3 and 4 that caused the issue to your site. With a specific commit, we know where to start searching for root causes.
• You write that it works with the code of domain B but not with the code of domain A. Instead of replacing the entire code, you could replace parts until you find a particular configuration or code part that causes the behavior. This then makes it easier for us to reproduce the issue on our end.

[SylvainAllignol]

I have the same problem and maybe find something interesting : I use multilanguage. If I create a page in 1 language only and try to delete, I get the same error.
For exemple:

• I use french and english
• I create a page with slug some-page. French is my default language
• If I don't do anything else, only some-page.fr.php is created, not some-page.en.php
• Then I can't delete the page

[bastianallgeier]

@SylvainAllignol this is really helpful. Thank you!

[bastianallgeier]

I still cannot reproduce this properly, but I found an issue that could be related: #7097

[twiro]

I have the same problem and maybe find something interesting : I use multilanguage. If I create a page in 1 language only and try to delete, I get the same error…

Are you sure, that it actually depends on the translated files existing? Because my setup is multilingual too (DE/EN) and while trying to reproduce your conclusions I noticed something different that's enabling/disabling these errors in my setup:

No matter if my content files exist only in german, only in english or in both languages and no matter what language was active when first creating them, the error when trying to delete them stayed always the same as long as I tried deleting them while using the panel in my main language (DE) – as soon as I switch to the secondary language (EN) and try deleting pages (and files) then everything works as expected!

Just wanted to share this quickly, as it might help finding the root of the problem… I don't have much time left today, but will check if Bastians PR makes a difference…

[twiro]

I still cannot reproduce this properly, but I found an issue that could be related: #7097

I manually changed that line in the file PageActions.php but so far I see no difference regarding the errors…

[twiro]

You could install the kirby folder as a Git repository and use git bisect to find the commit between beta 3 and 4 that caused the issue to your site. With a specific commit, we know where to start searching for root causes.

I'm sorry, but I'm too much of a Git-Noob to choose this method…

You write that it works with the code of domain B but not with the code of domain A. Instead of replacing the entire code, you could replace parts until you find a particular configuration or code part that causes the behavior. This then makes it easier for us to reproduce the issue on our end.

… but I will try this route again after I found out that it depends on the panel language while trying deleting something – might have missed something at my first try to reproduce a setuo with the same errors, because I didn't focus on the language settings!

[distantnative]

No matter if my content files exist only in german, only in english or in both languages and no matter what language was active when first creating them, the error when trying to delete them stayed always the same as long as I tried deleting them while using the panel in my main language (DE) – as soon as I switch to the secondary language (EN) and try deleting pages (and files) then everything works as expected!

I created the same setup as @twiro - DE as default, EN as secondary. But no matter which combination of creating a page and deleting it in whichever language it does work for me without problems.

[twiro]

So after having found out that those failed deletions depend on the current language in the panel I was able to reproduce this behaviour on my NGINX server (where the problems first appeared) with the following steps:

Steps to reproduce these errors (on an nginx server):

1. Upload Starterkit (via FTP)
2. Replace the Kirby folder with 5.0.0.beta4
3. Add these lines to the config: 'panel' =>['install' => true],
4. Call domain.com/panel and create an account

Check deleting pages & files: ✅

5. Add these lines to the config: 'languages' => true,

Check deleting pages & files: ✅

6. Create first language "German" (de)
7. Create second language "English" (en)

Check deleting pages & files (with de as panel language): ❌
Check deleting pages & files (with en as panel language): ✅

Not reproducable on an apache server

I then counterchecked these steps on an Apache based server and didn't hit these errors there, so it might actually have to do something with NGINX in general or my NGINX-configuration in spcific (which worked flawlessly for all kirby versions from 4.X to 5.0.0.beta3) – I will check my configuration and test some variations as soon as possible.

Language weirdness

I also started adding more languages to my setup to see how deleting content works with them, and here things start to get a little weird, because it doesn't seem like the problem comes from "de" being the main language but stems from the language itself as additional languages also behave differently:

8. Create third language "French" (fr)
9. Create fourth language "Spanish" (es)

Check deleting pages & files (with fr as panel language): ❌
Check deleting pages & files (with es as panel language): ✅

I will perform another test run with EN as main language later…

[twiro]

So after another test I can confirm that these errors don't depend on whether a language is the default language but somehow on the languages itself.

I repeated the steps described in my previous post, but this time added ten languages in the following order and then tested deleting pages and files with each language selected as panel language…

Panel Language	Locale	Deleting pages & Files
Albanian (SQ)	sq_AL	✅
Dutch (NL)	nl_NL	❌
English (EN)	en_US	✅
Finish (FI)	fi_FI	❌
French (FR)	fr_FR	❌
German (DE)	de_DE	❌
Greek (EL)	el_GR	❌
Italian (IT)	it_IT	❌
Japanese (JA)	ja_JP	✅
Korean (KO)	ko_KR	✅
Polish (PL)	pl_PL	❌
Portuguese (PT)	pt	✅
Spanish (ES)	es	✅
Swedish (SV)	sv_SE	❌
Turkish (TR)	tr_TR	❌
Ukrainian (UK)	uk_UA	✅
Vietnamese (VI)	vi_VN	✅

Pretty weird 😲

@SylvainAllignol – is there anything you can reproduce and confirm about this?

[bastianallgeier]

@twiro thank you for the extensive tests. Is the result reproducable for the languages that cannot be deleted? I somehow start thinking that this could also be some sort of filesystem cache issue on the server.

[twiro]

Is the result reproducable for the languages that cannot be deleted?

Yep! And while doublechecking this I noticed an error in my previous test (used 'po' for polish instead of 'pl') and that led me one step further in tracing down this error – the locales seem to be the source of the trouble!

After fixing the polish language code from 'po' to 'pl' the polish locale (that was empty before) got autofilled to 'pl_PL' and after that the deletion of pages/files stopped working with polish as panel language too.

So I checked all locales and languages with 'simple' locales ('es','pl','pt') didn't have problems with deleting pages/files while languages with 'complex' locales ('de_DE', 'it_IT') do cause these problems – as soon as i shorten the locale deleting pages/files works again!

This conclusion once more comes with an exception - english comes with the locale 'en_US' and does not cause any trouble…

[bastianallgeier]

@twiro that's a great next step to debug this.

Could you send me the filename in site/languages of a language that cannot be deleted and also the content of the language.php file?

[distantnative]

And maybe too far fetched, but the created content files are named .de.txt or .de_DE.txt?

[twiro]

Could you send me the filename in site/languages of a language that cannot be deleted and also the content of the language.php file?

Just to be clear and avoid misunderstandings – the languages / language files can always be deleted.

The issue this is all about is that (in my nginx environment and after the update to beta4) every page and file in the panel can no more be deleted when the panel is set to a language that uses a locale that follows thexy_XY scheme (except for en_US).

As soon as I manually edit the locale of an affected language and reduce it to xy there are no more errors when trying to delete a page or file while having the panel set to this language.

The language files itself are properly named de.php or en.php and look perfectly fine imho:

return [
    'code' => 'de',
    'default' => true,
    'direction' => 'ltr',
    'locale' => [
        'LC_ALL' => 'de_DE'
    ],
    'name' => 'Deutsch',
    'url' => NULL
];

return [
    'code' => 'en',
    'default' => false,
    'direction' => 'ltr',
    'locale' => [
        'LC_ALL' => 'en_US'
    ],
    'name' => 'Englisch',
    'url' => NULL
];

Should I still send the files…? Or would a short video help maybe? I could also give you panel access to a test installation with this behavior.

And maybe too far fetched, but the created content files are named .de.txt or .de_DE.txt?

Properly named *.de.txt, *.en.txt and so on.

As this issue affects all pages and files (created prior to the update to beta4 and all named properly) It doesn't look like it has to do something with wrong naming…

[distantnative]

@twiro We are not asking for all those details implying you named something wrong. We just haven't been able to replicate this problem ourselves, so we are asking for all the details to see if in our attempts to recreate it, we're doing anything differently form you that might help to replicate it. :)

[twiro]

@twiro We are not asking for all those details implying you named something wrong. We just haven't been able to replicate this problem ourselves, so we are asking for all the details to see if in our attempts to recreate it, we're doing anything differently form you that might help to replicate it. :)

@distantnative no problem :) Didn't even understand it that way as I thought you meant Kirby could somehow use the full locale for the naming of files ;)

[twiro]

I tested a few more languages/locales this morning, and the results don't make things any easier to grasp…

It's not that "en_US" is the only longer locale that doesn't cause problems, there's actually quite a few of them (e.g. Albaninan, Japanese, Ukrainian, Vietnames)! But I really don't see any pattern in this :/

The only thing that seems to be consistent is, that if I (however) remove the underscore (e.g. "de-DE" instead of "de_DE") every locale that caused deletion-errors before immediately stops throwing errors.

So isn't the main question now, what the locale of the current panel language actually has to do with deleting pages and files in the panel and what has changed inbetween beta3 and beta4 that could make some locales with an underscore causing these kind of errors shown in the screenshot in my first posting?

This is actually as far as I can dig, I don't know what more I could test now… I guess it would be really helpful if someone could confirm this behavior, so maybe @SylvainAllignol could try to also test deleting of pages/files with different panel languages/locales and see if he can confirm the locales beeing involved here…

[afbora]

I couldn't reproduce the issue either on my local Herd setup.

[twiro]

* You could install the `kirby` folder as a Git repository and use `git bisect` to find the commit between beta 3 and 4 that caused the issue to your site. With a specific commit, we know where to start searching for root causes.

Didn't really do it the git way, but I started testing different steps of the repository between beta3 and beta4 and could track down the error to commit d0e671a

Before this commit everything works as expected, after that commit the weird deleting errors show up!

I hope this helps in getting closer to a solution…

[afbora]

@twiro Maybe you can share a kit with us for the issue so we can reproduce it. I tried again locally, but I couldn't reproduce the issue. I put it on my Apache server, and it still seems to works fine.

It seems like this issue is server related. I just wonder that can you reproduce this issue on any nginx machine?

[twiro]

Alright… I spent half the day digging further into this issue and I think I finally know what's going on and how this issue is related to languages/locales. The crazy thing is that my first screenshot already shows the crucial detail - in the form of the german-language error message "Datei oder Verzeichnis nicht gefunden".

But for a better understanding let's start with how I got there…

After having found the commit that started this issues I started my search at these new lines in the delete-function in kirby/src/Content/Version.php:

1) kirby/src/Content/Version.php

if ($language === '*') {
	foreach (Languages::ensure() as $language) {
		$this->delete($language);
	}
	return;
}

This calls the delete-function for every panel language, no matter if the version has a file in that language or not. It then passes version and language to the delete-function of the model ($this->model->storage()->delete($this->id, $language);) which somehow (I actually forgot and can't recall) led me to the delete-function in kirby/src/Content/PlainTextStorage.php:

2) kirby/src/Content/PlainTextStorage.php

/**
 * Deletes an existing version in an idempotent way if it was already deleted
 */
public function delete(VersionId $versionId, Language $language): void
{
	$contentFile = $this->contentFile($versionId, $language);

	// @codeCoverageIgnoreStart
	if (F::unlink($contentFile) !== true) {
		throw new Exception(message: 'Could not delete content file');
	}
	// @codeCoverageIgnoreEnd

…

This confused me a little bit, as this function is described as deleting an ‘existing version’, but since the Beta4 version it is called for both the ‘latest’ and ‘changes’ versions in every available panel language - regardless of whether files exist for these versions or languages. The unlink function is therefore called for a whole series of non-existent files, especially in configurations with many languages (and translations that have not been entered consistently).

But this function also drew my attention to the error messages, because they didn't have the wording "Could not delete content file", but "Datei oder Verzeichnis nicht gefunden" if I used german as panel language or alternatively "Aucun fichier ou dossier de ce type" if I used French as the panel language, for example.

3) kirby/src/Filesystem/F.php

So I had a look into the unlink-function in kirby/src/Filesystem/F.php and stumbled upon this english error message that was used in a string-comparison:

public static function unlink(string $file): bool
{
	return Helpers::handleErrors(
		fn (): bool => unlink($file),
		// if the file or link was already deleted (race condition),
		fn (int $errno, string $errstr): bool => Str::endsWith($errstr, 'No such file or directory'),
		// consider it a success
		true
	);
}

And this looked like the source of all theses troubles… "Datei oder Verzeichnis nicht gefunden" is not the same as "No such file or directory"!

4) kirby/src/CMS/Helpers.php

I then jumped into Helpers::handleErrors and included a little log-function, that wrote the values of the $errstr to my php-logfile and this is the (at this moment no more really suprising) result of my logs:

5.A) Test Results with full locales

Panel Language	Locale	PHP Unlink Error Message	Deleting pages & Files
Albanian (SQ)	sq_AL	No such file or directory	✅
Dutch (NL)	nl_NL	Bestand of map bestaat niet	❌
English (EN)	en_US	No such file or directory	✅
Finish (FI)	fi_FI	Tiedostoa tai hakemistoa ei ole	❌
French (FR)	fr_FR	Aucun fichier ou dossier de ce type	❌
German (DE)	de_DE	Datei oder Verzeichnis nicht gefunden	❌
Greek (EL)	el_GR	ÔøΩÔøΩÔøΩ…	❌
Italian (IT)	it_IT	File o directory non esistente	❌
Japanese (JA)	ja_JP	No such file or directory	✅
Korean (KO)	ko_KR	No such file or directory	✅
Polish (PL)	pl_PL	Nie ma takiego pliku ani katalogu	❌
Portuguese (PT)	pt	No such file or directory	✅
Spanish (ES)	es	No such file or directory	✅
Swedish (SV)	sv_SE	Filen eller katalogen finns inte	❌
Turkish (TR)	tr_TR	BÔøΩyle bir dosya ya da dizin yok	❌
Ukrainian (UK)	uk_UA	No such file or directory	✅
Vietnamese (VI)	vi_VN	No such file or directory	✅

5.B) Test Results with stripped locales

The last thing I checked was if the error message actually changes if I remove the second part of the locales and as expected it does:

Panel Language	Locale	PHP Unlink Error Message	Deleting pages & Files
Albanian (SQ)	sq	No such file or directory	✅
Dutch (NL)	nl	No such file or directory	✅
English (EN)	en	No such file or directory	✅
Finish (FI)	fi	No such file or directory	✅
French (FR)	fr	No such file or directory	✅
German (DE)	de	No such file or directory	✅
Greek (EL)	el	No such file or directory	✅
Italian (IT)	it	No such file or directory	✅
Japanese (JA)	ja	No such file or directory	✅
Korean (KO)	ko	No such file or directory	✅
Polish (PL)	pl	No such file or directory	✅
Portuguese (PT)	pt	No such file or directory	✅
Spanish (ES)	es	No such file or directory	✅
Swedish (SV)	sv	No such file or directory	✅
Turkish (TR)	tr	No such file or directory	✅
Ukrainian (UK)	uk	No such file or directory	✅
Vietnamese (VI)	vi	No such file or directory	✅

Conclusion

I haven't investigated further for today and I don't know if the translated error messages have anything to do with the my Server, PHP- or NGINX-settings, but regardless, the problem should probably be best fixed directly with a more reliable solution in the unlink-function in kirby/src/Filesystem/F.php, right?

[distantnative]

@twiro that's great debugging, thanks for getting to the bottom of this!

Indeed we will need a more reliable way to check for the error type if the error messages not always are in English. Pinging @lukasbestle here as well.

[lukasbestle]

@twiro Thank you very much for your research and detailed explanations. I didn't make the connection between the change to F::unlink() and the warning text of the PHP unlink() warning. But now it makes a lot of sense.

However what I'm really confused by is that your PHP installation localizes warnings in the first place (and then only to some locales). I have looked into the PHP source code and found that they have a unit test for the English warning and hardcode that in all sorts of places. I can't get my local installation to localize that warning, even though it is compiled with libintl.

I have an idea though. You wrote that the warning message depends on the locale. Could you please test what output the following script creates on your affected machine:

setlocale(LC_MESSAGES, 'de_DE');
unlink('this-file-does-not-exist');

setlocale(LC_MESSAGES, 'C');
unlink('this-file-does-not-exist');

[twiro]

However what I'm really confused by is that your PHP installation localizes warnings in the first place (and then only to some locales). I have looked into the PHP source code and found that they have a unit test for the English warning and hardcode that in all sorts of places. I can't get my local installation to localize that warning, even though it is compiled with libintl.

indeed quite strange! Especially since translated error messages would only make sense - if at all - if they were dependent on the account/system language and not on the panel/content language (I have just realised that this point could also be misunderstood in my previous comments - by ‘panel language’ I have always meant the languages of the contents (that can be managed & changed in the panel) and not the language of the Kirby interface (that can be changed in the account settings)).

I'm using the default PHP settings from my hoster (Timme) and haven't had any similar issues, but if this is something that shouldn't be happening and the PHP settings are to blame, I'll try to contact them and ask if this could be avoided by using different default PHP settings on their end.

I have an idea though. You wrote that the warning message depends on the locale. Could you please test what output the following script creates on your affected machine:

setlocale(LC_MESSAGES, 'de_DE');
unlink('this-file-does-not-exist');

setlocale(LC_MESSAGES, 'C');
unlink('this-file-does-not-exist');

Just checked this (with a little varied file names) and the result is the following:

unlink(datei-de): Datei oder Verzeichnis nicht gefunden in /x/y/z/
unlink(datei-c): No such file or directory in /x/y/z/

[bastianallgeier]

@twiro thank you so much for all your help. We finally seem to have a solution :)

[lukasbestle]

but if this is something that shouldn't be happening and the PHP settings are to blame, I'll try to contact them and ask if this could be avoided by using different default PHP settings on their end.

Quick reply as I'm currently not in the office: Everything is alright, the hosting provider does not need to change anything. It's a bug in Kirby and the result of your test confirmed it. We are working on a solution.

[lukasbestle]

Longer reply:

It turns out to be expected that error and warning messages can be localized by PHP. This is handled by the libintl library that can optionally be compiled into PHP. Still not sure why my local PHP installation with libintl doesn't come with the localized strings, but that's not important.

The LC_MESSAGES locale controls exactly these localized error and warning messages. Kirby sets the full locale (which includes LC_MESSAGES) based on the content language as it also controls stuff like number formatting. This is why you were seeing the localized warning message depending on the content language. It's not optimal and ideally we would control LC_MESSAGES based on the user language, but PHP errors and warnings are not really common in Kirby.

So TL;DR: We didn't realize that the warning can be localized at all because it wasn't localized in our local setups. With Bastian's PR, we fix it by always using the default/unlocalized C locale during the error handling process.

[twiro]

@twiro thank you so much for all your help. We finally seem to have a solution :)

Thank you :) I'm glad the investigations payed off and I could help solving this just in time for the release of RC1!

Quick reply as I'm currently not in the office: Everything is alright, the hosting provider does not need to change anything. It's a bug in Kirby and the result of your test confirmed it. We are working on a solution.

The LC_MESSAGES locale controls exactly these localized error and warning messages. Kirby sets the full locale (which includes LC_MESSAGES) based on the content language as it also controls stuff like number formatting. This is why you were seeing the localized warning message depending on the content language. It's not optimal and ideally we would control LC_MESSAGES based on the user language, but PHP errors and warnings are not really common in Kirby.

Thank you for these explanations, makes more sense now (and I'm also relieved that there's nothing wrong with the settings on my server).