Changeset replication stopped

[pa5cal]

The changeset replication stopped this morning again.
Latest state file has been created at:
last_run: 2021-01-02 07:11:01.391104000 +00:00
sequence: 4262676

See: https://planet.openstreetmap.org/replication/changesets/

[zerebubuth]

Happy New Year 🎆

Thanks for letting me know - I've just restarted it.

For posterity, the error leading to a zero-size state file (with an intact state.yaml.tmp, though), was:

Error! Couldn't update state.
ERROR: Bad file descriptor @ fptr_finalize_flush - /store/planet/replication/changesets/state.yaml.tmp

So it looks like the temporary state file handle is bad when the file pointer is finalized. The file handle is only in scope for a single block, and should be automatically closed (and flushed) when that block ends. But apparently it isn't, and I can't figure out why. 😖

It looks like the call to FileUtils.cp to copy the temporary state to the main state file is the point at which it fails, based on the contents/existence of the files on disk. What I could see was:

1. The target data file existed (004/262/678.osm.gz)
2. The target state file did not exist (004/262/678.state.txt)
3. The state temporary file existed, and was complete (state.yaml.tmp)
4. The main state file existed, and was zero-size (state.yaml)

FileUtils.cp seems to be implemented pretty much as File.open(src) {|s| File.open(dst, 'wb') {|d| d.write(s.read()) }}, so the File.open(dst, 'wb') seems to have run, as that would truncate the target file. But then the d.write hasn't run, therefore it must be the s.read() which triggered the finalizer on the previous open of that file?

This is some strange stuff going on. @gravitystorm, you're much more familiar with Ruby than I - does this ring any bells for you?

(PS: @pa5cal, thanks for letting me know. Please don't feel like you need to, though! I had a very full inbox when I woke up this morning, something like >200 error emails from the changeset replication process.)

[pa5cal]

@zerebubuth Could you/someone else please check the changeset replication process?
It stopped working yesterday, thanks!

[zerebubuth]

Thanks @pa5cal! And apologies - I would have fixed it sooner, but it has been a busy morning 🙇‍♂️

[pa5cal]

@zerebubuth Could you check the changeset replication process?

See https://planet.openstreetmap.org/replication/changesets/
state.yaml 2021-08-23 01:55 0

[zerebubuth]

Hi Pascal! Thanks for pointing that out. Should be fixed now. The error was the same as before - and I'm still not sure how it gets into that state 😖🤔 23 Aug 2021 07:58:30 Pascal Neis ***@***.***>:

…

@zerebubuth[https://github.com/zerebubuth] Could you check the changeset replication process? See https://planet.openstreetmap.org/replication/changesets/ state.yaml 2021-08-23 01:55 0 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub[#8 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AACCIAFTD3DZUJRJNGTMQPDT6HIPNANCNFSM4VQ4PARA]. Triage notifications on the go with GitHub Mobile for iOS[https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675] or Android[https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email]. [data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEIAAABCCAYAAADjVADoAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAAAAnSURBVHic7cEBDQAAAMKg909tDwcUAAAAAAAAAAAAAAAAAAAAAPBjRFIAASHKmKkAAAAASUVORK5CYII=###24x24:true###][Tracking image][https://github.com/notifications/beacon/AACCIADPMZ5UIY477SAD3LTT6HIPNA5CNFSM4VQ4PARKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGXMXJEI.gif]