Deprecate pyfilesystem backend?

[timkpaine]

Such a deprecation plan would require:

• fsspec Localfilesystem replaces pyfilesystem osfs this appears to already be the case
• s3fs replaces pyfilesystem fs-s3fs - testing but its looking good
• fsspec smb replaces fs.smbfs - this has historically been the difficulty

[tharwan]

Is it already possible to configure jupyter-fs to use fsspec for smb? I could not quite tell from the readme but I am happy to try it in our setup and help stabilise it.

[vidartf]

A major improvement that can be made to the fsspec SMB code would be to have it rely on the smbclient.scandir call (at least when detail=True is passed to fsspec's .listdir method). Then it wouldn't need to do a stat call per directory entry in listdir, which can be very expensive in terms of number of network requests. But at least that can be patched in locally, and the current behavior matches that of PyFilesystem, so I'm mentioning it more as an FYI rather than something that should block this issue.

[vidartf]

Ah, I got distracted, I meant to ask what the issue is with the fsspec smb client implementation, as you mentioned that as a requirement.

[timkpaine]

Ah, I got distracted, I meant to ask what the issue is with the fsspec smb client implementation, as you mentioned that as a requirement.

I honestly dont remember at this point but i very vaguely recall @telamonian looking into it and there being some problem. I plan on testing shortly, I'm hoping to make our integration tests against smb and s3 parameterized by both pyfilesystem and fsspec.

[timkpaine]

Is it already possible to configure jupyter-fs to use fsspec for smb? I could not quite tell from the readme but I am happy to try it in our setup and help stabilise it.

@tharwan Yes, we're now testing both pyfilesystem and fsspec against local files, s3, and samba, after #247.

Feel free to give it a test, we're probably not ready to do a release yet but maybe can do an alpha or something. @vidartf what do you think?

[vidartf]

@timkpaine If at all possible, think we should definitely have a decent length release cycle where we support both. Once fsspec has stabilized (initial rounds of bugfixes after first release with fsspec support), we can add deprecation warnings for pyfs, and then later remove support for it. This should allow to transition in a step-wise fashion (upgrade package, gradually transition their code/resources to use fsspec) instead of forcing users to upgrade in lock-step (package rollout + code/resource changes need to be rolled out transactionally).