When loading OptoDAS data with truncated files, the routine crashes instead of giving a warning

[speleos]

The expected behaviour when dealing with truncated files would be to warn the user and ignore them.

xdas/xdas/core/routines.py

Line 337 in 3ef0714

iterator = as_completed(futures)

File ~/anaconda3/envs/simpledas/lib/python3.11/site-packages/xdas/core/routines.py:338, in open_mfdataarray(paths, dim, tolerance, squeeze, engine, verbose, **kwargs)
336 else:
337 iterator = as_completed(futures)
--> 338 objs = [future.result() for future in iterator]
339 return combine_by_coords(objs, dim, tolerance, squeeze, None, verbose)

File ~/anaconda3/envs/simpledas/lib/python3.11/site-packages/xdas/core/routines.py:338, in (.0)
336 else:
337 iterator = as_completed(futures)
--> 338 objs = [future.result() for future in iterator]
339 return combine_by_coords(objs, dim, tolerance, squeeze, None, verbose)

File ~/anaconda3/envs/simpledas/lib/python3.11/concurrent/futures/_base.py:449, in Future.result(self, timeout)
447 raise CancelledError()
448 elif self._state == FINISHED:
--> 449 return self.__get_result()
451 self._condition.wait(timeout)
453 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:

File ~/anaconda3/envs/simpledas/lib/python3.11/concurrent/futures/_base.py:401, in Future.__get_result(self)
399 if self._exception:
400 try:
--> 401 raise self._exception
402 finally:
403 # Break a reference cycle with the exception in self._exception
404 self = None

OSError: Unable to synchronously open file (truncated file: eof = 5738496, sblock->base_addr = 0, stored_eof = 114278280)

[atrabattoni]

Thank you for reporting this. Indeed this routine fails when any of the file is corrupted/cannot be opened for any reason. A more user friendly solution would be to warn the user of the file that could not be opened. I will try to work on this in the next days.

[atrabattoni]

In the meantime, the workaround is to pass a list of file instead of a wildcard string being sure to exclude corrupted files from that list. Python glob module can be useful here.

[atrabattoni]

Please see the the related PR #29.