Skip to content

fix: BROS-23: Don't fail on parsing errors in storage #7567

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: develop
Choose a base branch
from
Open
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions label_studio/io_storages/base_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -420,25 +420,26 @@ def _scan_and_create_links(self, link_class):
# w/o Dataflow
# pubsub.push(topic, key)
# -> GF.pull(topic, key) + env -> add_task()
logger.debug(f'Scanning key {key}')
logger.debug('Scanning key %s', key)
self.info_update_progress(last_sync_count=tasks_created, tasks_existed=tasks_existed)

# skip if key has already been synced
if n_tasks_linked := link_class.n_tasks_linked(key, self):
logger.debug(f'{self.__class__.__name__} already has {n_tasks_linked} tasks linked to {key=}')
logger.debug('%s link %s already exists', self.__class__.__name__, key)
tasks_existed += n_tasks_linked # update progress counter
continue

logger.debug(f'{self}: found new key {key}')
logger.debug('%s: found new key %s', self.__class__.__name__, key)
try:
tasks_data = self.get_data(key)
except (UnicodeDecodeError, json.decoder.JSONDecodeError) as exc:
logger.debug(exc, exc_info=True)
raise ValueError(
logger.warning(
f'Error loading JSON from file "{key}".\nIf you\'re trying to import non-JSON data '
f'(images, audio, text, etc.), edit storage settings and enable '
f'"Treat every bucket object as a source file"'
)
continue

if isinstance(tasks_data, dict):
tasks_data = [tasks_data]
Expand Down
Loading