Skip to content

Commit 09ddae1

Browse files
committed
fix oscar filter
1 parent efec1c4 commit 09ddae1

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

Diff for: src/datatrove/pipeline/filters/oscar_filter.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,6 @@ def filter(self, doc: Document) -> bool | tuple[bool, str]:
5656
return False, 'kenlm_min_harmful_ppl'
5757
if doc.metadata['harmful_pp'] and doc.metadata['harmful_pp'] > self.max_harmful_ppl:
5858
return False, 'kenlm_max_harmful_ppl'
59-
if doc['medatdata']['oscar_categories'] and len(set(doc['medatdata']['oscar_categories']) & self.exclude_categories) > 0:
59+
if doc.metadata['oscar_categories'] and len(set(doc.metadata['oscar_categories']) & self.exclude_categories) > 0:
6060
return False, 'oscar_category'
6161
return True

0 commit comments

Comments
 (0)