Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/import audiobooks #888

Merged
merged 5 commits into from
May 13, 2024
Merged

Feature/import audiobooks #888

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
bdfdfe7
importer: Populate eitem_type in EItem object
sakshamarora1 Apr 10, 2024
66407e5
importer: Add video importing feature and tests
sakshamarora1 Apr 10, 2024
ded6210
installation: upgrade react-invenio-app-ils
kpsherva May 13, 2024
fa8a20c
installation: upgrade invenio-app-ils
kpsherva May 13, 2024
e789b99
tests: add eitem_type
kpsherva May 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions cds_ils/importer/XMLRecordToJson.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,15 +53,20 @@ def dump(self):
else:
is_deletable = False

init_fields = {}
if "im" in marc_record.get("leader", []):
eitem_type = "audiobook"
init_fields.update({"_eitem": {"_type": "audiobook"}})
elif "gm" in marc_record.get("leader", []):
init_fields.update(
{"document_type": "MULTIMEDIA", "_eitem": {"_type": "video"}}
)
else:
eitem_type = "e-book"
init_fields.update({"_eitem": {"_type": "e-book"}})
# MARCXML -> JSON fields translation
val = self.dojson_model.do(
marc_record,
exception_handlers=xml_import_handlers,
init_fields={"_eitem": {"_type": eitem_type}},
init_fields=init_fields,
)

if not self.ignore_missing:
Expand Down
1 change: 1 addition & 0 deletions cds_ils/importer/eitems/importer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,7 @@ def _build_eitem_json(self, eitem_json, document_pid, urls=None, description=Non
eitem_json.update(
dict(
document_pid=document_pid,
eitem_type=self.eitem_json.get("_type", "E-BOOK").upper(),
open_access=self.open_access,
identifiers=dois,
urls=self.eitem_json.get("urls", []),
Expand Down
1 change: 1 addition & 0 deletions cds_ils/importer/providers/cds/rules/values_mapping.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@
"AUDIOBOOK",
"ELECTRONIC AUDIO BK",
],
"VIDEO": ["VIDEO", "ELECTRONIC VIDEO"],
"AUDIO_CD": ["AUDIO CD"],
"DISKETTE": ["DISKETTE"],
"DVD": ["DVD"],
Expand Down
5 changes: 5 additions & 0 deletions cds_ils/vocabularies/data/document_identifiers_materials.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,5 +88,10 @@
"key": "VHS",
"type": "doc_identifiers_materials",
"text": "VHS"
},
{
"key": "VIDEO",
"type": "doc_identifiers_materials",
"text": "Video"
}
]
2 changes: 1 addition & 1 deletion requirements.pinned.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ invenio-access==2.0.0
invenio-accounts==3.5.2
invenio-admin==1.4.0
invenio-app==1.3.4
invenio-app-ils==2.0.0rc9
invenio-app-ils==3.0.0rc1
invenio-assets==3.0.2
invenio-banners==1.0.0a1
invenio-base==1.2.16
Expand Down
2 changes: 1 addition & 1 deletion setup.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ zip_safe = False
install_requires =
fuzzywuzzy>=0.18.0
python-ldap>=3.4.0,<3.5.0
invenio-app-ils[lorem,opensearch2,postgresql]==2.0.0rc9
invenio-app-ils[lorem,opensearch2,postgresql]==3.0.0rc1
sentry-sdk>=0.10.2
# migrator deps
cds-dojson==0.9.0
Expand Down
4 changes: 3 additions & 1 deletion tests/data/eitems.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@
"document_pid": "docid-2",
"internal_notes": "An internal note",
"description": "Description of the electronic item",
"open_access": true
"open_access": true,
"eitem_type": "E-BOOK"
},
{
"pid": "eitemid-2",
Expand All @@ -14,6 +15,7 @@
"internal_notes": "An internal note",
"description": "Description of the electronic item",
"open_access": false,
"eitem_type": "E-BOOK",
"urls": [
{
"description": "Protected URL",
Expand Down
60 changes: 60 additions & 0 deletions tests/importer/data/documents_with_video.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
[
{
"_eitem": {
"_type": "video",
"urls": [
{
"description": "video",
"value": "https://learning.oreilly.com/library/view/-/9781804615362/?ar"
}
]
},
"abstract": "JavaScript is the base for many other languages; if you know JavaScript, you can work with a lot of other languages and dependent frameworks easily. This course is based on the newer features that were released with the ECMAScript specification 6 and above. So, we will only discuss things from ES6 and above. In the first section of this course, you will touch base on JavaScript history, and you will get in place all the required stuff that you should have in your machine for this course. In the second section of the course, you will focus on variables and scope, and you will see the newer patterns to work with variables. Following that, in the third section, we will discuss functions and arguments, which is a critical part of this course because JavaScript treats functions as a first-class citizens, so knowing what changes have been incorporated into them in newer versions is also very essential. In the fourth section, you will learn about operators and how to better use and code them. Then you will understand the new functionality of error handling; in an application where errors are not handled properly, the usability of such an application is nearly impossible. Then we have a section dedicated to async patterns and promises where you will see a few of the latest functions in comparison with similar older functions. By the end of the course, you will get to know the hacks and tricks to improve your coding skills. What You Will Learn Learn various JavaScript hacks Learn various JavaScript concepts Understand the spread operators in JavaScript Explore optional chaining operators Explain prototypal chains Understand error handling in JavaScript Audience This course can be taken by any advanced and intermediate JavaScript learners. This course expects you to have a clear understanding of the basics of JavaScript. About The Author Basics Strong: Basics Strong is a team of technocrats from IITs who focus on solving problems using technology. They work on mission-critical projects in AI, machine learning, and BlockChain as a domain and use Java, Python, JavaScript, and a lot of tools and technologies. They love to code and program. The team believes that a strong foundation in the basics of programming concepts can help you solve any technical problem and excel in your career. Therefore, they create courses that help you build your basics and come up with ways to make complicated concepts easy to learn. All their courses are carefully crafted to include hands-on examples and comprehensive working files for practical learning.",
"agency_code": "OCoLC",
"alternative_identifiers": [
{ "scheme": "SAFARI", "value": "on1351466591" }
],
"alternative_titles": [
{ "type": "SUBTITLE", "value": "modern and advanced JavaScript" }
],
"authors": [
{
"full_name": "ILS,CDS Else,Someone",
"roles": ["AUTHOR"],
"type": "PERSON"
}
],
"document_type": "MULTIMEDIA",
"edition": "1st",
"identifiers": [
{
"material": "VIDEO",
"scheme": "ISBN",
"value": "9781804615362"
},
{
"material": "VIDEO",
"scheme": "ISBN",
"value": "1804615366"
}
],
"imprint": {
"place": "Place of publication not identified",
"publisher": "Packt Publishing"
},
"keywords": [
{
"source": "SAFARI",
"value": "JavaScript (Computer program language"
}
],
"languages": ["ENG"],
"provider_recid": "on1351466591",
"publication_year": "2022",
"subjects": [
{ "scheme": "LOC", "value": "QA76.73.J39" },
{ "scheme": "DEWEY", "value": "005.2/762" }
],
"title": "Quick JavaScript crash course"
}
]
9 changes: 9 additions & 0 deletions tests/importer/data/existing_eitems.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
"pid": "eitemid-1",
"created_by": {"type": "script", "value": "test"},
"document_pid": "docid-2",
"eitem_type": "E-BOOK",
"internal_notes": "An internal note",
"description": "Description of the electronic item",
"open_access": true
Expand All @@ -11,6 +12,7 @@
"pid": "eitemid-2",
"created_by": {"type": "script", "value": "test"},
"document_pid": "docid-2",
"eitem_type": "E-BOOK",
"internal_notes": "An internal note",
"description": "Description of the electronic item",
"open_access": false,
Expand All @@ -31,6 +33,7 @@
"pid": "eitemid-3",
"created_by": {"type": "import", "value": "springer"},
"document_pid": "docid-3",
"eitem_type": "E-BOOK",
"internal_notes": "EITEM TO MODIFY",
"description": "Description of the electronic item",
"open_access": false,
Expand All @@ -51,6 +54,7 @@
"pid": "eitemid-4",
"created_by": {"type": "import", "value": "ebl"},
"document_pid": "docid-1",
"eitem_type": "E-BOOK",
"internal_notes": "An internal note",
"description": "Description of the electronic item",
"open_access": false,
Expand All @@ -71,6 +75,7 @@
"pid": "eitemid-6",
"created_by": {"type": "import", "value": "ebl"},
"document_pid": "docid-6",
"eitem_type": "E-BOOK",
"internal_notes": "An internal note",
"description": "Description of the electronic item",
"open_access": false,
Expand All @@ -91,6 +96,7 @@
"pid": "eitemid-6A",
"created_by": {"type": "import", "value": "safari"},
"document_pid": "docid-6A",
"eitem_type": "E-BOOK",
"internal_notes": "An internal note",
"description": "Description of the electronic item",
"open_access": false,
Expand All @@ -111,6 +117,7 @@
"pid": "eitemid-7",
"created_by": {"type": "import", "value": "springer"},
"document_pid": "docid-7",
"eitem_type": "E-BOOK",
"internal_notes": "An internal note",
"description": "Description of the electronic item",
"open_access": false,
Expand All @@ -131,6 +138,7 @@
"pid": "eitemid-8",
"created_by": {"type": "user", "value": "1"},
"document_pid": "docid-8",
"eitem_type": "E-BOOK",
"internal_notes": "An internal note",
"description": "Description of the electronic item",
"open_access": false,
Expand All @@ -151,6 +159,7 @@
"pid": "eitemid-9",
"created_by": {"type": "user", "value": "1"},
"document_pid": "docid-9",
"eitem_type": "AUDIOBOOK",
"internal_notes": "An internal note",
"description": "Description of the electronic item",
"open_access": false,
Expand Down
2 changes: 2 additions & 0 deletions tests/importer/data/modify_document_data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
{"scheme": "ISBN", "value": "0987654321"}],
"_eitem": {
"pid": "eitmid-1",
"eitem_type": "E-BOOK",
"created_by": {"type": "script", "value": "springer"},
"internal_notes": "Modified",
"description": "Modified description",
Expand Down Expand Up @@ -47,6 +48,7 @@
"identifiers": [{ "scheme": "ISBN", "value": "0123456789" }],
"_eitem": {
"pid": "eitmid-1",
"eitem_type": "E-BOOK",
"created_by": {"type": "script", "value": "springer"},
"document_pid": "docid-1",
"internal_notes": "Modified",
Expand Down
140 changes: 140 additions & 0 deletions tests/importer/data/safari_video.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
<?xml version="1.0" encoding="UTF-8"?>
<collection xmlns="http://www.loc.gov/MARC21/slim" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd">
<record xmlns="http://www.loc.gov/MARC21/slim">
<leader>00000ngm a22000007i 4500</leader>
<controlfield tag="001">on1351466591</controlfield>
<controlfield tag="003">OCoLC</controlfield>
<controlfield tag="005">20221129213020.0</controlfield>
<controlfield tag="006">m o c </controlfield>
<controlfield tag="007">vz czazuu</controlfield>
<controlfield tag="007">cr cnannnuuuuu</controlfield>
<controlfield tag="008">221122s2022 xx 215 o vleng d</controlfield>
<datafield tag="040" ind1=" " ind2=" ">
<subfield code="a">ORMDA</subfield>
<subfield code="b">eng</subfield>
<subfield code="e">rda</subfield>
<subfield code="e">pn</subfield>
<subfield code="c">ORMDA</subfield>
</datafield>
<datafield tag="020" ind1=" " ind2=" ">
<subfield code="a">9781804615362</subfield>
<subfield code="q">(electronic video)</subfield>
</datafield>
<datafield tag="020" ind1=" " ind2=" ">
<subfield code="a">1804615366</subfield>
<subfield code="q">(electronic video)</subfield>
</datafield>
<datafield tag="035" ind1=" " ind2=" ">
<subfield code="a">(OCoLC)1351466591</subfield>
</datafield>
<datafield tag="037" ind1=" " ind2=" ">
<subfield code="a">9781804615362</subfield>
<subfield code="b">O'Reilly Media</subfield>
</datafield>
<datafield tag="050" ind1=" " ind2="4">
<subfield code="a">QA76.73.J39</subfield>
</datafield>
<datafield tag="082" ind1="0" ind2="4">
<subfield code="a">005.2/762</subfield>
<subfield code="2">23/eng/20221122</subfield>
</datafield>
<datafield tag="049" ind1=" " ind2=" ">
<subfield code="a">MAIN</subfield>
</datafield>
<datafield tag="245" ind1="0" ind2="0">
<subfield code="a">Quick JavaScript crash course :</subfield>
<subfield code="b">modern and advanced JavaScript.</subfield>
</datafield>
<datafield tag="250" ind1=" " ind2=" ">
<subfield code="a">[First edition].</subfield>
</datafield>
<datafield tag="264" ind1=" " ind2="1">
<subfield code="a">[Place of publication not identified] :</subfield>
<subfield code="b">Packt Publishing,</subfield>
<subfield code="c">[2022]</subfield>
</datafield>
<datafield tag="100" ind1="1" ind2=" ">
<subfield code="a">ILS,CDS Else,Someone</subfield>
</datafield>
<datafield tag="300" ind1=" " ind2=" ">
<subfield code="a">1 online resource (1 video file (3 hr., 35 min.)) :</subfield>
<subfield code="b">sound, color.</subfield>
</datafield>
<datafield tag="306" ind1=" " ind2=" ">
<subfield code="a">033500</subfield>
</datafield>
<datafield tag="336" ind1=" " ind2=" ">
<subfield code="a">two-dimensional moving image</subfield>
<subfield code="b">tdi</subfield>
<subfield code="2">rdacontent</subfield>
</datafield>
<datafield tag="337" ind1=" " ind2=" ">
<subfield code="a">computer</subfield>
<subfield code="b">c</subfield>
<subfield code="2">rdamedia</subfield>
</datafield>
<datafield tag="338" ind1=" " ind2=" ">
<subfield code="a">online resource</subfield>
<subfield code="b">cr</subfield>
<subfield code="2">rdacarrier</subfield>
</datafield>
<datafield tag="344" ind1=" " ind2=" ">
<subfield code="a">digital</subfield>
<subfield code="2">rdatr</subfield>
</datafield>
<datafield tag="347" ind1=" " ind2=" ">
<subfield code="a">video file</subfield>
<subfield code="2">rdaft</subfield>
</datafield>
<datafield tag="380" ind1=" " ind2=" ">
<subfield code="a">Instructional films</subfield>
<subfield code="2">lcgft</subfield>
</datafield>
<datafield tag="511" ind1="0" ind2=" ">
<subfield code="a">Basics Strong, presenter.</subfield>
</datafield>
<datafield tag="500" ind1=" " ind2=" ">
<subfield code="a">Published in October 2022.</subfield>
</datafield>
<datafield tag="520" ind1=" " ind2=" ">
<subfield code="a">JavaScript is the base for many other languages; if you know JavaScript, you can work with a lot of other languages and dependent frameworks easily. This course is based on the newer features that were released with the ECMAScript specification 6 and above. So, we will only discuss things from ES6 and above. In the first section of this course, you will touch base on JavaScript history, and you will get in place all the required stuff that you should have in your machine for this course. In the second section of the course, you will focus on variables and scope, and you will see the newer patterns to work with variables. Following that, in the third section, we will discuss functions and arguments, which is a critical part of this course because JavaScript treats functions as a first-class citizens, so knowing what changes have been incorporated into them in newer versions is also very essential. In the fourth section, you will learn about operators and how to better use and code them. Then you will understand the new functionality of error handling; in an application where errors are not handled properly, the usability of such an application is nearly impossible. Then we have a section dedicated to async patterns and promises where you will see a few of the latest functions in comparison with similar older functions. By the end of the course, you will get to know the hacks and tricks to improve your coding skills. What You Will Learn Learn various JavaScript hacks Learn various JavaScript concepts Understand the spread operators in JavaScript Explore optional chaining operators Explain prototypal chains Understand error handling in JavaScript Audience This course can be taken by any advanced and intermediate JavaScript learners. This course expects you to have a clear understanding of the basics of JavaScript. About The Author Basics Strong: Basics Strong is a team of technocrats from IITs who focus on solving problems using technology. They work on mission-critical projects in AI, machine learning, and BlockChain as a domain and use Java, Python, JavaScript, and a lot of tools and technologies. They love to code and program. The team believes that a strong foundation in the basics of programming concepts can help you solve any technical problem and excel in your career. Therefore, they create courses that help you build your basics and come up with ways to make complicated concepts easy to learn. All their courses are carefully crafted to include hands-on examples and comprehensive working files for practical learning.</subfield>
</datafield>
<datafield tag="588" ind1=" " ind2=" ">
<subfield code="a">Online resource; title from title details screen (O'Reilly, viewed November 21, 2022).</subfield>
</datafield>
<datafield tag="590" ind1=" " ind2=" ">
<subfield code="a">O'Reilly</subfield>
<subfield code="b">O'Reilly Online Learning Platform: Academic Edition (SAML SSO Access)</subfield>
</datafield>
<datafield tag="650" ind1=" " ind2="0">
<subfield code="a">JavaScript (Computer program language)</subfield>
</datafield>
<datafield tag="655" ind1=" " ind2="7">
<subfield code="a">Instructional films.</subfield>
<subfield code="2">lcgft</subfield>
</datafield>
<datafield tag="655" ind1=" " ind2="7">
<subfield code="a">Nonfiction films.</subfield>
<subfield code="2">lcgft</subfield>
</datafield>
<datafield tag="655" ind1=" " ind2="7">
<subfield code="a">Internet videos.</subfield>
<subfield code="2">lcgft</subfield>
</datafield>
<datafield tag="710" ind1="2" ind2=" ">
<subfield code="a">Basisc Strong (Firm),</subfield>
<subfield code="e">presenter.</subfield>
</datafield>
<datafield tag="710" ind1="2" ind2=" ">
<subfield code="a">Packt Publishing,</subfield>
<subfield code="e">publisher.</subfield>
</datafield>
<datafield tag="856" ind1="4" ind2="0">
<subfield code="u">https://learning.oreilly.com/library/view/-/9781804615362/?ar</subfield>
</datafield>
<datafield tag="994" ind1=" " ind2=" ">
<subfield code="a">92</subfield>
<subfield code="b">CHCER</subfield>
</datafield>
</record>
</collection>
Loading
Loading