Skip to content

Commit e56b85b

Browse files
Merge pull request #10 from Unity-Technologies/release/AME.2025.04
release/AME.2025.04
2 parents 0256073 + 09adee6 commit e56b85b

13 files changed

+620
-284
lines changed

CHANGELOG.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,22 @@ All notable changes to this package will be documented in this file.
55
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
66
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
77

8+
## [0.4.0] - 2025-02-28
9+
10+
### Added
11+
- Added keybidings to quit the application and to go back to the previous question in interactive mode.
12+
13+
### Changed
14+
- Updated Python SDK dependency for bulk_upload to 0.10.6
15+
816
## [0.3.0] - 2025-02-04
917

1018
### Added
1119
- Added the column description in the csv to allow edit of individual asset descriptions.
1220
- Add multiple user input validations during the interactive config mode.
1321
- Added `app_settings.json` file to configure applications amount of parallel workers and environment variables when needed.
22+
- Added the capacity to retry step when there's a failure during the pipeline.
23+
- Added a default timeout of 5 mins for http calls to help with the uploads of bigger files.
1424

1525
### Changed
1626
- Organizations and projects are now selected via a list in the interactive config mode and delete mode.

bulk_upload_cli/README.md

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ Find and connect support services on the [Help & Support](https://cloud.unity.co
2020
- [Editing metadata in the csv file](#editing-metadata-in-the-csv-file)
2121
- [Use an existing configuration file](#use-an-existing-configuration-file)
2222
- [Fine-tune the asset creation and upload](#fine-tune-the-asset-creation-and-upload)
23+
- [Use keybindings](#use-keybindings)
2324
- [Troubleshoot](#troubleshoot)
2425
- [See also](#see-also)
2526
- [Tell us what you think](#tell-us-what-you-think)
@@ -62,27 +63,31 @@ The bulk upload sample script is provided under the [Unity ToS license](../LICEN
6263
* On Mac: `python3 bulk_cli.py --create`
6364
* On Windows: `python bulk_cli.py --create`
6465

66+
### Select an action
67+
68+
Select one of the two possible actions:
69+
70+
- **Upload local assets**: Select this option to upload assets from your local machine to the cloud. See the [Select the input method](#select-the-input-method) section for more information.
71+
- **Update assets' metadata**: Select this option to update the metadata of assets in the cloud. See the [Creating a csv from a Unity Cloud project](#creating-a-csv-from-a-unity-cloud-project) section for more information.
6572
### Select the input method
6673

67-
Select one of the three strategies as the input method for bulk asset creation:
74+
Select one of the four strategies as the input method for bulk asset creation:
6875

69-
- **listed in a casv respecting the CLI tool template**: Select this option if you built a CSV listing your assets location and details using the provided template.
76+
- **listed in a csv respecting the CLI tool template**: Select this option if you built a CSV listing your assets location and details using the provided template.
7077
* Provide the path to the csv file.
7178
- **in a .unitypackage file**: Select this option if your assets are in a .unitypackage file. The tool extracts the assets from the .unitypackage file and uploads them to the cloud.
7279
* Provide the path to the .unitypackage file.
7380
- **in a local unity project**: Select this option if your assets are in a local Unity project.
7481
* Provide the path to the asset folder of the Unity project.
75-
- **in UnityCloud**: Select this option if your assets are in a Unity Cloud project.
76-
* Provide the organization ID and project ID. As of now, this features is solely to update tags and metadata of assets in the cloud. See section [Creating a csv from a Unity Cloud project](#creating-a-csv-from-a-unity-cloud-project) for more information.
7782
- **in a folder**: Select this option if your assets are in a folder on your local machine.
7883
* Chose the grouping strategy for the assets:
7984
- group files by name: Select this option if your assets are following a naming convention, for example, blueasset.fbx, blueasset.png.
8085
![Using the group by name convention](./documentation/group-by-name.png)
8186
- group files by folder: Select this option if your assets are organized by folder, that is, all relevant files are in distinct folders.
8287
![Using the group by folder convention](./documentation/group-by-folder.png)
8388
- one file = one asset: Select this option if no grouping is necessary. Each file in the asset folder and its subfolders is created as an asset.
84-
- Confirm if you want automatic preview detection:
85-
- If you said yes: any picture file with the suffix `_preview` will be associated to the file of the same name as a preview.
89+
- Confirm if you want automatic preview detection:
90+
- If you said yes: any picture file with the suffix `_preview` will be associated to the file of the same name as a preview.
8691

8792
### Validation step
8893

@@ -104,7 +109,7 @@ To manage and customize the upload of assets, create an upload plan as follows:
104109

105110
To create a .csv file from a Unity Cloud project, follow these steps:
106111
1. Run the CLI tool with the `--create` flag.
107-
2. When prompted to chose where the assets are located, select `in Unity Cloud`.
112+
2. When prompted to chose an action, select `Update assets' metadata`.
108113
3. Answer the next questions normally.
109114
> **Note**: The files won't be downloaded nor will they appear in the .csv file. Using the `Unity Cloud` assets source will only allow you to update assets tags and metadata.
110115
> **Note**: The collection won't appear in the csv as this is a known limitation at the moment. You can still edit this colum in the csv file to update the collection of the assets.
@@ -131,11 +136,18 @@ To use an existing configuration file, follow these steps:
131136

132137
With the `app_settings.json` file, you can fine-tune the amount of assets created and uploaded in parallel. Depending on your network, the number of assets, and the size of the assets, you can adjust the following settings:
133138
- `parallelCreationEdit`: The number of assets created and updated in parallel. This settings can be kept high as it is not resource intensive.
134-
- `parallelAssetUpload`: The number of assets that will have their files uploaded in parallel. This setting should be adjusted depending on the size of the assets and the network speed. When dealing with large files (>100MB), it is recommended to keep this setting low (1-2) to avoid time out.
139+
- `parallelAssetUpload`: The number of assets that will have their files uploaded in parallel. This setting should be adjusted depending on the size of the assets and the network speed. When dealing with large files (>100MB), it is recommended to keep this setting low (3-4) to avoid time out.
135140
- `parallelFileUploadPerAsset`: The number of files uploaded in parallel for each asset. This setting should be adjusted depending on the number of files and the network speed. It is recommended to adjust it according to `parallelAssetUpload`, as the total number of files uploaded in parallel will be `parallelAssetUpload * parallelFileUploadPerAsset`.
141+
- `httpTimeout`: The time (in seconds) before the http client triggers a timeout exception. When handling very large files (> 1GB) or when on a slow connection, it might be necessary to raise this value.
136142

137143
In the `app_settings.json` file, you can also add environment variables that will be set at runtime. This is useful when running the CLI tool in a private network environment.
138144

145+
### Use keybindings
146+
147+
When used in interactive mode, the CLI tool provides keybindings to help you navigate the tool more efficiently. The keybindings are as follows:
148+
- `Ctrl + Q`: Exit the tool.
149+
- `Ctrl + Z`: Go back to the previous question.
150+
139151
## Troubleshoot
140152

141153
Here's a list of common problems you might encounter while using the CLI Tool.

bulk_upload_cli/app_settings.json

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
{
22
"parallelCreationEdit": 20,
3-
"parallelAssetUpload": 2,
3+
"parallelAssetUpload": 5,
44
"parallelFileUploadPerAsset": 5,
5-
"environmentVariables": {}
6-
}
5+
"environmentVariables": {
6+
},
7+
"featureFlags":[],
8+
"httpTimeout": 300
9+
}

bulk_upload_cli/bulk_upload/asset_deleter.py

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,16 @@
11
from InquirerPy import inquirer
2+
from shared.utils import execute_prompt
3+
24
import unity_cloud as uc
35

46

57
def ask_for_login():
6-
login_type = inquirer.select(message="Choose authentication method?",
7-
choices=["User login", "Service account"]).execute()
8+
login_type = execute_prompt(inquirer.select(message="Choose authentication method?",
9+
choices=["User login", "Service account"]))
810

911
if login_type == "Service account":
10-
key_id = inquirer.text(message="Enter your key ID:").execute()
11-
key = inquirer.secret(message="Enter your key:").execute()
12+
key_id = execute_prompt(inquirer.text(message="Enter your key ID:"))
13+
key = execute_prompt(inquirer.secret(message="Enter your key:"))
1214

1315
return key_id, key
1416

@@ -43,21 +45,21 @@ def delete_assets_in_project():
4345
if len(organizations) == 0:
4446
print("No organizations found. Please create an organization first.")
4547
exit(1)
46-
org_selected = inquirer.select(message="Select an organization:",
47-
choices=[org.name for org in organizations]).execute()
48+
org_selected = execute_prompt(inquirer.select(message="Select an organization:",
49+
choices=[org.name for org in organizations]))
4850
org_id = [org.id for org in organizations if org.name == org_selected][0]
4951

5052
projects = uc.identity.get_project_list(org_id)
5153
if len(projects) == 0:
5254
print("No projects found. Please create a project first.")
5355
exit(1)
5456

55-
selected_project = inquirer.select(message="Select a project:",
56-
choices=[project.name for project in projects]).execute()
57+
selected_project = execute_prompt(inquirer.select(message="Select a project:",
58+
choices=[project.name for project in projects]))
5759
project_id = [project.id for project in projects if project.name == selected_project][0]
5860

5961
project_assets = uc.assets.get_asset_list(org_id, project_id)
60-
confirm = inquirer.confirm(message=f"Are you sure you want to delete {len(project_assets)} assets?").execute()
62+
confirm = execute_prompt(inquirer.confirm(message=f"Are you sure you want to delete {len(project_assets)} assets?"))
6163
if not confirm:
6264
print("Deletion canceled. Program will exit.")
6365
exit(0)

bulk_upload_cli/bulk_upload/asset_mappers.py

Lines changed: 60 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,13 @@
66
import unity_cloud as uc
77

88
from abc import ABC, abstractmethod
9-
from glob import glob
109
from pathlib import PurePath, PurePosixPath, Path
1110
from bulk_upload.models import AssetInfo, FileInfo, ProjectUploaderConfig, Strategy, Metadata
11+
from bulk_upload.file_explorers import FileExplorer
12+
13+
14+
def is_directory_path(file_path) -> bool:
15+
return len(file_path.name.split("/")[-1].split(".")) == 1
1216

1317

1418
class AssetMapper(ABC):
@@ -23,16 +27,19 @@ def clean_up(self):
2327

2428
class UnityProjectAssetMapper(AssetMapper):
2529

30+
def __init__(self, file_explorer: FileExplorer):
31+
self.file_explorer = file_explorer
32+
2633
def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
27-
files = [y for x in os.walk(config.assets_path) for y in glob(os.path.join(x[0], '*'))]
34+
files = self.file_explorer.list_files(config.assets_path)
2835
# remove files with excluded extensions
29-
files = [f for f in files if not any(f.endswith(ext) for ext in config.excluded_file_extensions)]
36+
files = [f for f in files if not any(f.suffix.endswith(ext) for ext in config.excluded_file_extensions)]
3037

3138
assets = dict()
3239
for f in files:
33-
if os.path.isdir(f):
40+
if self.is_directory_path(f):
3441
continue
35-
if f.endswith(".meta"): # meta file should not be considered as an asset alone
42+
if f.name.endswith(".meta"): # meta file should not be considered as an asset alone
3643
continue
3744

3845
file_name = os.path.basename(f)
@@ -58,21 +65,28 @@ def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
5865
return list(assets.values())
5966

6067
def clean_up(self):
61-
pass
68+
print("No clean up needed")
69+
70+
@staticmethod
71+
def is_directory_path(file_path) -> bool:
72+
return len(file_path.name.split("/")[-1].split(".")) == 1
6273

6374

6475
class NameGroupingAssetMapper(AssetMapper):
6576

77+
def __init__(self, file_explorer: FileExplorer):
78+
self.file_explorer = file_explorer
79+
6680
def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
67-
files = [y for x in os.walk(config.assets_path) for y in glob(os.path.join(x[0], '*'))]
81+
files = self.file_explorer.list_files(config.assets_path)
6882
# remove files with excluded extensions
6983
files = [f for f in files if not any(f.endswith(ext) for ext in config.excluded_file_extensions)]
7084

7185
assets = dict()
7286
for f in files:
73-
if os.path.isdir(f):
87+
if is_directory_path(f):
7488
continue
75-
if f.endswith(".meta"): # meta file should not be considered as an asset alone
89+
if f.name.endswith(".meta"): # meta file should not be considered as an asset alone
7690
continue
7791

7892
base_name = os.path.splitext(os.path.basename(f))[0]
@@ -97,16 +111,18 @@ def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
97111
return list(assets.values())
98112

99113
def clean_up(self):
100-
pass
114+
print("No clean up needed")
101115

102116

103117
class FolderGroupingAssetMapper(AssetMapper):
104118

119+
def __init__(self, file_explorer: FileExplorer):
120+
self.file_explorer = file_explorer
121+
105122
def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
106123

107124
abs_path = os.path.abspath(config.assets_path)
108-
hierarchical_level = int(config.hierarchical_level) + abs_path.count(os.sep)
109-
folders = [x[0] for x in os.walk(abs_path) if x[0].count(os.sep) == hierarchical_level]
125+
folders = self.file_explorer.get_folders_at_hierarchy_level(abs_path, config.hierarchical_level)
110126

111127
if len(folders) == 0:
112128
print(f"No folders found in the assets path. Only the root folder will be considered as an asset")
@@ -117,16 +133,16 @@ def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
117133
asset_name = PurePath(folder).name
118134
assets[asset_name] = AssetInfo(asset_name)
119135

120-
files = [get_file_info(PurePath(f), abs_path) for x in os.walk(folder) for f in glob(os.path.join(x[0], '*'))]
136+
files = self.file_explorer.list_files(folder)
121137
# remove files with excluded extensions
122-
files = [f for f in files if not any(f.path.suffix.endswith(ext) for ext in config.excluded_file_extensions)]
123-
assets[asset_name].files = files
138+
files = [f for f in files if not any(f.suffix.endswith(ext) for ext in config.excluded_file_extensions)]
139+
assets[asset_name].files = [get_file_info(f, abs_path) for f in files]
124140

125141
for asset in assets.values():
126142
# detect preview files with extension as png
127143
asset_files = asset.files.copy()
128144
for file in asset_files:
129-
if os.path.isdir(file.path):
145+
if is_directory_path(file.path):
130146
asset.files.remove(file)
131147

132148
if config.preview_detection and self.is_preview_file(file.path):
@@ -150,7 +166,7 @@ def is_preview_file(file_path) -> bool:
150166
return is_picture_file and file_name_is_preview or is_picture_file and parent_folder_is_preview
151167

152168
def clean_up(self):
153-
pass
169+
print("No clean up needed")
154170

155171

156172
class UnityPackageAssetMapper(AssetMapper):
@@ -203,6 +219,7 @@ def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
203219

204220
def clean_up(self):
205221
shutil.rmtree("tempo")
222+
print("Extracted files have been deleted")
206223

207224

208225
@staticmethod
@@ -216,45 +233,44 @@ def get_path_from_pathname_file(tar, path_file):
216233

217234
class SingleFileAssetMapper(AssetMapper):
218235

236+
def __init__(self, file_explorer: FileExplorer):
237+
self.file_explorer = file_explorer
238+
219239
def clean_up(self):
220-
pass
240+
print("No clean up needed")
221241

222242
def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
223-
absolute_path = os.path.abspath(config.assets_path)
224-
# find all files in the assets folder and sub folders
225-
files = [y for x in os.walk(config.assets_path) for y in glob(os.path.join(x[0], '*'))]
243+
files = self.file_explorer.list_files(config.assets_path)
226244
# remove files with excluded extensions
227-
files = [f for f in files if not any(f.endswith(ext) for ext in config.excluded_file_extensions)]
245+
files = [f for f in files if not any(f.suffix.endswith(ext) for ext in config.excluded_file_extensions)]
228246

229247
assets = []
230248

231249
potential_previews = {}
232250
if config.preview_detection:
233251
for file in files:
234-
if os.path.isdir(file):
252+
if self.is_directory_path(file):
235253
continue
236254

237-
file_path = PurePath(file)
238-
if self.is_preview_file(file_path):
239-
file_stem = file_path.stem.lower().replace("_preview", "")
240-
potential_previews[file_stem] = FileInfo(file_path,
241-
PurePosixPath(file_path.relative_to(config.assets_path)))
255+
if self.is_preview_file(file):
256+
file_stem = file.stem.lower().replace("_preview", "")
257+
potential_previews[file_stem] = FileInfo(file,
258+
PurePosixPath(file.relative_to(config.assets_path)))
242259

243260
for file in files:
244-
if os.path.isdir(file):
261+
if self.is_directory_path(file):
245262
continue
246263

247264
if file in potential_previews.values():
248265
continue
249266

250-
file_path = PurePath(file)
251267
asset = AssetInfo(os.path.basename(file))
252-
asset.files.append(FileInfo(file_path, PurePosixPath(file_path.relative_to(config.assets_path))))
268+
asset.files.append(FileInfo(file, PurePosixPath(file.relative_to(config.assets_path))))
253269

254-
if file_path.stem.lower() in potential_previews:
255-
preview_file = potential_previews[file_path.stem.lower()]
270+
if file.stem.lower() in potential_previews:
271+
preview_file = potential_previews[file.stem.lower()]
256272
asset.preview_files.append(preview_file)
257-
del potential_previews[file_path.stem.lower()]
273+
del potential_previews[file.stem.lower()]
258274

259275
assets.append(asset)
260276

@@ -275,6 +291,10 @@ def is_preview_file(file_path) -> bool:
275291

276292
return file_stem.endswith("_preview") and is_picture_file
277293

294+
@staticmethod
295+
def is_directory_path(file_path) -> bool:
296+
return len(file_path.name.split("/")[-1].split(".")) == 1
297+
278298

279299
class CsvAssetMapper(AssetMapper):
280300

@@ -284,6 +304,9 @@ def __init__(self):
284304
def clean_up(self):
285305
if self.sub_strategy == "unityPackage":
286306
shutil.rmtree("tempo")
307+
print("Extracted files have been deleted")
308+
else:
309+
print("No clean up needed")
287310

288311
def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
289312
assets = []
@@ -295,10 +318,9 @@ def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
295318
if len(inputs) == 2:
296319
self.sub_strategy = Strategy(inputs[0])
297320
sub_path = inputs[1]
298-
if self.sub_strategy == "unityPackage":
321+
if self.sub_strategy == Strategy.UNITY_PACKAGE:
299322
self.extract_unity_package(sub_path)
300-
elif self.sub_strategy == "cloudAsset":
301-
print("File update not supported for cloud assets, upload will be skipped", flush=True)
323+
elif self.sub_strategy == Strategy.CLOUD_ASSET:
302324
config.update_files = False
303325

304326
for row in reader:
@@ -357,7 +379,7 @@ def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
357379
return asset_infos
358380

359381
def clean_up(self):
360-
pass
382+
print("No clean up needed")
361383

362384

363385
def get_unity_id_from_meta_file(meta_file_content) -> str:

0 commit comments

Comments
 (0)