Merge pull request #10 from Unity-Technologies/release/AME.2025.04

release/AME.2025.04

Author: GuillaumeSingland

CHANGELOG.md

@@ -5,12 +5,22 @@ All notable changes to this package will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [0.4.0] - 2025-02-28
+
+### Added
+- Added keybidings to quit the application and to go back to the previous question in interactive mode.
+
+### Changed
+- Updated Python SDK dependency for bulk_upload to 0.10.6
+
 ## [0.3.0] - 2025-02-04
 
 ### Added
 - Added the column description in the csv to allow edit of individual asset descriptions.
 - Add multiple user input validations during the interactive config mode.
 - Added `app_settings.json` file to configure applications amount of parallel workers and environment variables when needed.
+- Added the capacity to retry step when there's a failure during the pipeline.
+- Added a default timeout of 5 mins for http calls to help with the uploads of bigger files.
 
 ### Changed
 - Organizations and projects are now selected via a list in the interactive config mode and delete mode.

bulk_upload_cli/README.md

@@ -20,6 +20,7 @@ Find and connect support services on the [Help & Support](https://cloud.unity.co
     - [Editing metadata in the csv file](#editing-metadata-in-the-csv-file)
     - [Use an existing configuration file](#use-an-existing-configuration-file)
     - [Fine-tune the asset creation and upload](#fine-tune-the-asset-creation-and-upload)
+    - [Use keybindings](#use-keybindings)
   - [Troubleshoot](#troubleshoot)
   - [See also](#see-also)
   - [Tell us what you think](#tell-us-what-you-think)
@@ -62,27 +63,31 @@ The bulk upload sample script is provided under the [Unity ToS license](../LICEN
    * On Mac: `python3 bulk_cli.py --create`
    * On Windows: `python bulk_cli.py --create`
 
+### Select an action
+
+Select one of the two possible actions:
+
+- **Upload local assets**: Select this option to upload assets from your local machine to the cloud. See the [Select the input method](#select-the-input-method) section for more information.
+- **Update assets' metadata**: Select this option to update the metadata of assets in the cloud. See the [Creating a csv from a Unity Cloud project](#creating-a-csv-from-a-unity-cloud-project) section for more information.
 ### Select the input method
 
-Select one of the three strategies as the input method for bulk asset creation:
+Select one of the four strategies as the input method for bulk asset creation:
 
-- **listed in a casv respecting the CLI tool template**: Select this option if you built a CSV listing your assets location and details using the provided template.
+- **listed in a csv respecting the CLI tool template**: Select this option if you built a CSV listing your assets location and details using the provided template.
   * Provide the path to the csv file.
 - **in a .unitypackage file**: Select this option if your assets are in a .unitypackage file. The tool extracts the assets from the .unitypackage file and uploads them to the cloud.
   * Provide the path to the .unitypackage file.
 - **in a local unity project**: Select this option if your assets are in a local Unity project.
   * Provide the path to the asset folder of the Unity project.
-- **in UnityCloud**: Select this option if your assets are in a Unity Cloud project.
-  * Provide the organization ID and project ID. As of now, this features is solely to update tags and metadata of assets in the cloud. See section [Creating a csv from a Unity Cloud project](#creating-a-csv-from-a-unity-cloud-project) for more information.
 - **in a folder**: Select this option if your assets are in a folder on your local machine.
   * Chose the grouping strategy for the assets:
     - group files by name: Select this option if your assets are following a naming convention, for example, blueasset.fbx, blueasset.png.
 ![Using the group by name convention](./documentation/group-by-name.png)
     - group files by folder: Select this option if your assets are organized by folder, that is, all relevant files are in distinct folders.
 ![Using the group by folder convention](./documentation/group-by-folder.png)
     - one file = one asset: Select this option if no grouping is necessary. Each file in the asset folder and its subfolders is created as an asset.
-      - Confirm if you want automatic preview detection:
-        - If you said yes: any picture file with the suffix `_preview` will be associated to the file of the same name as a preview.
+    - Confirm if you want automatic preview detection:
+      - If you said yes: any picture file with the suffix `_preview` will be associated to the file of the same name as a preview.
 
 ### Validation step
 
@@ -104,7 +109,7 @@ To manage and customize the upload of assets, create an upload plan as follows:
 
 To create a .csv file from a Unity Cloud project, follow these steps:
 1. Run the CLI tool with the `--create` flag.
-2. When prompted to chose where the assets are located, select `in Unity Cloud`.
+2. When prompted to chose an action, select `Update assets' metadata`.
 3. Answer the next questions normally.
 >  **Note**: The files won't be downloaded nor will they appear in the .csv file. Using the `Unity Cloud` assets source will only allow you to update assets tags and metadata.
 >  **Note**: The collection won't appear in the csv as this is a known limitation at the moment. You can still edit this colum in the csv file to update the collection of the assets.
@@ -131,11 +136,18 @@ To use an existing configuration file, follow these steps:
 
 With the `app_settings.json` file, you can fine-tune the amount of assets created and uploaded in parallel. Depending on your network, the number of assets, and the size of the assets, you can adjust the following settings:
 - `parallelCreationEdit`: The number of assets created and updated in parallel. This settings can be kept high as it is not resource intensive.
-- `parallelAssetUpload`: The number of assets that will have their files uploaded in parallel. This setting should be adjusted depending on the size of the assets and the network speed. When dealing with large files (>100MB), it is recommended to keep this setting low (1-2) to avoid time out.
+- `parallelAssetUpload`: The number of assets that will have their files uploaded in parallel. This setting should be adjusted depending on the size of the assets and the network speed. When dealing with large files (>100MB), it is recommended to keep this setting low (3-4) to avoid time out.
 - `parallelFileUploadPerAsset`: The number of files uploaded in parallel for each asset. This setting should be adjusted depending on the number of files and the network speed. It is recommended to adjust it according to `parallelAssetUpload`, as the total number of files uploaded in parallel will be `parallelAssetUpload * parallelFileUploadPerAsset`.
+- `httpTimeout`: The time (in seconds) before the http client triggers a timeout exception. When handling very large files (> 1GB) or when on a slow connection, it might be necessary to raise this value.
 
 In the `app_settings.json` file, you can also add environment variables that will be set at runtime. This is useful when running the CLI tool in a private network environment.
 
+### Use keybindings
+
+When used in interactive mode, the CLI tool provides keybindings to help you navigate the tool more efficiently. The keybindings are as follows:
+- `Ctrl + Q`: Exit the tool.
+- `Ctrl + Z`: Go back to the previous question.
+
 ## Troubleshoot
 
 Here's a list of common problems you might encounter while using the CLI Tool.

bulk_upload_cli/app_settings.json

@@ -1,6 +1,9 @@
 {
     "parallelCreationEdit": 20,
-    "parallelAssetUpload": 2,
+    "parallelAssetUpload": 5,
     "parallelFileUploadPerAsset": 5,
-    "environmentVariables": {}
-}
+    "environmentVariables": {
+	},
+	"featureFlags":[],
+	"httpTimeout": 300
+}

bulk_upload_cli/bulk_upload/asset_deleter.py

@@ -1,14 +1,16 @@
 from InquirerPy import inquirer
+from shared.utils import execute_prompt
+
 import unity_cloud as uc
 
 
 def ask_for_login():
-    login_type = inquirer.select(message="Choose authentication method?",
-                                 choices=["User login", "Service account"]).execute()
+    login_type = execute_prompt(inquirer.select(message="Choose authentication method?",
+                                 choices=["User login", "Service account"]))
 
     if login_type == "Service account":
-        key_id = inquirer.text(message="Enter your key ID:").execute()
-        key = inquirer.secret(message="Enter your key:").execute()
+        key_id = execute_prompt(inquirer.text(message="Enter your key ID:"))
+        key = execute_prompt(inquirer.secret(message="Enter your key:"))
 
         return key_id, key
 
@@ -43,21 +45,21 @@ def delete_assets_in_project():
     if len(organizations) == 0:
         print("No organizations found. Please create an organization first.")
         exit(1)
-    org_selected = inquirer.select(message="Select an organization:",
-                                   choices=[org.name for org in organizations]).execute()
+    org_selected = execute_prompt(inquirer.select(message="Select an organization:",
+                                   choices=[org.name for org in organizations]))
     org_id = [org.id for org in organizations if org.name == org_selected][0]
 
     projects = uc.identity.get_project_list(org_id)
     if len(projects) == 0:
         print("No projects found. Please create a project first.")
         exit(1)
 
-    selected_project = inquirer.select(message="Select a project:",
-                                       choices=[project.name for project in projects]).execute()
+    selected_project = execute_prompt(inquirer.select(message="Select a project:",
+                                       choices=[project.name for project in projects]))
     project_id = [project.id for project in projects if project.name == selected_project][0]
 
     project_assets = uc.assets.get_asset_list(org_id, project_id)
-    confirm = inquirer.confirm(message=f"Are you sure you want to delete {len(project_assets)} assets?").execute()
+    confirm = execute_prompt(inquirer.confirm(message=f"Are you sure you want to delete {len(project_assets)} assets?"))
     if not confirm:
         print("Deletion canceled. Program will exit.")
         exit(0)

bulk_upload_cli/bulk_upload/asset_mappers.py

@@ -6,9 +6,13 @@
 import unity_cloud as uc
 
 from abc import ABC, abstractmethod
-from glob import glob
 from pathlib import PurePath, PurePosixPath, Path
 from bulk_upload.models import AssetInfo, FileInfo, ProjectUploaderConfig, Strategy, Metadata
+from bulk_upload.file_explorers import FileExplorer
+
+
+def is_directory_path(file_path) -> bool:
+    return len(file_path.name.split("/")[-1].split(".")) == 1
 
 
 class AssetMapper(ABC):
@@ -23,16 +27,19 @@ def clean_up(self):
 
 class UnityProjectAssetMapper(AssetMapper):
 
+    def __init__(self, file_explorer: FileExplorer):
+        self.file_explorer = file_explorer
+
     def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
-        files = [y for x in os.walk(config.assets_path) for y in glob(os.path.join(x[0], '*'))]
+        files = self.file_explorer.list_files(config.assets_path)
         # remove files with excluded extensions
-        files = [f for f in files if not any(f.endswith(ext) for ext in config.excluded_file_extensions)]
+        files = [f for f in files if not any(f.suffix.endswith(ext) for ext in config.excluded_file_extensions)]
 
         assets = dict()
         for f in files:
-            if os.path.isdir(f):
+            if self.is_directory_path(f):
                 continue
-            if f.endswith(".meta"):  # meta file should not be considered as an asset alone
+            if f.name.endswith(".meta"):  # meta file should not be considered as an asset alone
                 continue
 
             file_name = os.path.basename(f)
@@ -58,21 +65,28 @@ def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
         return list(assets.values())
 
     def clean_up(self):
-        pass
+        print("No clean up needed")
+
+    @staticmethod
+    def is_directory_path(file_path) -> bool:
+        return len(file_path.name.split("/")[-1].split(".")) == 1
 
 
 class NameGroupingAssetMapper(AssetMapper):
 
+    def __init__(self, file_explorer: FileExplorer):
+        self.file_explorer = file_explorer
+
     def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
-        files = [y for x in os.walk(config.assets_path) for y in glob(os.path.join(x[0], '*'))]
+        files = self.file_explorer.list_files(config.assets_path)
         # remove files with excluded extensions
         files = [f for f in files if not any(f.endswith(ext) for ext in config.excluded_file_extensions)]
 
         assets = dict()
         for f in files:
-            if os.path.isdir(f):
+            if is_directory_path(f):
                 continue
-            if f.endswith(".meta"):  # meta file should not be considered as an asset alone
+            if f.name.endswith(".meta"):  # meta file should not be considered as an asset alone
                 continue
 
             base_name = os.path.splitext(os.path.basename(f))[0]
@@ -97,16 +111,18 @@ def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
         return list(assets.values())
 
     def clean_up(self):
-        pass
+        print("No clean up needed")
 
 
 class FolderGroupingAssetMapper(AssetMapper):
 
+    def __init__(self, file_explorer: FileExplorer):
+        self.file_explorer = file_explorer
+
     def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
 
         abs_path = os.path.abspath(config.assets_path)
-        hierarchical_level = int(config.hierarchical_level) + abs_path.count(os.sep)
-        folders = [x[0] for x in os.walk(abs_path) if x[0].count(os.sep) == hierarchical_level]
+        folders = self.file_explorer.get_folders_at_hierarchy_level(abs_path, config.hierarchical_level)
 
         if len(folders) == 0:
             print(f"No folders found in the assets path. Only the root folder will be considered as an asset")
@@ -117,16 +133,16 @@ def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
             asset_name = PurePath(folder).name
             assets[asset_name] = AssetInfo(asset_name)
 
-            files = [get_file_info(PurePath(f), abs_path) for x in os.walk(folder) for f in glob(os.path.join(x[0], '*'))]
+            files = self.file_explorer.list_files(folder)
             # remove files with excluded extensions
-            files = [f for f in files if not any(f.path.suffix.endswith(ext) for ext in config.excluded_file_extensions)]
-            assets[asset_name].files = files
+            files = [f for f in files if not any(f.suffix.endswith(ext) for ext in config.excluded_file_extensions)]
+            assets[asset_name].files = [get_file_info(f, abs_path) for f in files]
 
         for asset in assets.values():
             # detect preview files with extension as png
             asset_files = asset.files.copy()
             for file in asset_files:
-                if os.path.isdir(file.path):
+                if is_directory_path(file.path):
                     asset.files.remove(file)
 
                 if config.preview_detection and self.is_preview_file(file.path):
@@ -150,7 +166,7 @@ def is_preview_file(file_path) -> bool:
         return is_picture_file and file_name_is_preview or is_picture_file and parent_folder_is_preview
 
     def clean_up(self):
-        pass
+        print("No clean up needed")
 
 
 class UnityPackageAssetMapper(AssetMapper):
@@ -203,6 +219,7 @@ def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
 
     def clean_up(self):
         shutil.rmtree("tempo")
+        print("Extracted files have been deleted")
 
 
     @staticmethod
@@ -216,45 +233,44 @@ def get_path_from_pathname_file(tar, path_file):
 
 class SingleFileAssetMapper(AssetMapper):
 
+    def __init__(self, file_explorer: FileExplorer):
+        self.file_explorer = file_explorer
+
     def clean_up(self):
-        pass
+        print("No clean up needed")
 
     def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
-        absolute_path = os.path.abspath(config.assets_path)
-        # find all files in the assets folder and sub folders
-        files = [y for x in os.walk(config.assets_path) for y in glob(os.path.join(x[0], '*'))]
+        files = self.file_explorer.list_files(config.assets_path)
         # remove files with excluded extensions
-        files = [f for f in files if not any(f.endswith(ext) for ext in config.excluded_file_extensions)]
+        files = [f for f in files if not any(f.suffix.endswith(ext) for ext in config.excluded_file_extensions)]
 
         assets = []
 
         potential_previews = {}
         if config.preview_detection:
             for file in files:
-                if os.path.isdir(file):
+                if self.is_directory_path(file):
                     continue
 
-                file_path = PurePath(file)
-                if self.is_preview_file(file_path):
-                    file_stem = file_path.stem.lower().replace("_preview", "")
-                    potential_previews[file_stem] = FileInfo(file_path,
-                                                             PurePosixPath(file_path.relative_to(config.assets_path)))
+                if self.is_preview_file(file):
+                    file_stem = file.stem.lower().replace("_preview", "")
+                    potential_previews[file_stem] = FileInfo(file,
+                                                             PurePosixPath(file.relative_to(config.assets_path)))
 
         for file in files:
-            if os.path.isdir(file):
+            if self.is_directory_path(file):
                 continue
 
             if file in potential_previews.values():
                 continue
 
-            file_path = PurePath(file)
             asset = AssetInfo(os.path.basename(file))
-            asset.files.append(FileInfo(file_path, PurePosixPath(file_path.relative_to(config.assets_path))))
+            asset.files.append(FileInfo(file, PurePosixPath(file.relative_to(config.assets_path))))
 
-            if file_path.stem.lower() in potential_previews:
-                preview_file = potential_previews[file_path.stem.lower()]
+            if file.stem.lower() in potential_previews:
+                preview_file = potential_previews[file.stem.lower()]
                 asset.preview_files.append(preview_file)
-                del potential_previews[file_path.stem.lower()]
+                del potential_previews[file.stem.lower()]
 
             assets.append(asset)
 
@@ -275,6 +291,10 @@ def is_preview_file(file_path) -> bool:
 
         return file_stem.endswith("_preview") and is_picture_file
 
+    @staticmethod
+    def is_directory_path(file_path) -> bool:
+        return len(file_path.name.split("/")[-1].split(".")) == 1
+
 
 class CsvAssetMapper(AssetMapper):
 
@@ -284,6 +304,9 @@ def __init__(self):
     def clean_up(self):
         if self.sub_strategy == "unityPackage":
             shutil.rmtree("tempo")
+            print("Extracted files have been deleted")
+        else:
+            print("No clean up needed")
 
     def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
         assets = []
@@ -295,10 +318,9 @@ def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
             if len(inputs) == 2:
                 self.sub_strategy = Strategy(inputs[0])
                 sub_path = inputs[1]
-                if self.sub_strategy == "unityPackage":
+                if self.sub_strategy == Strategy.UNITY_PACKAGE:
                     self.extract_unity_package(sub_path)
-                elif self.sub_strategy == "cloudAsset":
-                    print("File update not supported for cloud assets, upload will be skipped", flush=True)
+                elif self.sub_strategy == Strategy.CLOUD_ASSET:
                     config.update_files = False
 
             for row in reader:
@@ -357,7 +379,7 @@ def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
         return asset_infos
 
     def clean_up(self):
-        pass
+        print("No clean up needed")
 
 
 def get_unity_id_from_meta_file(meta_file_content) -> str: