Feat(image): Image upload controller + service #175
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…e and deprecate UploadFlowService - Updated ImageController to return detailed UploadFileResponse including image URL. - Refactored UploadClient to directly return UploadFileResponse. - Introduced UploadFileResponse interface in common types for consistency. - Deprecated UploadFlowService methods with appropriate error messages. - Updated ImageService to support new image formats and maintain original format option. - Improved documentation and comments for clarity.
…ageService and streamline configuration - Removed unnecessary environment variables from docker-compose for image migration. - Updated migrate-images command to eliminate direct MinIO configuration parameters. - Refactored migration logic to use ImageService for database connection and image uploads. - Introduced new methods for processing callout images and improved error handling.
- Introduced helper functions `isImageMigrated` and `extractImageInfo` to streamline image URL processing. - Added `processOptionImages` and `processOptionImage` functions to handle image migration for options settings. - Improved error handling and logging for image migration operations. - Removed deprecated `formatSkipMessage` function from utils. - Updated migration statistics tracking to provide detailed insights during the migration process.
…ntication in ImageController - Deleted UploadFlowService and related DTOs to streamline the image upload process. - Updated ImageController to require authentication for uploads, enhancing security. - Removed references to UploadFlow in common types and core models. - Improved code clarity by eliminating unused and deprecated components.
…Controller - Removed unused MINIO_S3FS_BUCKET environment variable from .env.example and docker-compose.yml. - Deleted the debugConnection method from ImageController to streamline the API. - Added new S3 utility module for improved image handling. - Updated ImageService to throw a more descriptive error message for invalid uploads.
- Added DocumentController for handling document uploads, retrieval, and deletion. - Introduced DocumentService for managing document storage in S3/MinIO. - Created UploadDocumentClient for client-side document uploads. - Updated common types to include support for document metadata. - Refactored upload middleware to handle file uploads more generically. - Enhanced error handling for document operations and improved response structures. - Removed unused S3 utility functions and consolidated file handling logic.
…tion and ownership checks - Updated DocumentController and ImageController to include file type validation using common utility functions. - Added ownership checks for document and image deletion to ensure only owners or admins can delete resources. - Refactored upload middleware to utilize a constant for maximum file size. - Introduced new constants for allowed file types and sizes in the common package. - Enhanced DocumentService and ImageService to handle owner metadata during uploads and retrievals. - Removed unused constants from the client package to streamline the codebase.
- Renamed image migration commands and types to reflect the new uploads migration functionality. - Removed deprecated image migration code and replaced it with uploads migration logic. - Updated environment variables and Docker configurations to support uploads migration. - Enhanced the backend CLI to include the new migrate-uploads command for migrating uploads to MinIO. - Improved documentation and comments for clarity on the new migration process.
- Added functionality to process document uploads from callout responses during the uploads migration. - Introduced new helper functions for handling document uploads and checking file types. - Updated migration statistics to track document upload success, errors, and skipped counts. - Improved logging and error handling for document processing within the migration script. - Marked existing methods in CalloutsService as deprecated, as they are only used for the uploads migration.
21 tasks
- Removed the '/uploads' route from the Nginx configuration to streamline image handling. - Updated the Vite configuration to exclude the '/uploads' path from the proxy settings, focusing on API and login routes. - Refactored the ImageService to simplify the constructor and ensure available widths are sorted correctly. - Adjusted the default image service configuration to include quality and format settings from environment variables.
JumpLink
commented
Apr 22, 2025
JumpLink
commented
Apr 22, 2025
This comment was marked as outdated.
This comment was marked as outdated.
- Updated CalloutController to use the new calloutsService instance instead of the default import. - Introduced a new debug endpoint in CalloutController to list callout responses with file uploads. - Enhanced error handling in CalloutsService to return false for invalid slide answers and component answers. - Refactored BeabeeStorage to simplify URL handling for uploaded files.
- Updated DocumentController and ImageController to include the hash of uploaded files in the response. - Refactored migration logic in backend-cli to utilize the new FormioFile type for better type safety. - Introduced FormioFile interface in common types to standardize file upload structure. - Enhanced DocumentService and ImageService to retrieve and return the hash (ETag) of uploaded documents and images. - Updated frontend TypeScript configuration to include the new FormioFile type for improved type checking. - Improved logging and error handling in the upload process to provide better insights during file operations.
- Deleted the debug endpoint from CalloutController to streamline API functionality. - Removed commented-out code for original filename extraction in migrate-uploads action to clean up the codebase. - Eliminated unnecessary console logging in BeabeeStorage to enhance performance and reduce noise in logs. - Cleaned up logging statements in CalloutsService to improve clarity and maintainability.
…y and structure - Enhanced the `.env.example` files for both core and frontend applications by adding detailed comments and organization for better readability. - Introduced new environment variables for third-party integrations, email settings, and localization configurations. - Updated the `env.ts` files to reflect new environment variables and provide default values for test configurations. - Improved the `config.ts` in the core package to include comprehensive documentation for each configuration option, ensuring better understanding and maintainability.
…figurations - Added new environment variables for image processing configuration in `.env.example`, including `BEABEE_IMAGE_QUALITY` and `BEABEE_IMAGE_FORMAT`. - Updated `DocumentController` and `ImageController` to return the path of uploaded files in the response. - Refactored upload logic in `ImageService` and `DocumentService` to utilize a new path property for uploaded files. - Improved file type validation in the upload process to support additional image formats. - Updated frontend components to handle new upload response structure, including path handling for images. - Removed the deprecated S3 utility functions as part of the cleanup process. - Enhanced logging and error handling for better insights during file operations.
- Added functionality to migrate general content background images during uploads migration in the backend CLI. - Introduced new utility functions for processing image URLs and handling absolute URLs in the frontend. - Updated AppLogo and CalloutSummary components to dynamically construct image URLs using the API base URL. - Added a default "no image" placeholder for missing images in CalloutSummary. - Refactored AppImageUpload component to improve image URL processing and ensure correct URL emission. - Included new URL utility functions in the common package for better URL handling across the application.
- Updated the AppImageUpload component to streamline image URL processing by removing unnecessary debug logging and simplifying the URL handling logic. - Introduced a new utility function in generalContent for consistent image URL processing, ensuring proper handling of absolute URLs and default background images. - Enhanced the background style computation to utilize the new image URL processing function for improved reliability in displaying background images.
wpf500
requested changes
Apr 30, 2025
There was a problem hiding this comment.
Looks good so far. I don't fully understand everything and would like to do a bit more testing, but in general it looks good and it will be very nice to remove pictshare.
I have a few concerns to discuss:
- Support SVG upload comes with security implications. How are we guarding against those?
- Is upload rate limiting still in place? The old service had (different) rate limits for guests and users, this is important (especially for guests).
- Serving files through
/api/1.0. It's probably fine for now but really static assets shouldn't be served by an API, they are much more cacheable and could be served by a different route that is more optimised for content delivery. Dynamic image generation does require some sort of middle man service I guess, but I would aim for this to be on a different thread to the API service in the medium term
apps/backend/src/api/interceptors/ValidateResponseInterceptor.ts
Outdated
Show resolved
Hide resolved
The SVG file is cleaned up with SVGO which removes tags like script / embed / object and stuff like event handlers from the SVG. we only store this cleaned up svg file. But it would certainly make sense to test the security stuff again.
No, not yet. I will implement this
Good point too, although I use the header |
- Refactored Minio environment variable names in `.env.example`, `docker-compose.yml`, and `docker-compose.test.yml` to use the `BEABEE_` prefix for clarity and consistency. - Updated the `DocumentService` and `ImageService` to utilize the new environment variable names for MinIO configuration. - Enhanced the `DocumentController` to improve security by changing the `X-Frame-Options` header to `SAMEORIGIN`. - Introduced a new utility function for resolving image URLs, ensuring consistent handling of relative and absolute URLs across the frontend components. - Refactored image URL processing in `AppImageUpload`, `CalloutSummary`, and other components to utilize the new utility function for improved reliability.
…ion and error handling in upload services - Added new environment variables for MinIO admin and console ports in `.env.example` and `.env.test`. - Updated `docker-compose.yml` and `docker-compose.test.yml` to utilize the new MinIO port variables for better configurability. - Improved error handling in `DocumentService` and `ImageService` to provide more informative error messages during upload failures. - Enhanced `DocumentController` and `ImageController` to include the filename in the upload response when available. - Introduced new test cases for the upload API in the client package, covering various file types and error scenarios.
* feat(rate-limiting): Implement rate limiting for upload endpoints - Added `rate-limiter-flexible` package to manage rate limiting for uploads in the backend. - Introduced `RateLimit` decorator to enforce limits on upload requests in `DocumentController` and `ImageController`. - Created a new `TooManyRequestsError` class for handling HTTP 429 responses. - Added `RateLimitOptions` type for configuring rate limits for guests and authenticated users. - Updated `ValidateResponseInterceptor` to include detailed documentation for response validation. - Implemented tests for upload functionality to handle rate limit errors gracefully. * test(rate-limiting): Update rate limit error handling and add tests - Changed error code from "RATE_LIMIT_EXCEEDED" to "TOO_MANY_REQUESTS" in the upload document and image client classes for consistency with the backend. - Enhanced the AppImageUpload component to reflect the updated error code. - Added new environment variables for rate limit test users in `.env.test` and `env.ts`. - Created a new test suite for rate limit enforcement in the upload API, ensuring proper handling of rate limits for both guest and authenticated users. - Updated the docker-compose setup to create a test user specifically for rate limit testing.
…tialization - Added a new MinIO service in the Docker setup, utilizing a custom Dockerfile for enhanced functionality. - Implemented an initialization script to automatically create and configure buckets at startup. - Updated docker-compose files to reflect changes in MinIO service configuration and environment variables. - Removed the minio_client service as its functionality is now integrated into the MinIO service. - Enhanced the deployment workflow to include building and pushing the new MinIO image. - Added documentation for the new MinIO service, detailing its features and environment variables.
…oad handling - Added a new `UnsupportedFileType` error class for handling unsupported file type responses with HTTP 415 status. - Updated `DocumentController` and `ImageController` to throw `UnsupportedFileType` error when an unsupported file type is detected. - Enhanced the `AppImageUpload` component to display a specific error message for unsupported file types. - Updated localization files to include messages for unsupported file type errors in multiple languages.
- Changed the email, first name, last name, and password for the rate limit test user in the `.env.test` file to ensure accurate testing.
- Updated `docker-compose.yml` to include a volume for temporary storage of old uploads. - Modified `nginx.conf` to implement fallback routes for accessing old uploads, handling both single and deeper paths.
- Updated `docker-compose.yml` to restore the volume for temporary storage of old uploads. - Modified `nginx.conf` to refine the fallback route for accessing old image uploads, ensuring compatibility with nested directory structures.
|
@wpf500 I'm now finished and have implemented everything that was mentioned in the review. All that's left to do is adjust the deploy script and the production docker compose stack, but that will be separate pull requests anyway. |
- Added a `steps` option to the `migrateUploads` command, allowing users to specify which migration steps to execute. - Updated the `migrateUploads` function to conditionally process migration steps based on the provided options. - Enhanced the `MigrateUploadsArgs` and `MigrateUploadsOptions` interfaces to include the new `steps` property with detailed documentation.
… process - Enhanced the `extractImageInfo` and `extractDocumentInfo` functions to support additional URL formats, allowing for better extraction of image and document keys. - Updated the `processDocumentUpload` function to differentiate between image and document uploads, utilizing the appropriate services for each type. - Improved error handling and logging for upload operations, ensuring better feedback during the migration process. - Modified `nginx.conf` to include a check for size directory patterns in upload URLs, improving compatibility with existing upload structures.
- Refactored the CalloutCard component to compute the image URL using a new utility function, `resolveImageUrl`, ensuring proper handling of image paths. - Added a fallback image for cases where the callout does not have an associated image, improving user experience. - Cleaned up the template by removing unnecessary conditional rendering for the image element.
…onfiguration - Updated the nginx configuration to enhance fallback routes for old image uploads. - Added a specific route for handling size-prefixed upload paths, improving compatibility with existing upload structures. - Simplified the fallback logic for regular old image uploads, ensuring better handling of nested directory structures.
…iguration - Updated the nginx configuration to correct the fallback routes for size-prefixed upload paths, ensuring proper directory structure handling. - Adjusted the fallback logic for regular old image uploads to improve compatibility with existing upload structures.
- Updated the `resolveImageUrl` utility function to accept an optional width parameter, allowing for dynamic image resizing. - Modified the `CalloutCard` and `CalloutSummary` components to utilize the new width parameter when resolving image URLs, improving image quality and loading performance. - Added fallback images for cases where the callout does not have an associated image, enhancing user experience.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a comprehensive image and document handling service using MinIO for storage while keeping all access controlled through our API.
Solutions and advantages
Changes
pathattribute for files in addition tourland markedurlas deprecated. This should make future changes to file URLs easier; for example, the host domain can now be changed without any problems.TODO
Testing
ContentGeneral)docker-compose.ymlon hive-deploy-stack (PR ready for review)docker-compose.ymlon beabee-deploy (PR ready for review)Breaking changes
Usage Examples
Upload an image:
Upload a document (PDF):
Retrieve an image (original size):
Retrieve a resized image:
Retrieve a document:
This implementation provides a solid foundation for all our image and document handling needs while keeping security and performance at the forefront.