Enhance Organization Mirroring with Improved Reliability and New Options #34
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…thods, and new ONLY_MIRROR_ORGS option
|
@jaedle #!/bin/bash
# Docker test commands for mirror-to-gitea
# Replace these variables with your actual values
GITHUB_USERNAME="your-github-username"
GITEA_URL="http://your-gitea-instance:3000/"
GITEA_TOKEN="your-gitea-token"
GITHUB_TOKEN="your-github-token"
# Basic test with dry run mode
echo "Running basic test with dry run..."
docker run -it --rm \
-e GITHUB_USERNAME="$GITHUB_USERNAME" \
-e GITEA_URL="$GITEA_URL" \
-e GITEA_TOKEN="$GITEA_TOKEN" \
-e GITHUB_TOKEN="$GITHUB_TOKEN" \
-e MIRROR_ORGANIZATIONS=true \
-e DRY_RUN=true \
arunavo4/mirror-to-gitea:latest
# Test organization mirroring only (no personal repos)
echo "Running organization mirroring test..."
docker run -it --rm \
-e GITHUB_USERNAME="$GITHUB_USERNAME" \
-e GITEA_URL="$GITEA_URL" \
-e GITEA_TOKEN="$GITEA_TOKEN" \
-e GITHUB_TOKEN="$GITHUB_TOKEN" \
-e MIRROR_ORGANIZATIONS=true \
-e ONLY_MIRROR_ORGS=true \
-e PRESERVE_ORG_STRUCTURE=true \
-e DRY_RUN=true \
arunavo4/mirror-to-gitea:latest
# Test with specific organizations included
echo "Running test with specific organizations..."
docker run -it --rm \
-e GITHUB_USERNAME="$GITHUB_USERNAME" \
-e GITEA_URL="$GITEA_URL" \
-e GITEA_TOKEN="$GITEA_TOKEN" \
-e GITHUB_TOKEN="$GITHUB_TOKEN" \
-e MIRROR_ORGANIZATIONS=true \
-e INCLUDE_ORGS="org1,org2" \
-e PRESERVE_ORG_STRUCTURE=true \
-e DRY_RUN=true \
arunavo4/mirror-to-gitea:latest
# Uncomment to run with actual mirroring (no dry run)
# echo "Running actual mirroring..."
# docker run -it --rm \
# -e GITHUB_USERNAME="$GITHUB_USERNAME" \
# -e GITEA_URL="$GITEA_URL" \
# -e GITEA_TOKEN="$GITEA_TOKEN" \
# -e GITHUB_TOKEN="$GITHUB_TOKEN" \
# -e MIRROR_ORGANIZATIONS=true \
# -e ONLY_MIRROR_ORGS=true \
# -e PRESERVE_ORG_STRUCTURE=true \
# -e DRY_RUN=false \
# arunavo4/mirror-to-gitea:latest
# Uncomment to run as a background service
# echo "Running as background service..."
# docker run -d --name mirror-to-gitea \
# -e GITHUB_USERNAME="$GITHUB_USERNAME" \
# -e GITEA_URL="$GITEA_URL" \
# -e GITEA_TOKEN="$GITEA_TOKEN" \
# -e GITHUB_TOKEN="$GITHUB_TOKEN" \
# -e MIRROR_ORGANIZATIONS=true \
# -e ONLY_MIRROR_ORGS=true \
# -e PRESERVE_ORG_STRUCTURE=true \
# -e DRY_RUN=false \
# arunavo4/mirror-to-gitea:latest |
|
@arunavo4 |
@CrazyWolf13 You are right, I have updated and fixed it. |
This commit adds two major improvements: 1. Add public organization mirroring feature: - New MIRROR_PUBLIC_ORGS option to enable mirroring of public organizations - New PUBLIC_ORGS option to specify which public organizations to mirror - Separate handling for public organizations vs. member organizations - Updated documentation with examples 2. Remove hardcoded organization names: - Removed hardcoded organization names from the codebase - Added proper handling for empty organization lists - Made organization name matching case-insensitive - Updated debug script to use environment variables These changes allow users to mirror public repositories from any GitHub organization (even if they're not a member) and fix issues with case sensitivity in organization names.
This commit adds tests for the new configuration options: - Add tests for MIRROR_PUBLIC_ORGS flag - Add tests for PUBLIC_ORGS list parsing - Add test for GitHub token requirement with public organization mirroring - Add skipped test placeholder for public organization repository fetching The tests ensure that the new configuration options are properly parsed and validated, maintaining code quality and test coverage for the new feature.
This commit fixes an issue where the early return in the organization handling code was preventing the rest of the function from executing, which could cause single repository migrations to be skipped. The fix: 1. Removes the early return when no organizations are specified in INCLUDE_ORGS 2. Properly handles the empty organization list case 3. Fixes indentation and code structure for better readability This ensures that single repository migrations and other functionality continue to work even when no organizations are specified.
This commit adds several improvements to help diagnose and fix issues with organization repository fetching: 1. Add multiple fallback mechanisms for fetching organization repositories: - Try additional API endpoints when standard methods fail - Add more robust error handling and retry logic 2. Add detailed logging throughout the process: - Log repository counts and details - Log organization access information - Show detailed information about repositories being mirrored 3. Improve user feedback: - Provide clearer error messages - Show more information about what's happening during the process These changes should help diagnose and fix issues where organizations are detected but their repositories aren't being mirrored properly.
This commit fixes a syntax error in the src/index.mjs file that was causing the Docker build to fail. The issue was related to mismatched try-catch blocks that resulted in an "Unexpected catch" error during the build process. The fix: 1. Restructured the try-catch blocks to ensure proper nesting 2. Fixed indentation issues for better code readability 3. Ensured all try blocks have matching catch blocks This ensures the Docker build process completes successfully for both amd64 and arm64 architectures.
This commit fixes an issue where the tool would detect organizations but not fetch or mirror their repositories. The problem was that the code would return early when no organizations were specified in INCLUDE_ORGS, even if organizations were found through the public endpoint. The fix: 1. Refactored the organization repository fetching logic into a separate function 2. Removed the early return when no organizations are specified in INCLUDE_ORGS 3. Added logic to use organizations found through the public endpoint This ensures that repositories from detected organizations (like community-scripts) are properly fetched and mirrored, even when not explicitly specified in INCLUDE_ORGS.
This commit fixes an issue where the tool would detect organizations through the public endpoint but not fetch or mirror their repositories. The problem was that the detected organizations weren't being added to the includeOrgs list for repository fetching. The fix: 1. Add code to add detected organizations to the includeOrgs list 2. Log the updated includeOrgs list for debugging 3. Ensure repositories from detected organizations are fetched and mirrored This ensures that repositories from detected organizations (like community-scripts) are properly fetched and mirrored, even when not explicitly specified in INCLUDE_ORGS.
This commit adds a fallback mechanism to try some default organizations (community-scripts, Proxmox) when no organizations are specified in INCLUDE_ORGS and none are found through the standard endpoints. The fix: 1. Add a list of default organizations to try when no organizations are found 2. Try to fetch each default organization and add it to the list if found 3. Process the default organizations if any are found This restores the previous behavior of trying some common organizations while still allowing the user to override it by specifying organizations in INCLUDE_ORGS.
… security scanning
…ild, and security scanning" This reverts commit da65c95.
CrazyWolf13
reviewed
Apr 17, 2025
| #!/bin/bash | ||
| # Script to build and push multi-architecture Docker images for mirror-to-gitea | ||
|
|
||
| DOCKER_USERNAME="arunavo4" |
There was a problem hiding this comment.
This probably needs changing for this repo :)
There was a problem hiding this comment.
Yea missed it thanks, will update it with other stuff
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes:
This PR enhances the organization mirroring functionality in mirror-to-gitea with improved reliability and new options. It addresses issues with fetching organizations and repositories, particularly when dealing with organizations that have specific token policies.
Key Changes
Added New Option:
ONLY_MIRROR_ORGS: When set totrue, only repositories from organizations will be mirrored, skipping personal repositories. This provides better control over what gets mirrored.Improved Organization Fetching:
Enhanced Repository Fetching:
Better Error Handling:
Updated Documentation:
ONLY_MIRROR_ORGSoptionAdded Testing Script:
test-org-mirror.shscript for testing organization mirroring specificallyTesting
The changes have been tested with various GitHub organizations, including those with token lifetime restrictions. The code now properly handles these cases and provides helpful error messages to guide users in resolving issues.
To test these changes:
test-org-mirror.shscript to test organization mirroring specifically