Enhance Organization Mirroring with Improved Reliability and New Options

README.md

@@ -27,6 +27,7 @@ Additionally, you can now mirror:
 - Repositories from organizations you belong to
   - Filter which organizations to include or exclude
   - Maintain original organization structure in Gitea
+- Public repositories from any GitHub organization (even if you're not a member)
 - A single repository instead of all repositories
 - Repositories to a specific Gitea organization
 
@@ -48,14 +49,17 @@ All configuration is performed through environment variables. Flags are consider
 | GITHUB_USERNAME             | yes      | string | -       | The name of the GitHub user or organisation to mirror.                                                                                                                                                 |
 | GITEA_URL                   | yes      | string | -       | The url of your Gitea server.                                                                                                                                                                          |
 | GITEA_TOKEN                 | yes      | string | -       | The token for your gitea user (Settings -> Applications -> Generate New Token). **Attention: if this is set, the token will be transmitted to your specified Gitea instance!**                         |
-| GITHUB_TOKEN                | no*      | string | -       | GitHub token (PAT). Is mandatory in combination with `MIRROR_PRIVATE_REPOSITORIES`, `MIRROR_ISSUES`, `MIRROR_STARRED`, `MIRROR_ORGANIZATIONS`, or `SINGLE_REPO`.                                       |
+| GITHUB_TOKEN                | no*      | string | -       | GitHub token (PAT). Is mandatory in combination with `MIRROR_PRIVATE_REPOSITORIES`, `MIRROR_ISSUES`, `MIRROR_STARRED`, `MIRROR_ORGANIZATIONS`, `MIRROR_PUBLIC_ORGS`, or `SINGLE_REPO`.                |
 | MIRROR_PRIVATE_REPOSITORIES | no       | bool   | FALSE   | If set to `true` your private GitHub Repositories will be mirrored to Gitea. Requires `GITHUB_TOKEN`.                                                                                                  |
 | MIRROR_ISSUES               | no       | bool   | FALSE   | If set to `true` the issues of your GitHub repositories will be mirrored to Gitea. Requires `GITHUB_TOKEN`.                                                                                           |
 | MIRROR_STARRED              | no       | bool   | FALSE   | If set to `true` repositories you've starred on GitHub will be mirrored to Gitea. Requires `GITHUB_TOKEN`.                                                                                             |
 | MIRROR_ORGANIZATIONS        | no       | bool   | FALSE   | If set to `true` repositories from organizations you belong to will be mirrored to Gitea. Requires `GITHUB_TOKEN`.                                                                                     |
+| MIRROR_PUBLIC_ORGS          | no       | bool   | FALSE   | If set to `true` repositories from public organizations specified in `PUBLIC_ORGS` will be mirrored to Gitea, even if you're not a member. Requires `GITHUB_TOKEN`.                                   |
+| PUBLIC_ORGS                 | no       | string | ""      | Comma-separated list of public GitHub organization names to mirror when `MIRROR_PUBLIC_ORGS=true`. Case-insensitive.                                                                                    |
+| ONLY_MIRROR_ORGS            | no       | bool   | FALSE   | If set to `true` only repositories from organizations will be mirrored, skipping personal repositories. Requires `MIRROR_ORGANIZATIONS=true` or `MIRROR_PUBLIC_ORGS=true`.                           |
 | USE_SPECIFIC_USER           | no       | bool   | FALSE   | If set to `true`, the tool will use public API endpoints to fetch starred repositories and organizations for the specified `GITHUB_USERNAME` instead of the authenticated user.                        |
-| INCLUDE_ORGS                | no       | string | ""      | Comma-separated list of GitHub organization names to include when mirroring organizations. If not specified, all organizations will be included.                                                        |
-| EXCLUDE_ORGS                | no       | string | ""      | Comma-separated list of GitHub organization names to exclude when mirroring organizations. Takes precedence over `INCLUDE_ORGS`.                                                                       |
+| INCLUDE_ORGS                | no       | string | ""      | Comma-separated list of GitHub organization names to include when mirroring organizations you belong to. If not specified, all organizations will be included. Case-insensitive.                       |
+| EXCLUDE_ORGS                | no       | string | ""      | Comma-separated list of GitHub organization names to exclude when mirroring organizations. Takes precedence over `INCLUDE_ORGS`. Case-insensitive.                                                    |
 | PRESERVE_ORG_STRUCTURE      | no       | bool   | FALSE   | If set to `true`, each GitHub organization will be mirrored to a Gitea organization with the same name. If the organization doesn't exist, it will be created.                                         |
 | SINGLE_REPO                 | no       | string | -       | URL of a single GitHub repository to mirror (e.g., https://github.com/username/repo or username/repo). When specified, only this repository will be mirrored. Requires `GITHUB_TOKEN`.                 |
 | GITEA_ORGANIZATION          | no       | string | -       | Name of a Gitea organization to mirror repositories to. If doesn't exist, will be created.                                                                                                             |
@@ -66,7 +70,7 @@ All configuration is performed through environment variables. Flags are consider
 | DELAY                       | no       | int    | 3600    | Number of seconds between program executions. Setting this will only affect how soon after a new repo was created a mirror may appear on Gitea, but has no effect on the ongoing replication.           |
 | DRY_RUN                     | no       | bool   | FALSE   | If set to `true` will perform no writing changes to your Gitea instance, but log the planned actions.                                                                                                  |
 | INCLUDE                     | no       | string | "*"     | Name based repository filter (include): If any filter matches, the repository will be mirrored. It supports glob format, multiple filters can be separated with commas (`,`)                           |
-| EXCLUDE                     | no       | string | ""      | Name based repository filter (exclude). If any filter matches, the repository will not be mirrored. It supports glob format, multiple filters can be separated with commas (`,`). `EXCLUDE` filters are applied after `INCLUDE` ones. 
+| EXCLUDE                     | no       | string | ""      | Name based repository filter (exclude). If any filter matches, the repository will not be mirrored. It supports glob format, multiple filters can be separated with commas (`,`). `EXCLUDE` filters are applied after `INCLUDE` ones.
 | SINGLE_RUN                  | no       | bool   | FALSE   | If set to `TRUE` the task is only executed once.                                                                                                                                                       |
 
 ### Docker
@@ -116,6 +120,22 @@ docker container run \
  jaedle/mirror-to-gitea:latest
 ```
 
+### Mirror Only Organization Repositories
+
+```sh
+docker container run \
+ -d \
+ --restart always \
+ -e GITHUB_USERNAME=github-user \
+ -e GITEA_URL=https://your-gitea.url \
+ -e GITEA_TOKEN=please-exchange-with-token \
+ -e GITHUB_TOKEN=your-github-token \
+ -e MIRROR_ORGANIZATIONS=true \
+ -e ONLY_MIRROR_ORGS=true \
+ -e PRESERVE_ORG_STRUCTURE=true \
+ jaedle/mirror-to-gitea:latest
+```
+
 ### Mirror a Single Repository
 
 ```sh
@@ -163,6 +183,24 @@ docker container run \
 
 This configuration will mirror all starred repositories to a Gitea organization named "github" and will not mirror issues for these starred repositories.
 
+### Mirror Public Organizations
+
+```sh
+docker container run \
+ -d \
+ --restart always \
+ -e GITHUB_USERNAME=github-user \
+ -e GITEA_URL=https://your-gitea.url \
+ -e GITEA_TOKEN=please-exchange-with-token \
+ -e GITHUB_TOKEN=your-github-token \
+ -e MIRROR_PUBLIC_ORGS=true \
+ -e PUBLIC_ORGS=proxmox,kubernetes,microsoft \
+ -e PRESERVE_ORG_STRUCTURE=true \
+ jaedle/mirror-to-gitea:latest
+```
+
+This configuration will mirror public repositories from the specified public organizations (Proxmox, Kubernetes, and Microsoft) even if you're not a member of these organizations. The repositories will be organized under matching organization names in Gitea.
+
 ### Docker Compose
 
 ```yaml
@@ -184,6 +222,10 @@ services:
       # - INCLUDE_ORGS=org1,org2
       # - EXCLUDE_ORGS=org3,org4
       # - PRESERVE_ORG_STRUCTURE=true
+      # - ONLY_MIRROR_ORGS=true
+      # Public organization options
+      # - MIRROR_PUBLIC_ORGS=true
+      # - PUBLIC_ORGS=proxmox,kubernetes,microsoft
       # Other options
       # - SINGLE_REPO=https://github.com/organization/repository
       # - GITEA_ORGANIZATION=my-organization
@@ -213,12 +255,17 @@ export GITHUB_USERNAME='...'
 export GITHUB_TOKEN='...'
 export GITEA_URL='...'
 export GITEA_TOKEN='...'
-export MIRROR_ISSUES='true'
-export MIRROR_STARRED='true'
-export MIRROR_ORGANIZATIONS='true'
+export MIRROR_ISSUES=true
+export MIRROR_STARRED=true
+export MIRROR_ORGANIZATIONS=true
+# export ONLY_MIRROR_ORGS=true
 # export INCLUDE_ORGS='org1,org2'
 # export EXCLUDE_ORGS='org3,org4'
-# export PRESERVE_ORG_STRUCTURE='true'
+# export PRESERVE_ORG_STRUCTURE=true
+# Public organization options
+# export MIRROR_PUBLIC_ORGS=true
+# export PUBLIC_ORGS='proxmox,kubernetes,microsoft'
+# Other options
 # export SINGLE_REPO='https://github.com/user/repo'
 # export GITEA_ORGANIZATION='my-organization'
 # export GITEA_ORG_VISIBILITY='public'
@@ -230,6 +277,94 @@ Execute the script in foreground:
 task run-local
 ```
 
+### Testing Organization Mirroring
+
+To test organization mirroring specifically, you can use the provided `test-org-mirror.sh` script:
+
+```sh
+./test-org-mirror.sh
+```
+
+This script will:
+1. Build the Docker image
+2. Run the container with the following settings:
+   - `MIRROR_ORGANIZATIONS=true` - Enable organization mirroring
+   - `ONLY_MIRROR_ORGS=true` - Only mirror organization repositories, skip personal repositories
+   - `PRESERVE_ORG_STRUCTURE=true` - Create matching organizations in Gitea
+
+### Common Issues and Troubleshooting
+
+#### GitHub Token Requirements
+
+When mirroring organizations, be aware that some organizations have policies that restrict access via personal access tokens. If you encounter an error like:
+
+```
+The 'OrgName' organization forbids access via a fine-grained personal access tokens if the token's lifetime is greater than 366 days.
+```
+
+You'll need to:
+1. Go to your GitHub account settings
+2. Navigate to Personal Access Tokens
+3. Create a new token with a lifetime less than 366 days
+4. Update the `GITHUB_TOKEN` in your `.secrets.rc` file
+
+#### No Organizations Found
+
+If you see a message like:
+
+```
+Found 0 organizations:
+No organizations to process after filtering. Check your INCLUDE_ORGS and EXCLUDE_ORGS settings.
+```
+
+Possible causes and solutions:
+- **Token permissions**: Ensure your GitHub token has the `read:org` scope
+- **Organization membership**: Verify you are a member of the organizations you're trying to mirror
+- **Include/Exclude settings**: Check your `INCLUDE_ORGS` and `EXCLUDE_ORGS` settings
+
+#### No Repositories Found for Organization
+
+If you see a message like:
+
+```
+Found 0 repositories for org: OrgName
+```
+
+Possible causes and solutions:
+- **Repository access**: Ensure you have access to the repositories in the organization
+- **Empty organization**: The organization might not have any repositories
+- **Token permissions**: Ensure your GitHub token has the `repo` scope for private repositories
+
+#### Organization Creation Fails in Gitea
+
+If you see errors when creating organizations in Gitea:
+
+```
+Error creating Gitea organization OrgName: ...
+```
+
+Possible causes and solutions:
+- **Gitea token permissions**: Ensure your Gitea token has organization creation permissions
+- **Organization already exists**: The organization might already exist in Gitea with a different case (Gitea is case-insensitive for organization names)
+- **Gitea version**: Ensure you're using a compatible version of Gitea
+
+> Note: Local Gitea instance for testing
+```sh
+docker network create gitea
+docker volume create --driver local gitea
+
+docker run -d \
+  --name gitea \
+  --restart always \
+  --network gitea \
+  -v gitea:/data \
+  -v /etc/timezone:/etc/timezone:ro \
+  -v /etc/localtime:/etc/localtime:ro \
+  -p 3000:3000 \
+  -p 222:22 \
+  docker.gitea.com/gitea:1.23.6
+```
+
 ## Kudos
 
 Kudos to all contributors! 🙏

build-multiarch.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+# Script to build and push multi-architecture Docker images for mirror-to-gitea
+
+DOCKER_USERNAME="arunavo4"
+REPO_NAME="mirror-to-gitea"
+TAG="latest"
+BUILDER_NAME="multiarch-builder"
+
+# Check if builder exists, else create with docker-container driver
+if ! docker buildx inspect "$BUILDER_NAME" >/dev/null 2>&1; then
+  echo "Creating buildx builder '$BUILDER_NAME' with docker-container driver..."
+  docker buildx create --name "$BUILDER_NAME" --driver docker-container --use
+else
+  echo "Using existing buildx builder '$BUILDER_NAME'"
+  docker buildx use "$BUILDER_NAME"
+fi
+
+# Ensure builder is bootstrapped
+docker buildx inspect --bootstrap
+
+echo "Building and pushing multi-architecture images for $DOCKER_USERNAME/$REPO_NAME:$TAG"
+docker buildx build --platform linux/amd64,linux/arm64 \
+  --tag $DOCKER_USERNAME/$REPO_NAME:$TAG \
+  --push \
+  .
+
+if [ ! -z "$1" ]; then
+  VERSION_TAG="$1"
+  echo "Also tagging as $DOCKER_USERNAME/$REPO_NAME:$VERSION_TAG"
+  docker buildx build --platform linux/amd64,linux/arm64 \
+    --tag $DOCKER_USERNAME/$REPO_NAME:$VERSION_TAG \
+    --push \
+    .
+fi
+
+echo "Multi-architecture images built and pushed successfully!"
+echo "Supported architectures:"
+echo "- linux/amd64 (Intel/AMD 64-bit)"
+echo "- linux/arm64 (ARM 64-bit, e.g., Apple Silicon, newer Raspberry Pi)"
+
+echo ""
+echo "Usage:"
+echo "docker pull $DOCKER_USERNAME/$REPO_NAME:$TAG"
+echo ""
+echo "Docker will automatically select the correct image for your architecture."

debug.sh

@@ -51,12 +51,17 @@ else
     echo "$PUBLIC_USER_ORGS" | jq '.[].login'
 fi
 
-echo "Method 3 - Looking for specific organizations:"
-for org in "Gameplex-labs" "Neucruit" "uiastra"; do
+echo "Method 3 - Looking for specific organizations (if any):"
+# Get organizations from INCLUDE_ORGS environment variable
+INCLUDE_ORGS_ARR=(${INCLUDE_ORGS//,/ })
+if [ ${#INCLUDE_ORGS_ARR[@]} -eq 0 ]; then
+    echo "No organizations specified in INCLUDE_ORGS. Skipping direct organization checks."
+else
+    for org in "${INCLUDE_ORGS_ARR[@]}"; do
     ORG_DETAILS=$(curl -s -H "Authorization: token $GITHUB_TOKEN" "https://api.github.com/orgs/$org")
     if [[ $(echo "$ORG_DETAILS" | jq 'has("login")') == "true" ]]; then
         echo "Found organization: $org"
-        
+
         # Check if we can access the organization's repositories
         ORG_REPOS=$(curl -s -H "Authorization: token $GITHUB_TOKEN" "https://api.github.com/orgs/$org/repos?per_page=1")
         REPO_COUNT=$(echo "$ORG_REPOS" | jq '. | length')
@@ -70,6 +75,7 @@ for org in "Gameplex-labs" "Neucruit" "uiastra"; do
         echo "Could not find organization: $org (or no permission to access it)"
     fi
 done
+fi
 
 echo -e "\nTesting GitHub starred repos access:"
 echo "Method 1 - Using /user/starred endpoint (authenticated user):"
@@ -105,16 +111,16 @@ echo "Found $REPO_COUNT recently updated repositories to check for issues"
 for i in $(seq 0 $(($REPO_COUNT - 1))); do
     REPO=$(echo "$USER_REPOS" | jq -r ".[$i].full_name")
     REPO_HAS_ISSUES=$(echo "$USER_REPOS" | jq -r ".[$i].has_issues")
-    
+
     if [ "$REPO_HAS_ISSUES" = "true" ]; then
         ISSUES_RESPONSE=$(curl -s -H "Authorization: token $GITHUB_TOKEN" "https://api.github.com/repos/$REPO/issues?state=all&per_page=1")
         ISSUES_COUNT=$(curl -s -H "Authorization: token $GITHUB_TOKEN" -I "https://api.github.com/repos/$REPO/issues?state=all" | grep -i "^link:" | grep -o "page=[0-9]*" | sort -r | head -1 | cut -d= -f2 || echo "0")
-        
+
         if [ -z "$ISSUES_COUNT" ]; then
             # If we couldn't get the count from Link header, count the array length
             ISSUES_COUNT=$(echo "$ISSUES_RESPONSE" | jq '. | length')
         fi
-        
+
         if [ "$ISSUES_COUNT" -gt 0 ]; then
             echo "Repository $REPO has approximately $ISSUES_COUNT issues"
             echo "Latest issue: $(echo "$ISSUES_RESPONSE" | jq -r '.[0].title // "No title"')"

package.json

@@ -25,6 +25,7 @@
 		"@babel/plugin-transform-runtime": "^7.25.4",
 		"@octokit/rest": "^21.0.2",
 		"@types/jest": "^29.5.13",
+		"dotenv": "^16.5.0",
 		"minimatch": "^10.0.1",
 		"p-queue": "^8.0.1",
 		"pino": "^9.6.0",

run-local.sh

@@ -18,15 +18,16 @@ docker container run \
  -e GITEA_URL="$GITEA_URL_DOCKER" \
  -e GITEA_TOKEN="$GITEA_TOKEN" \
  -e GITHUB_TOKEN="$GITHUB_TOKEN" \
- -e MIRROR_PRIVATE_REPOSITORIES="true" \
- -e MIRROR_ISSUES="true" \
- -e MIRROR_STARRED="true" \
- -e MIRROR_ORGANIZATIONS="true" \
+ -e MIRROR_PRIVATE_REPOSITORIES=true \
+ -e MIRROR_ISSUES=false \
+ -e MIRROR_STARRED=false \
+ -e MIRROR_ORGANIZATIONS=true \
+ -e ONLY_MIRROR_ORGS="$ONLY_MIRROR_ORGS" \
  -e USE_SPECIFIC_USER="$USE_SPECIFIC_USER" \
  -e INCLUDE_ORGS="$INCLUDE_ORGS" \
  -e EXCLUDE_ORGS="$EXCLUDE_ORGS" \
  -e PRESERVE_ORG_STRUCTURE="$PRESERVE_ORG_STRUCTURE" \
  -e GITEA_STARRED_ORGANIZATION="$GITEA_STARRED_ORGANIZATION" \
  -e SKIP_STARRED_ISSUES="$SKIP_STARRED_ISSUES" \
- -e DRY_RUN="true" \
+ -e DRY_RUN=false \
  jaedle/mirror-to-gitea:development