Organisation mirror not working #31
Comments
|
Thanks for raising the issue and sorry for the (potential) bug. I will have a look this evening. |
|
@CrazyWolf13 Use this endpoint to check if your PAT token has the proper access curl -L \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer <YOUR-TOKEN>" \
-H "X-GitHub-Api-Version: 2022-11-28" \
https://api.github.com/orgs/<ORG_NAME>/personal-access-tokens/<PAT_TOKEN>/repositories |
|
@arunavo4 I've just tested the above, and my PAT where my account is the owner did not work, and a PAT where I made the org community-scripts ower did not work. I added the following permissions: read-access to code, commits, statuses and metadata. Also wouldn't that setup then require two PATs, one to get my personal repos and then one per org, which at the moment seems not to be possible in this project? |
Edit: Its a bug will fix it |
|
@arunavo4 In my eyes it shouldn't require a per-org access token, if I just want public repos of an org, only if I want the private ones too right? Also back to the original issue, shouldn't then the ProxmoxVE org be backed up too? In the current state the project didn't mirror a single repo from any org, not from repos where my GH User is member, nor any of the additional orgs to be backed up. |
|
@CrazyWolf13 Yes this seems to be a bug as its public repos should be accessible. Will make a PR for this and fix this issue. For private you can use the same PAT token but you need to turn on this option. 
|
|
@CrazyWolf13 as a quick fix I have just released a docker image with the last working version for your use-case v1.5.0. Feel free to downgrade until this is finally resolved. Thanks for raising the issue 🙏 |
|
Thanks a lot for looking into this! Sadly I don't see any real change with the 1.5.0 tag, with the exact same env variables : The output just looks different, and all the organisation stuffs seems to be removed again? Now mirroring my users repositorys seems to be working, however this worked before too. |
#!/bin/bash
# Docker test commands for mirror-to-gitea
# Replace these variables with your actual values
GITHUB_USERNAME="your-github-username"
GITEA_URL="http://your-gitea-instance:3000/"
GITEA_TOKEN="your-gitea-token"
GITHUB_TOKEN="your-github-token"
# Basic test with dry run mode
echo "Running basic test with dry run..."
docker run -it --rm \
-e GITHUB_USERNAME="$GITHUB_USERNAME" \
-e GITEA_URL="$GITEA_URL" \
-e GITEA_TOKEN="$GITEA_TOKEN" \
-e GITHUB_TOKEN="$GITHUB_TOKEN" \
-e MIRROR_ORGANIZATIONS=true \
-e DRY_RUN=true \
arunavo4/mirror-to-gitea:latest
# Test organization mirroring only (no personal repos)
echo "Running organization mirroring test..."
docker run -it --rm \
-e GITHUB_USERNAME="$GITHUB_USERNAME" \
-e GITEA_URL="$GITEA_URL" \
-e GITEA_TOKEN="$GITEA_TOKEN" \
-e GITHUB_TOKEN="$GITHUB_TOKEN" \
-e MIRROR_ORGANIZATIONS=true \
-e ONLY_MIRROR_ORGS=true \
-e PRESERVE_ORG_STRUCTURE=true \
-e DRY_RUN=true \
arunavo4/mirror-to-gitea:latest
# Test with specific organizations included
echo "Running test with specific organizations..."
docker run -it --rm \
-e GITHUB_USERNAME="$GITHUB_USERNAME" \
-e GITEA_URL="$GITEA_URL" \
-e GITEA_TOKEN="$GITEA_TOKEN" \
-e GITHUB_TOKEN="$GITHUB_TOKEN" \
-e MIRROR_ORGANIZATIONS=true \
-e INCLUDE_ORGS="org1,org2" \
-e PRESERVE_ORG_STRUCTURE=true \
-e DRY_RUN=true \
arunavo4/mirror-to-gitea:latest
# Uncomment to run with actual mirroring (no dry run)
# echo "Running actual mirroring..."
# docker run -it --rm \
# -e GITHUB_USERNAME="$GITHUB_USERNAME" \
# -e GITEA_URL="$GITEA_URL" \
# -e GITEA_TOKEN="$GITEA_TOKEN" \
# -e GITHUB_TOKEN="$GITHUB_TOKEN" \
# -e MIRROR_ORGANIZATIONS=true \
# -e ONLY_MIRROR_ORGS=true \
# -e PRESERVE_ORG_STRUCTURE=true \
# -e DRY_RUN=false \
# arunavo4/mirror-to-gitea:latest
# Uncomment to run as a background service
# echo "Running as background service..."
# docker run -d --name mirror-to-gitea \
# -e GITHUB_USERNAME="$GITHUB_USERNAME" \
# -e GITEA_URL="$GITEA_URL" \
# -e GITEA_TOKEN="$GITEA_TOKEN" \
# -e GITHUB_TOKEN="$GITHUB_TOKEN" \
# -e MIRROR_ORGANIZATIONS=true \
# -e ONLY_MIRROR_ORGS=true \
# -e PRESERVE_ORG_STRUCTURE=true \
# -e DRY_RUN=false \
# arunavo4/mirror-to-gitea:latest |
|
The README now also contains how to fix any issues and the log also shows why it fails in mirroring orgs if it fails |
This comment has been minimized.
This comment has been minimized.
|
Ah got it, it's only available for arm, I have amd_64. |
|
@CrazyWolf13 Sorry My bad i pushed only from my macbook so there is only arm image. Try again I have added build for other platforms also now. |
|
@arunavo4 Thanks! Just deployed, and it seemed to create the org ProxmoxVE (that however was the wrong org, which contains nothing: https://github.com/ProxmoxVE ) I expect the following:
|
|
@CrazyWolf13 please share your options that you ran this with, also I am assuming you are a member of Proxmox Org? |
|
@arunavo4 The options are listed on the top in the run command right?
No I'm not a member of the ProxmoxVE org, but why would I need to be? I think I once got that working when experimenting with python. |
|
@CrazyWolf13 Hey I think we have a misunderstanding here this only mirrors what you are a member / owner of. THis option does not mirror random public repos that you specify. But I understand the use case. Let me add this feature in. |
|
Update there seems to be a lower/highcase problem, I entered the following in the INCLUDE_ORGS="Proxmox", using "proxmox" it worked |
|
@CrazyWolf13 It worked ? ohk casuse alright i get it the endpoint to fetch is same. will fix the lower/ uppeer case issue |
|
@arunavo4 What I don't understand, is why the orgs I'm member are not mirrored/detected. Do I need a special token for that? |
|
@CrazyWolf13 I have tested it with my orgs that I am a memebr of and it works fine, you do need to setup this screen
Can you show me the error where its unable to find it ? |
|
@CrazyWolf13 You can see here I try to mirror my orgs I am a member of Two of the Orgs dont have the right permissons set and one of them has the permissions set so that works Using authenticated user endpoint to fetch organizations
Direct API call response status: 200
Direct API call found 0 organizations
Paginated API call found 0 organizations
No organizations found through standard endpoints. Trying direct API calls to specific organizations.
GET /orgs/Gameplex-labs - 403 with id D0DD:B7167:1F2B96:24E515:67FE0859 in 305ms
ERROR: The 'Gameplex-labs' organization has a policy that forbids access via fine-grained personal access tokens with a lifetime greater than 366 days.
Please adjust your token's lifetime or create a new token with a shorter lifetime.
See the error message for details: The 'Gameplex-labs' organization forbids access via a fine-grained personal access tokens if the token's lifetime is greater than 366 days. Please adjust your token's lifetime at the following URL: https://github.com/settings/personal-access-tokens/5859585 - https://docs.github.com/rest/orgs/orgs#get-an-organization
Successfully found organization: uiastra
GET /orgs/Neucruit - 403 with id D0DD:B7167:1F2BF8:24E58E:67FE085A in 274ms
ERROR: The 'Neucruit' organization has a policy that forbids access via fine-grained personal access tokens with a lifetime greater than 366 days.
Please adjust your token's lifetime or create a new token with a shorter lifetime.
See the error message for details: The 'Neucruit' organization forbids access via a fine-grained personal access tokens if the token's lifetime is greater than 366 days. Please adjust your token's lifetime at the following URL: https://github.com/settings/personal-access-tokens/5859585 - https://docs.github.com/rest/orgs/orgs#get-an-organization
Found 1 organizations: uiastra
Processing repositories from 1 organizations: uiastra
Fetching repositories for organization: uiastra
Using search API to fetch both public and private repositories for org: uiastra
Direct search API call response status: 200
Direct search API call found 3 repositories
Found 0 repositories (public and private) for org: uiastra
No repositories found using search API for org: uiastra. Trying standard API...
Found 3 repositories using standard API for org: uiastra
Only mirroring organization repositories
Found 2 repositories to mirror |
|
@arunavo4
Does the INCLUDE_ORGS actually limit it to only what is entered there? I also don't see any error, in any of the orgs I'm a member. Who is the owner of your GITHUB_PAT?
|
|
@CrazyWolf13 Yes it will only match I will make this seperate for public repos and own repos need to add some clarity.
I am the owner of the PAT token |
|
Oh okay, got it.
|
|
@CrazyWolf13 Yea cleaning it up |
|
@CrazyWolf13 Go ahead and test this out new option docker container run \
-d \
--restart always \
-e GITHUB_USERNAME=github-user \
-e GITEA_URL=https://your-gitea.url \
-e GITEA_TOKEN=please-exchange-with-token \
-e GITHUB_TOKEN=your-github-token \
-e MIRROR_PUBLIC_ORGS=true \
-e PUBLIC_ORGS=proxmox \
-e PRESERVE_ORG_STRUCTURE=true \
arunavo4/mirror-to-gitea:latestAlso you dont seem to be member of any organisation. So thats why 0 orgs to mirror.
|
|
Hmm weird, I'm member of two orgs, one is private and one should be public: community-scripts:
|
|
@CrazyWolf13 Weird cannot see you here also in community-scripts
|
|
@arunavo4 Seems like as a side effect now it doues not list/run single repo migration. |
|
@CrazyWolf13 what do you mean by single repo migration? Did you use the |
|
@arunavo4
Though I would expect a Github_PAT to give access to see what orgs a user is member of, even if not directly giving access to org content. Now it seems to detect the org :) However it only created the org on gitea and did not migrate any repos: |
|
@CrazyWolf13 Try now, added more debuggin to help |
|
@arunavo4 |
|
@CrazyWolf13 Try again, Added more logging to figure out the issue. |
|
@arunavo4
|
|
@CrazyWolf13 No more time today will come back in a few days and will help fix this |
|
Alright, sure :) btw. now it seems to be fixed for me with the latest commit! Maybe @jaedle can also test some scenarios, as with the current setup there are near coutless scenarious to be tested. Edit: From my side everything seems to be working. Only Private Orgs seem to be not detected at all, but maybe these require a special PAT, maybe this needs some clarification in the docs if so. I'll try to test some more possible scenarios. |
|
Hi @CrazyWolf13 |
|
Wow thanks, this looks exciting!! Ui looks great, is this nextjs? Will take a closer look during the day :) |
@CrazyWolf13 |
That looks pretty nice. |
Hi @jaedle
since the recent update yesterday, it seems to not correctly apply env vars, I started a container with:
startup log:
Starting to create mirrors... 2025-04-08T09:08:41.002Z INF config={"github":{"username":"CrazyWolf13","token":"[Redacted]","skipForks":false,"privateRepositories":true,"mirrorIssues":false,"mirrorStarred":false,"mirrorOrganizations":true,"useSpecificUser":false,"includeOrgs":["ProxmoxVE"],"excludeOrgs":["SomeOrg"],"preserveOrgStructure":true,"skipStarredIssues":false},"gitea":{"url":"XXXXX","token":"[Redacted]","organization":"CrazyWolf13-mirror","visibility":"public","starredReposOrg":"github"},"dryRun":false,"delay":3600,"include":["*"],"exclude":[""],"singleRun":false} msg=applied configuration Organization CrazyWolf13-mirror already exists Processing repositories from 0 organizations Found 12 repositories to mirror Repository Chameleons-Beauty is already mirrored in organization CrazyWolf13-mirror; doing nothing. Repository dashy is already mirrored in organization CrazyWolf13-mirror; doing nothing. Repository auto_gfpgan-installer is already mirrored in organization CrazyWolf13-mirror; doing nothing. Repository CrazyWolf13 is already mirrored in organization CrazyWolf13-mirror; doing nothing. Repository grade-store is already mirrored in organization CrazyWolf13-mirror; doing nothing. Repository ProxmoxVE is already mirrored in organization CrazyWolf13-mirror; doing nothing. Repository home-assets is already mirrored in organization CrazyWolf13-mirror; doing nothing. Repository dotfiles is already mirrored in organization CrazyWolf13-mirror; doing nothing. Repository chezmoi is already mirrored in organization CrazyWolf13-mirror; doing nothing. Repository streamlink-webui is already mirrored in organization CrazyWolf13-mirror; doing nothing. Repository web-check is already mirrored in organization CrazyWolf13-mirror; doing nothing. Repository unix-pwsh is already mirrored in organization CrazyWolf13-mirror; doing nothing. Waiting for 3600 seconds...So I expect this to mirror the following:
All my personal repos e.g. https://github.com/CrazyWolf13?tab=repositories --> to the CrazyWolf13-mirror on Gitea, seems to work
Mirror Single_Repo to CrazyWolf13-mirror, works.
Mirror the Org community-scripts for example, which I'm member of and all repos to a new Org called community-scripts on Gitea --> NOT WORKING
Mirror ProxmoxVE, which I'm not a Member, but is in additional to a new Org called ProxmoxVE on Gitea. --> NOT WORKING
Do I need some sort of special permissions on my github_pat?
I added the following:
Thanks!
EDIT: I allow myself to ping @arunavo4 as he is the author of the PR and maybe I understood something wrong about the functionality.
The text was updated successfully, but these errors were encountered: