search pagination not working

[ujeong1]

I am trying to use the search API (v1 and v2) through a bearer token for mastodon.social. I update the offset, min_id, and max_id each time I iterate through the search results, but none of them seem to work. The same issue persists when I only use offset.

Below is a snapshot of the code I’m using:

from mastodon import Mastodon, MastodonError

mastodon = Mastodon(api_base_url=host, access_token=access_token)

offset = 0
min_id, max_id = None, None

while True:
    try:
        results = mastodon.search(q=query, offset=offset, min_id=max_id)
        accounts = results['accounts']
        accounts = sorted(accounts, key=lambda x: x['id'])

        # Update min_id and max_id for the next iteration
        min_id = accounts[0]['id']
        max_id = accounts[-1]['id']

        # Increment the offset for pagination
        offset += 40

        # Print progress
        print(f"# Unique Users Found So Far: {len(accounts)}, Current Offset: {offset}")

    except MastodonError as e:
        print(f"Error: {e}")
        break

[halcy]

Hey, I will try to look at it tomorrow, ignore the weird bot post, it probably is nonsense.

[ujeong1]

Lol I found other way to tweak this issue by using account_search, but I had to modify the code to pass offset as a parameter (there is no offset parameter in the original mastodon.py code for account_search)

Another issue I encountered is that I cannot increase offset more than 10,000, and the server throws 500 error. Probably they wanted to prevent any excessive crawling.

Is there any way to crawl more than 10,000 accounts? I am not doing anything suspicious, but just wanted to know some statistics.