Granblue Automation For Android using MediaProjection, AccessibilityService, and OpenCV

Python version here: https://github.com/steve1316/granblue-automation-pyautogui

This Android application written in Kotlin is designed for educational research purposes on studying the Android architecture and using system services such as MediaProjection and AccessibilityService in conjunction with OpenCV to automate certain workflows via image template matching.

Screenshots are taken via the MediaProjection and OpenCV performs image processing. Gestures are performed by AccessibilityService in order to perform various actions such as tapping, swiping or scrolling the screen.

Table of Contents

• Features
• Requirements
• Instructions
• Technologies used

Disclaimer

By downloading this application, you consent to your account potentially getting flagged for excessive usage and banned by KMR. I hold no responsibility for how much you use this application for. I trust you have the self-control necessary to use this application in reasonable bursts of time and to give yourself breaks in between.

Features

• Choose between various Farming Modes:
  • Quest
  • Special
  • Coop
  • Raid
  • Event
  • Event (Token Drawboxes)
  • Rise of the Beasts
  • Guild Wars
  • Dread Barrage
• Choose your own custom combat script.
• Alert for when anti-bot CAPTCHA pops up.
• Customizable settings such as Item/Summon selection and delays between each run.

Requirements

1. Android Device (API 29+ or Android 10+)
   1. APK available to download is for devices that support arm64-v8a ABI. For devices that use a different ABI, follow the instructions below on how to build your own APK.
2. Granblue Fantasy account
3. Twitter Developer account (optional, needed to farm Raids)

Instructions

1. Download the .apk file from the Releases section on the right and install it on your Android device. If you want to build the .apk yourself, do the following:
   1. Download and extract the project repository.
   2. Go to https://opencv.org/releases/ and download OpenCV 4.5.1 (make sure to download the Android version of OpenCV) and extract it.
   3. Create a new folder inside the root of the project repository named opencv and copy the extracted files in /OpenCV-android-sdk/sdk/ from Step 2 into it.
   4. Open the project repository in Android Studio and you can now build and run on your Android Device or build your own .apk file.
   5. You can set universalApk to true in the build.gradle for the application to build a one-for-all .apk file or adjust the include 'arm64-v8a' to customize which ABI to build the .apk file for.
2. Once you have it running, fill out the required sections marked with * in the Settings page of the application.
3. Now go back to the Home page. The settings you have selected will be shown to you in the text box below the Start button.
4. Now tap on the Start button. If this is the first time, it will ask you to give the application Overlay permission and then to enable its Accessibility Service.
5. Once both are enabled, tapping on the Start button again will display an overlay button that you can move around the screen.
6. Navigate yourself to Granblue Fantasy's main page and press the overlay button to start. It is recommended to move the overlay button to the bottom left corner such that it covers the Back button to avoid interruption.
7. Whenever you want to stop, press the overlay button again to terminate the bot.

If you want to farm Raids, follow the instructions below:

1. Make sure to run the application at least once. Check the application's files folder located inside internal storage's Android/data/com.steve1316.granblueautomation_android/ directory.
2. Locate the file named config.yaml file. For ease of use, send it to your computer.
3. Now follow the instructions under the Instructions for the Raid component of this application (optional, only if you want to farm Raids) section of https://github.com/steve1316/granblue-automation-pyautogui to obtain your own consumer keys and tokens to connect to the Twitter API.
4. Once you have them, copy them into the respective fields inside config.yaml using your preferred text editor. After that, move the file back to where it belonged on your Android device.

Technologies Used

1. MediaProjection - Used to obtain full screenshots
2. AccessibilityService - Used to dispatch gestures like tapping and scrolling
3. OpenCV Android 4.5.1 - Used to template match
4. Google Firebase ML Kit - Used for text detection
5. Hoplite - Used to read config.yaml for customized user settings
6. Twitter4j - Used to connect to Twitter API to fetch room codes