Handle CredentialsExpired exception in worker (#84)

* Expose db port for testing

* consistently use `dev` in `Makefile`

we use it in other jobs as well

* update docs on how to run it locally

* Handle `CredentialsExpired` exception

This requires a version bump of the Spotify Connector,
because it wasn't exposed before

connector_manager/Makefile

@@ -5,8 +5,8 @@ SHELL := /bin/bash
 help: ## help message, list all command
 	@echo -e "$$(grep -hE '^\S+:.*##' $(MAKEFILE_LIST) | sed -e 's/:.*##\s*/:/' -e 's/^\(.\+\):\(.*\)/\\x1b[36m\1\\x1b[m:\2/' | column -c2 -t -s :)"
 
-.PHONY: run
-run: ## run the job
+.PHONY: run dev
+run dev: ## run the job
 	python3 -m manager
 
 .PHONY: test

connector_manager/requirements.txt

@@ -1,7 +1,7 @@
 mysql_connector_python==8.0.33
 python_gnupg==0.5.0
 appleconnector==0.4.0
-spotifyconnector==0.8.1
+spotifyconnector==0.8.2
 anchorconnector==0.4.0
 autopep8==1.7.0
 backoff==1.11.1

docker-compose.yaml

@@ -20,6 +20,8 @@ services:
             - db
     db:
         image: mysql:8.0
+        ports:
+            - "3306:3306"  # Bind port 3306 on the host to port 3306 on the container
         volumes:
             - mysqldata:/var/lib/mysql
             - ./db_local_dev/schema.sql:/docker-entrypoint-initdb.d/00_schema.sql
@@ -31,4 +33,4 @@ services:
         command: --innodb-buffer-pool-size=128M --innodb-buffer-pool-chunk-size=64M --key-buffer-size=5M --event-scheduler=ON
 
 volumes:
-    mysqldata:
+    mysqldata:

spotify/README.md

@@ -4,11 +4,23 @@ This uses our Spotify connector to fetch data for our podcasts from the Spotify
 API. To add a new podcast, create a new workflow file in the `.github/workflows`
 directory and set the required environment variables.
 
-The connector is available on [GitHub](https://github.com/openpodcast/spotify-connector).
+The Spotify connector project is located
+[here](https://github.com/openpodcast/spotify-connector).
 
-## Docker Image
+## Getting Started
 
-We provide a Docker image for the connector. You can use it to run the connector locally.
+### Local Development
+
+Here is how you can run the pipeline locally:
+
+1. Install the dependencies with `pip install -r requirements.txt`.
+2. Create a `.env` file with your Spotify API credentials.
+   Take a look at the `.env.sample` file for reference.
+3. Run the pipeline with `make dev`.
+
+### Docker Image
+
+Alternatively, we provide a Docker image for the connector.
 
 By default, the connector will fetch data from Spotify once a day.
 You can set your own schedule for executing the connector like so: