Feature/SK-1265 | Implement SQL store #785

niklastheman · 2025-01-09T21:32:54Z

Implement SQL store

In order to implement sql stores for all entities some refactoring was needed:

MongoStateStore has been removed. All db queries are now done via the separate stores.
The API interface class has been removed. It was dependent upon mongo state store.
All the legacy endpoints in root (with the exception of /health and /get_controller_status) has been deprecated and will return 410 if called.
All previously missing required endpoints such ass add_client has been moved to api/v1
Client V1 has been removed. It used a lot of the legacy endpoints.
Client V2 (current) only uses /api/v1 endpoints
Legacy start session has been removed.
SQLAlchemy and psycopg2 dependencies added.
Postgres service added in the docker compose file.
A lot of API-tests has been removed as they are no longer relevant. Plans are in place to write new once for the /api/v1 routes.

This implementation enables running FEDn with an SQL-database. It makes use of an ORM called SQLAlchemy. This way FEDn can run with a number of different SQL-databases including PostgreSQL and SQLite.

…_store in Network class

… injected into control

…roller only

fedn/network/api/v1/client_routes.py

fedn/network/api/v1/model_routes.py

fedn/network/api/v1/package_routes.py

fedn/network/storage/statestore/stores/client_store.py

fedn/network/storage/statestore/stores/combiner_store.py

fedn/network/storage/statestore/stores/prediction_store.py

fedn/network/storage/statestore/stores/round_store.py

fedn/network/storage/statestore/stores/session_store.py

fedn/network/storage/statestore/stores/status_store.py

fedn/network/storage/statestore/stores/validation_store.py

Wrede

Awesome work!

niklastheman and others added 30 commits December 20, 2024 15:01

added graphql server

2710e6e

sessions added

bab877e

get single session

9c4ad30

added models

0fe3cfd

added validations

a29f0a1

statuses added

750cf3a

added plura/singular naming to follow convention

6dcb6af

added abstract parent class, Store along with MongoDBStore

0e596dc

removed use_typings option

9271df6

clean up

e55ad72

merged SK-1109

b069844

removed use_typings, changes to complie with feature/SK-1109

0ec6d6c

started on replacing mongostatestore

4ebb1eb

removed get_combiners from mongostatestore and started using combiner…

eaf75c0

…_store in Network class

removed add/delete combiner from network and mongostatestore

08f8583

moved get and list clients from mongostatestore

fab412a

update sessions with session_store

328217b

removed set_session_status from mongostatestore

5a5303f

added round_store to combiner

1ac6880

clean up

9d9bfa8

upload model moved to api/v1/models

010528f

removed get model

d0df714

moved add client to api/v1/clients/add

70fc83d

moved download package to api/v1/packages/download

d542140

moved get checksum to api/v1/packages/checksum

a1ac5b1

removed old client and old package runtime class

839bf39

mongostatestore dependecy to control removed and repository dependecy…

db14426

… injected into control

get_client_config moved to api/v1/clients/config

57ba8d1

removed API (interface) class

5fdc75c

added indexes to mogno stores

64fdd83

niklastheman added 7 commits January 8, 2025 18:02

create sql tables only if sql stores

51c463c

linting fix

6920404

sqllite postgres or mongo based on env:s. moved table creation to cot…

34a5818

…roller only

implemented connected_client_count for client sql store

a7d7e08

clean up

361d967

client_store update should be like every other update

b8f89b7

run table creation script from combiner to fix timing issue

669c154

niklastheman requested review from ahellander and Wrede January 9, 2025 21:32

github-advanced-security bot found potential problems Jan 9, 2025

View reviewed changes

niklastheman added 2 commits January 10, 2025 11:44

merge of master

fdc77d9

removed debug logging

ab7eafc

Wrede added HOLD feature New feature or request labels Jan 10, 2025

This SQL query depends on a user-provided value fix.

6816327

github-actions bot added the minor label Jan 10, 2025

niklastheman added 3 commits January 10, 2025 13:27

ruff linting fix#

d734c55

api client should use api/v1

72bff6a

should be able to upload package without name and description

1de8855

niklastheman removed the HOLD label Jan 13, 2025

niklastheman and others added 5 commits January 14, 2025 17:40

Api client, set_active_model fix

8dd9a5b

fixed set_active_model for sql mode

ac8b46f

set_active_package fix

31fa959

fixed client.start_session()

223e4bf

add comment with supported DBs

23f70cb

Wrede approved these changes Jan 15, 2025

View reviewed changes

niklastheman and others added 2 commits January 16, 2025 01:44

default db should still be mongo

d7d270e

remove postgres from main compose file

e7fefe2

Wrede merged commit 449e6a3 into master Jan 20, 2025
17 checks passed

Wrede deleted the feature/SK-1265 branch January 20, 2025 15:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/SK-1265 | Implement SQL store #785

Feature/SK-1265 | Implement SQL store #785

niklastheman commented Jan 9, 2025

Wrede left a comment

Feature/SK-1265 | Implement SQL store #785

Feature/SK-1265 | Implement SQL store #785

Conversation

niklastheman commented Jan 9, 2025

Implement SQL store

Wrede left a comment

Choose a reason for hiding this comment