Code

Add cache at Step level#766

Code

Merged

gabrielmbmb merged 75 commits intodevelopargilla-io/distilabel:developfrom

cache-per-stepargilla-io/distilabel:cache-per-stepCopy head branch name to clipboard

Oct 7, 2024

Commits

Commits on Jul 1, 2024

• Add signature method for Serializable objects

  plaguss

  committed
  47d0bca

  Copy full SHA for 47d0bca

Commits on Jul 2, 2024

• Update signature to only keep track of the step names and not it's internal info

  plaguss

  committed
  a8d496b

  Copy full SHA for a8d496b

Commits on Jul 3, 2024

• Refactor hash generation

  plaguss

  committed
  614512e

  Copy full SHA for 614512e

• Add dummy batch manager from dag

  plaguss

  committed
  ba7de3d

  Copy full SHA for ba7de3d

• Update batch manager cache tests to start batch manager from a DAG

  plaguss

  committed
  0b7bd47

  Copy full SHA for 0b7bd47

• Draft of integration tests for new caching

  plaguss

  committed
  0a0fb6d

  Copy full SHA for 0a0fb6d

• Checkpoint draft

  plaguss

  committed
  03501cc

  Copy full SHA for 03501cc

• Merge and fix conflict

  plaguss

  committed
  813afe8

  Copy full SHA for 813afe8

Commits on Jul 4, 2024

• Merge branch 'develop' of https://github.com/argilla-io/distilabel into cache-per-step

  plaguss

  committed
  4a86130

  Copy full SHA for 4a86130

Commits on Jul 8, 2024

• Add cache directory location

  plaguss

  committed
  cbdbb60

  Copy full SHA for cbdbb60

• Add use_cache argument to Step for future use

  plaguss

  committed
  856cc8a

  Copy full SHA for 856cc8a

• Change output names to keep track of them while debugging

  plaguss

  committed
  952e921

  Copy full SHA for 952e921

• Make use of use_cache at the step level

  plaguss

  committed
  54b783e

  Copy full SHA for 54b783e

• Add docstrings for internal batch manager arguments

  plaguss

  committed
  09ee7e2

  Copy full SHA for 09ee7e2

Commits on Jul 9, 2024

• Remove path from add_batch method

  plaguss

  committed
  0d37200

  Copy full SHA for 0d37200

• Move step caching to get_batch method in batch manager step

  plaguss

  committed
  04507f9

  Copy full SHA for 04507f9

• Read batches from cached dir

  plaguss

  committed
  0a0c7ca

  Copy full SHA for 0a0c7ca

• Merge branch 'develop' of https://github.com/argilla-io/distilabel into cache-per-step

  plaguss

  committed
  4c40385

  Copy full SHA for 4c40385

• Set every step cache to False if the pipeline has the cache as False

  plaguss

  committed
  40647db

  Copy full SHA for 40647db

• Comment for the batch manager

  plaguss

  committed
  0bf7a02

  Copy full SHA for 0bf7a02

• Move back to caching from add_step

  plaguss

  committed
  333b749

  Copy full SHA for 333b749

• Merge and fix conflict

  plaguss

  committed
  6270025

  Copy full SHA for 6270025

Commits on Jul 15, 2024

• Checkpoint current status

  plaguss

  committed
  9e5674a

  Copy full SHA for 9e5674a

• Add use_cache on step

  plaguss

  committed
  e43fa1d

  Copy full SHA for e43fa1d

• If there's previous data saved, concatenate the content of the parquet files

  plaguss

  committed
  2a1baba

  Copy full SHA for 2a1baba

Commits on Jul 16, 2024

• Only read the distiset from cache if all the steps are the same, otherwise overwrite

  plaguss

  committed
  cbae3dc

  Copy full SHA for cbae3dc

• Add changes to make loading a new and modified step feasible

  plaguss

  committed
  e4a4967

  Copy full SHA for e4a4967

Commits on Jul 19, 2024

• Set use cache to True by default

  plaguss

  committed
  8df68d8

  Copy full SHA for 8df68d8

• Move logic of registering the batches to BasePipeline._register_batch to do it before calling _manage_batch_flows

  plaguss

  committed
  2f06943

  Copy full SHA for 2f06943

• Avoid reading parquet file from cache when any of the steps has use_cach=False

  plaguss

  committed
  8a01e17

  Copy full SHA for 8a01e17

• Add is_convergence method to DAG and cleanup batch_manager

  plaguss

  committed
  279b6a6

  Copy full SHA for 279b6a6

• Add integration tests for the new caching mechanism

  plaguss

  committed
  b96a353

  Copy full SHA for b96a353

• Merge branch 'develop' of https://github.com/argilla-io/distilabel into cache-per-step

  plaguss

  committed
  07c29c6

  Copy full SHA for 07c29c6

• Update unit tests related to register_batch

  plaguss

  committed
  a8431e9

  Copy full SHA for a8431e9

• Fix signature serialization case of void list

  plaguss

  committed
  822f8d4

  Copy full SHA for 822f8d4

• Add use_cache to argilla tests

  plaguss

  committed
  2a202df

  Copy full SHA for 2a202df

• Fix tests related to use_cache

  plaguss

  committed
  13726b9

  Copy full SHA for 13726b9

• Fix tests

  plaguss

  committed
  08e35a2

  Copy full SHA for 08e35a2

• Remove undefined object input

  plaguss

  committed
  a720d54

  Copy full SHA for a720d54

Commits on Jul 31, 2024

• Merge branch 'develop' into cache-per-step

  gabrielmbmb

  committed
  b03bc74

  Copy full SHA for b03bc74

Commits on Aug 1, 2024

• Merge branch 'develop' into cache-per-step

  gabrielmbmb

  committed
  423d90d

  Copy full SHA for 423d90d

• Add `_invalidate_steps_cache_if_required` method

  gabrielmbmb

  committed
  ed24321

  Copy full SHA for ed24321

• Initial work for loading batches from `batch_manager_data` directory

  gabrielmbmb

  committed
  48f2123

  Copy full SHA for 48f2123

Commits on Aug 14, 2024

• Merge and solve conflicts

  plaguss

  committed
  d8417a2

  Copy full SHA for d8417a2

• Merge and solve conflicts

  plaguss

  committed
  9efbfdc

  Copy full SHA for 9efbfdc

Commits on Aug 15, 2024

• Draft cache updates

  plaguss

  committed
  223f6b2

  Copy full SHA for 223f6b2

Commits on Sep 17, 2024

• Merge and fix conflict

  plaguss

  committed
  c1ce37b

  Copy full SHA for c1ce37b

• Update pipeline signature

  plaguss

  committed
  0080e89

  Copy full SHA for 0080e89

• Add signature mixin from other PR

  plaguss

  committed
  6cd78c8

  Copy full SHA for 6cd78c8

• Moved pipeline cache to executions folder with different data per pipeline

  plaguss

  committed
  3a0e4c3

  Copy full SHA for 3a0e4c3

Commits on Sep 20, 2024

• Testing new updates to read from cache

  plaguss

  committed
  a45b625

  Copy full SHA for a45b625

• Checkpoint with loading working while adding new steps

  plaguss

  committed
  37a3e24

  Copy full SHA for 37a3e24

Commits on Sep 25, 2024

• Point of control

  plaguss

  committed
  bb28b0b

  Copy full SHA for bb28b0b

Commits on Oct 1, 2024

• Merge branch 'develop' into cache-per-step

  gabrielmbmb

  committed
  5efe760

  Copy full SHA for 5efe760

Commits on Oct 2, 2024

• Fix not all the batches where being saved

  gabrielmbmb

  committed
  79cee36

  Copy full SHA for 79cee36

• Sort batches after loaded

  gabrielmbmb

  committed
  3d254e2

  Copy full SHA for 3d254e2

• Fix `load_from_cache` to load batches from `steps_data` directory

  Show description for 791a28a
  gabrielmbmb

  committed
  791a28a

  Copy full SHA for 791a28a

• Update test

  gabrielmbmb

  committed
  56babcf

  Copy full SHA for 56babcf

• Add `step_has_finished` method

  gabrielmbmb

  committed
  bd8d1f6

  Copy full SHA for bd8d1f6

Commits on Oct 3, 2024

• Update invalidate cache function

  gabrielmbmb

  committed
  05f1047

  Copy full SHA for 05f1047

• Update integration caching tests

  gabrielmbmb

  committed
  3b4b59e

  Copy full SHA for 3b4b59e

• Refactor to extract logic to methods

  gabrielmbmb

  committed
  2cb633b

  Copy full SHA for 2cb633b

• Refactor to remove `cached_data_dir`

  gabrielmbmb

  committed
  d039df2

  Copy full SHA for d039df2

• Update stages message

  gabrielmbmb

  committed
  a4967ac

  Copy full SHA for a4967ac

Commits on Oct 4, 2024

• Refactor `invalidate_cache_for` method

  gabrielmbmb

  committed
  1c4edd0

  Copy full SHA for 1c4edd0

• Fix `_BatchManager` unit tests

  gabrielmbmb

  committed
  72ab5cd

  Copy full SHA for 72ab5cd

• Merge branch 'develop' into cache-per-step

  gabrielmbmb

  committed
  dac6bf3

  Copy full SHA for dac6bf3

• Update to not serialize `exclude_from_signature` attribute

  gabrielmbmb

  committed
  09b588a

  Copy full SHA for 09b588a

• Fix pipeline unit tests

  gabrielmbmb

  committed
  95f7618

  Copy full SHA for 95f7618

Commits on Oct 7, 2024

• Merge branch 'develop' into cache-per-step

  gabrielmbmb

  committed
  333e346

  Copy full SHA for 333e346

• Remove write buffer data if `use_cache=False`

  gabrielmbmb

  committed
  0b35f96

  Copy full SHA for 0b35f96

• Fix offline batch generation attributes were being not ignored by

  Show description for f65d614
  gabrielmbmb

  committed
  f65d614

  Copy full SHA for f65d614

• Merge branch 'develop' into cache-per-step

  gabrielmbmb

  committed
  a6349da

  Copy full SHA for a6349da

• Fix print test

  gabrielmbmb

  committed
  6a79f0f

  Copy full SHA for 6a79f0f

• Fix routing batch function

  gabrielmbmb

  committed
  3ea2b85

  Copy full SHA for 3ea2b85