spliting stdata into chunks on the filesystem

[Frix-x]

with all the logic to load them back and reconstruct the .stdata file

Summary by Sourcery

Split measurement data into chunks to optimize memory usage and implement logic to reassemble these chunks into a complete data file. Introduce a method to ensure data transfers are completed before proceeding, and enhance resource management by cleaning up temporary files.

New Features:

• Implement chunking of measurement data to manage memory usage by saving data in temporary files when a threshold is exceeded.

Enhancements:

• Introduce a mechanism to reassemble chunked measurement data into a single file, ensuring data integrity and completeness.
• Add a method to wait for data transfers to complete, improving reliability in data handling processes.

Chores:

• Ensure temporary files and directories are cleaned up upon object deletion to prevent resource leaks.

[sourcery-ai]

Reviewer's Guide by Sourcery

This pull request implements a chunking mechanism for storing accelerometer data on the filesystem. It introduces a new approach to handle large amounts of measurement data by splitting it into smaller chunks, saving them temporarily, and reassembling them when needed. This change aims to improve memory management and performance when dealing with extensive accelerometer data.

File-Level Changes

Change	Details	Files
Implement chunking mechanism for storing accelerometer data	Add CHUNK_SIZE constant to limit measurements in memory Introduce _temp_dir for temporary storage of chunks Implement _save_chunk method to write measurements to temporary files Add _reassemble_chunks method to combine chunk files into final .stdata file Modify save_stdata method to use the new chunking mechanism Update get_measurements method to load data from chunks	shaketune/helpers/accelerometer.py
Improve file writing and data transfer management	Replace wait_for_file_writes with wait_for_data_transfers method Update error message for file write timeout Implement cleanup mechanism in del method	shaketune/helpers/accelerometer.py
Update MeasurementsManager usage in various commands	Create new MeasurementsManager instance for each axis in axes_shaper_calibration Add wait_for_data_transfers calls in multiple command files	shaketune/commands/axes_shaper_calibration.py shaketune/commands/create_vibrations_profile.py shaketune/commands/axes_map_calibration.py shaketune/commands/compare_belts_responses.py shaketune/commands/excitate_axis_at_freq.py

---

Tips

• Trigger a new Sourcery review by commenting @sourcery-ai review on the pull request.
• Continue your discussion with Sourcery by replying directly to review comments.
• You can change your review settings at any time by accessing your dashboard:
  • Enable or disable the Sourcery-generated pull request summary or reviewer's guide;
  • Change the review language;
• You can always contact us if you have any questions or feedback.

[sourcery-ai]

Hey @Frix-x - I've reviewed your changes - here's some feedback:

Overall Comments:

• Consider making CHUNK_SIZE configurable or based on system resources for optimal performance across different environments.
• It would be beneficial to add unit tests for the new methods introduced, particularly _save_chunk, _reassemble_chunks, and _load_measurements_from_file.

Here's what I looked at during the review

• 🟡 General issues: 1 issue found
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment to tell me if it was helpful.}

[sourcery-ai]

suggestion (bug_risk): Improve error handling in _reassemble_chunks method

If an exception occurs during the reassembly process, the method catches the exception and prints a warning, but it doesn't clean up the temporary files or reset the state. This could lead to inconsistent state and leftover temporary files. Consider adding cleanup code in the exception handler.

def _reassemble_chunks(self, filename: Path):
    try:
        os.nice(19)
        # Existing reassembly code here
    except Exception as e:
        logging.warning(f"Error during chunk reassembly: {e}")
        self._cleanup_temp_files()
        self._reset_state()
        raise
    finally:
        self._cleanup_temp_files()

[sourcery-ai]

suggestion (code-quality): Replace if statement with if expression (assign-if-exp)

Suggested change

	if keep_last:
	self.measurements = [self.measurements[-1]]
	else:
	self.measurements = []
	self.measurements = [self.measurements[-1]] if keep_last else []

[sourcery-ai]

issue (code-quality): Extract code out into method (extract-method)