Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(De)Serialization of data objects #103

Open
daphne-eu opened this issue Aug 12, 2021 · 0 comments
Open

(De)Serialization of data objects #103

daphne-eu opened this issue Aug 12, 2021 · 0 comments
Assignees
@pdamme

Comments

@daphne-eu
Copy link
Owner

In GitLab by @pdamme on Aug 12, 2021, 11:41

We need a way to serialize our in-memory data objects (DenseMatrix, CSRMatrix, Frame) to byte streams and to deserialize such byte streams to get back the data objects again.

This is required at multiple points, e.g.:

  • For the data transfer between nodes in the distributed run-time.
  • For complex IO handling, e.g. in connection to computational storage (even though that might also use its own (de)serialization routines)
  • For persistence using simple Daphne-specific binary file formats.
@aristotelis96 aristotelis96 mentioned this issue Oct 13, 2022
Open
@pdamme pdamme mentioned this issue Feb 7, 2023
Merged
pdamme pushed a commit that referenced this issue Jun 14, 2023
@aristotelis96 @psomas
Initial implementation of DaphneSerializer (#467)
478ae7c 
- Initial implementation of DaphneSerializer.
  - For Structure, DenseMatrix, CSRMatrix, and fundamental types (Frame still missing, see #545.
  - Variants for (de)serializing a data object as a whole or in chunks (for in-order and out-of-order deserialization).
- Updated WriteDaphne.h and ReadDaphne.h (reader/writer of DAPHNE binary data format) to use DaphneSerializer. 
- Updated distributed kernels with DaphneSerializer.
  - Distribute.h, Broadcast.h and DistributedCollect.h kernels updated with DaphneSerializer.
  - Updated gRPC and MPI distributed backends (e.g., workers).
  - Removed DAPHNE ProtoDataConverter.
- Test cases.
- API documentation in the source code.
- Fixed a small bug for MPI: MPI helper function, called by the distribute kernel, received the wrong rank.
- Contributes to #103, #465.

Co-authored-by: Stratos Psomadakis <774566+psomas@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pdamme pdamme

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pdamme @daphne-eu