Filename extensions: Allow specifying wildcards #1584

franzpoeschel · 2024-01-08T10:46:02Z

Merge CMake 3.27.1 generates warning when using FetchContent #1557 first

Diff: https://github.com/franzpoeschel/openPMD-api/compare/refactor-extract-bufferedactions..topic-wildcard-extension

Usage idea:

> bin/openpmd-pipe --infile ../samples/git-sample/3d-bp4/example-3d-bp4_%T.%E --outfile out.%E                                                              
Opening data source                                                                                                                                         
Opening data sink                                                                                                                                           
Traceback (most recent call last):                                                                                                                          
  File "/home/franzpoeschel/build/nbuild/bin/openpmd-pipe", line 17, in <module>                                                                            
    pipe.main()                                                                                                                                             
  File "/home/franzpoeschel/build/nbuild/lib/python3.11/site-packages/openpmd_api/pipe/__main__.py", line 366, in main                                      
    run_pipe.run()                                                                                                                                          
  File "/home/franzpoeschel/build/nbuild/lib/python3.11/site-packages/openpmd_api/pipe/__main__.py", line 211, in run                                       
    outseries = io.Series(self.outfile, io.Access.create,                                                                                                   
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                   
openpmd_api.openpmd_api_cxx.ErrorWrongAPIUsage: Wrong API usage: Unable to automatically determine filename extension. Please specify in some way.          
[~Series] An error occurred: fileBased output can not be written with no iterations.


> bin/openpmd-pipe --infile ../samples/git-sample/3d-bp4/example-3d-bp4_%T.%E --outfile out.%E --outconfig 'backend = "hdf5"'
Opening data source
Opening data sink
Opened input and output
Iteration 550 contains 9 meshes:
         B
         E
         e_all_chargeDensity
         e_all_energyDensity
         e_all_particleMomentumComponent
         i_all_chargeDensity
         i_all_energyDensity
         i_all_particleMomentumComponent
         picongpu_idProvider

Iteration 550 contains 2 particle species:
[…]


> ls 
out.h5


> bin/openpmd-ls out.%E                                  
openPMD series: out
openPMD standard: 1.1.0
openPMD extensions: 0

data author: unknown
data created: 2021-07-15 13:51:27 +0000 
data backend: HDF5
generating machine: unknown
generating software: PIConGPU (version: 0.6.0-dev)
generating software dependencies: unknown

number of iterations: 2 (groupBased)
  all iterations: 550 600 

number of meshes: 0 # this is a bug

number of particle species: 2
  all particle species:
    e
    i



> bin/openpmd-pipe --infile ../samples/git-sample/3d-bp4/example-3d-bp4_%T.%E --outfile out.%E --outconfig 'backend = "adios2"'
Opening data source
Opening data sink
Opened input and output
Iteration 550 contains 9 meshes:
         B
         E
         e_all_chargeDensity
         e_all_energyDensity
         e_all_particleMomentumComponent
         i_all_chargeDensity
         i_all_energyDensity
         i_all_particleMomentumComponent
         picongpu_idProvider

Iteration 550 contains 2 particle species:



> ls 
out.bp
out.h5



> bin/openpmd-ls out.%E            
An error occurred while opening the specified openPMD series!
Read Error in frontend 
Object type:    File
Error type:     Other
Further description:    Found ambiguous filename extensions on disk: '.h5' and '.bp'.

Todo:

Testing
Documentation
(Maybe) Adapt for Linear and Append access types
Fix (probably unrelated) bug found in openpmd-pipe

src/Series.cpp

include/openPMD/Series.hpp

src/Series.cpp

+    Access at,
+    std::string const &options,
+    // Either an MPI_Comm or none, the template works for both options
+    MPI_Communicator &&...comm)


include/openPMD/IO/Format.hpp

src/Series.cpp

include/openPMD/Series.hpp

src/Series.cpp

include/openPMD/IO/ADIOS/ADIOS2File.hpp

+
+    void markActive(Writable *);
+
+    // bool isActive(std::string const & path);


test/CoreTest.cpp

@@ -1180,6 +1181,98 @@
        "../samples/optionsViaJsonPseudoFilebased%T.json"));
 }

+TEST_CASE("wildcard_extension", "[core]")


src/Series.cpp

include/openPMD/Series.hpp

ax3l · 2024-02-27T18:38:21Z

include/openPMD/Series.hpp

-     * @brief Construct a new Series
-     *
-     * @param filepath The backend will be determined by the filepath extension.
+     * @brief Construct a new Series.


Let's add a .. tip:: in the First Read overview:
https://openpmd-api.readthedocs.io/en/latest/usage/firstread.html#open

That explains %T and %E in 1-2 sentences.

.. tip:: Use the wildcards `%T` to match filename encoded iterations and `%E` to match supported extensions such as `.h5` and `.bp`.

ax3l

This is awesome. Please add the .. tip:: and then we ready.

todo: 1. temporarily initialize with dummy io handler 2. move the late initialization logic to IOHandler()

Co-authored-by: Franz Pöschel <franz.poeschel@gmail.com>

PrometheusPi · 2024-03-07T14:48:10Z

Thanks for implementing that file ending wild card @franzpoeschel

franzpoeschel added frontend: C++17 frontend: Python3 labels Jan 8, 2024

github-advanced-security bot found potential problems Jan 8, 2024

View reviewed changes

src/Series.cpp Fixed Show resolved Hide resolved

github-advanced-security bot found potential problems Jan 9, 2024

View reviewed changes

src/Series.cpp Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Jan 10, 2024

View reviewed changes

include/openPMD/Series.hpp Fixed Show fixed Hide fixed

include/openPMD/Series.hpp Fixed Show fixed Hide fixed

src/Series.cpp Fixed Show fixed Hide fixed

src/Series.cpp Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Jan 11, 2024

View reviewed changes

src/Series.cpp Fixed Show fixed Hide fixed

src/Series.cpp Fixed Show fixed Hide fixed

src/Series.cpp Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Jan 15, 2024

View reviewed changes

src/Series.cpp

Access at,

std::string const &options,

// Either an MPI_Comm or none, the template works for both options

MPI_Communicator &&...comm)

Check notice

Code scanning / CodeQL

Unused local variable Note

Variable comm is not used.

franzpoeschel commented Jan 15, 2024

View reviewed changes

franzpoeschel force-pushed the topic-wildcard-extension branch from 75949da to 4708900 Compare January 18, 2024 17:43

github-advanced-security bot found potential problems Jan 18, 2024

View reviewed changes

include/openPMD/IO/ADIOS/ADIOS2File.hpp Outdated

void markActive(Writable *);

// bool isActive(std::string const & path);

Check notice

Code scanning / CodeQL

Commented-out code Note

This comment appears to contain commented-out code.

github-advanced-security bot found potential problems Jan 19, 2024

View reviewed changes

franzpoeschel requested a review from ax3l January 19, 2024 16:41

franzpoeschel force-pushed the topic-wildcard-extension branch 2 times, most recently from 38edc16 to 7d3d398 Compare January 23, 2024 16:58

franzpoeschel commented Jan 30, 2024

View reviewed changes

src/Series.cpp Outdated Show resolved Hide resolved

franzpoeschel force-pushed the topic-wildcard-extension branch from 7d3d398 to 015afa1 Compare February 5, 2024 14:54

franzpoeschel force-pushed the topic-wildcard-extension branch from 015afa1 to be4118f Compare February 14, 2024 10:39

franzpoeschel mentioned this pull request Feb 23, 2024

Unify Random-Access API and Streaming API into Series::snapshots() #1592

Merged

6 tasks

ax3l self-assigned this Feb 26, 2024

ax3l reviewed Feb 27, 2024

View reviewed changes

include/openPMD/Series.hpp Show resolved Hide resolved

ax3l reviewed Feb 27, 2024

View reviewed changes

include/openPMD/Series.hpp Show resolved Hide resolved

ax3l reviewed Feb 27, 2024

View reviewed changes

ax3l approved these changes Feb 27, 2024

View reviewed changes

franzpoeschel added 7 commits February 28, 2024 15:04

Rough, but fundamentally functioning draft

dbf8e8e

Properly initialize

639b73a

Only read extensions from disk if the access type is read

d338622

Avoid parameter shadowing

d849b70

Temporary attempt to defer initialization

a8d678a

todo: 1. temporarily initialize with dummy io handler 2. move the late initialization logic to IOHandler()

Without Cleanup: todos from previous commit

bef9d03

Cleanup for previous, to be squashed

a731906

franzpoeschel and others added 18 commits February 28, 2024 15:04

More fine-grained deferral of initialization

2d05ede

Fixes that would theoretically allow deferring also CREATE mode

7691dec

Initialize early if possible, defer only when needed

51d144e

Access type related fixes

9c0750c

CI fixes for MSVC

3262f6f

Select ADIOS2 file ending more specifically

dead1b8

Little fix

bbf1e77

Slight fixes

0b4d97e

Testing

fd56c57

Add to examples

b149395

Better error messages

abb65ed

Little fix

42bdb13

Documentation

7e93d79

Cleanup

6d7068d

Fix test

68a0568

ambiguous

eea1b3b

Co-authored-by: Franz Pöschel <franz.poeschel@gmail.com>

Include Order

afcef03

Missing Includes

d19a50b

Co-authored-by: Franz Pöschel <franz.poeschel@gmail.com>

franzpoeschel force-pushed the topic-wildcard-extension branch from adc9ed8 to d19a50b Compare February 28, 2024 14:08

franzpoeschel enabled auto-merge (squash) February 28, 2024 14:51

First read/write

8212999

franzpoeschel force-pushed the topic-wildcard-extension branch from aa673d2 to 8212999 Compare February 28, 2024 14:52

franzpoeschel merged commit c2e0e83 into openPMD:dev Feb 28, 2024
31 checks passed

franzpoeschel mentioned this pull request Oct 14, 2024

JSON/TOML arguments for all Series constructor arguments #1685

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filename extensions: Allow specifying wildcards #1584

Filename extensions: Allow specifying wildcards #1584

franzpoeschel commented Jan 8, 2024 •

edited by ax3l

Loading

ax3l Feb 27, 2024

ax3l Feb 27, 2024

ax3l left a comment

PrometheusPi commented Mar 7, 2024


		void markActive(Writable *);

		// bool isActive(std::string const & path);

Filename extensions: Allow specifying wildcards #1584

Filename extensions: Allow specifying wildcards #1584

Conversation

franzpoeschel commented Jan 8, 2024 • edited by ax3l Loading

ax3l Feb 27, 2024

Choose a reason for hiding this comment

ax3l Feb 27, 2024

Choose a reason for hiding this comment

ax3l left a comment

Choose a reason for hiding this comment

PrometheusPi commented Mar 7, 2024

franzpoeschel commented Jan 8, 2024 •

edited by ax3l

Loading