Floating Lidar File Format Standardization Development - 2024

[stephenholleran]

19-Sep-2024 First Meeting

Attendance:
Stephen Holleran (BrightWind), Jon Upton (Shell), Ryan Scott (RWE), Gibson Kersting (RWE), Jason Fields (RES), Amit Bohara (Altosphere), Anselme Troiville (Green Rebel), Hanna Torrens-Spence (BrightWind), Jose Miguel Garro (Eolos), Felix Kelberlau (Fugro), Kenneth John (RINA), Maximillian Schuberth (Recons), Richard Khaira-Creswell (Axys), David McCracken (BP), Philip Hargreaves (TGS) [15]

Minutes:

1. Quick intros.
2. Went through the presentation https://docs.google.com/presentation/d/1w-VNQEwlnxh1dC20qUz2uNqeTybNoFB18HVZ2esUNgg/edit?usp=sharing
3. A discussion followed.
4. Everyone on the call was committed to develop this standard.

Actions

• Prepare for next meeting in 2 weeks. Objective to pull from the data model the metadata we want to capture.

[stephenholleran]

17-Oct-2024 2nd Meeting

Attendance:
Stephen Holleran (BrightWind), Hanna Torrens-Spence (BrightWind), Jose Miguel Garro (Eolos), Ryan Scott (RWE), Jon Upton (Shell), Anselme Troiville (Green Rebel), Philip Hargreaves (TGS), Felix Kelberlau (Fugro) (temporarily), Amit Bohara (Altosphere) [9].

Minutes:

1. Quick intros.
2. We had to skip the call on the 3rd Oct due to the ACP conf.
3. I am thinking about this standard file format in 3 parts:
   1. File name and extension
   2. Header
   3. Column names
4. I think we should start with the header first. The below example is following the WRA Data Model from measurement_location to logger_main_config.

   name: Site X - FLS 01
   latitude_ddeg: 53.5017
   longitude_ddeg: -19.0966
   measurement_station_type: floating_lidar
   logger_main_config:
     - logger_oem: ZX Lidars
       logger_model_name: 300M
       logger_serial_number: 1234
       logger_firmware_version: v12345
       logger_id: 1234
       logger_name: FLS1
       offset_from_utc_hrs: 0.0
       averaging_period_minutes: 10
       timestamp_is_end_of_period: True
       lidar_config:
           flow_corrections_applied: false
           logger_stated_device_datum_plane_height_m: 2
           logger_stated_device_orientation_deg: 3
     - logger_oem: Nortek
       logger_model_name: Signature250
       logger_serial_number: 1234567
       offset_from_utc_hrs: 0.0
       averaging_period_minutes: 10

   1. Very minimal is actually required for the data model. The minimum would be name, latitude_ddeg, longitude_ddeg and measurement_station_type.
   2. If logger_main_config is provided, the minimal from that would be logger_oem and logger_serial_number.
   3. Leaving out the date_from and date_to as this config is to cover the config used to produce the timeseries data in the file. Yes, we all agree.
   4. Using YAML format instead of JSON to make it more readable and simple. We could just use the WRA Data Model JSON but I think it would be too long and complicated? JSON equivalent: Best to stick with JSON as it is one less thing to explain. And we are all used to JSON and the data model is already in JSON.

   {
       "name": "Site X - FLS 01",
       "latitude_ddeg": 53.5017,
       "longitude_ddeg": -19.0966,
       "measurement_station_type_id": "floating_lidar",
       "logger_main_config": 
       [{
   	    "logger_oem": "ZX Lidars",
   	    "logger_model_name": "300M",
   	    "logger_serial_number": "1234",
   	    "logger_firmware_version": "v12345",
   	    "logger_id": "1234",
   	    "logger_name": "FLS1",
   	    "offset_from_utc_hrs": 0.0,
   	    "averaging_period_minutes": 10,
   	    "timestamp_is_end_of_period": true,
   	    "lidar_config":
   	    {
   		    "flow_corrections_applied": false,
   		    "logger_stated_device_datum_plane_height_m": 2,
   		    "logger_stated_device_orientation_deg": 3
   	    }
       },{
   	    "logger_oem": "Nortek",
   	    "logger_model_name": "Signature250",
   	    "logger_serial_number": "1234567",
   	    "offset_from_utc_hrs": 0.0,
   	    "averaging_period_minutes": 10,
       }]
   }

   9. Do we need a section to capture 'file format version', 'software processing version', 'firmware version', ???? Yes. What are they:
      1. 'file format version' is the version of this file we are developing now.
      2. 'software processing version' could be the version of the floating lidar OEM pipeline to produce the file.
      3. 'buoy version' Do have buoy version numbers but probably not that important to capture. But could be used to verify the type verification. No harm in including as all these may be optional.
      4. Maybe these should be compulsory to set a standard that consumers of the data will know that this info is in there and any new entrants to the market would be encourage to have good processes in place for example, that they have good processes to track their pipeline process and can trace which version of the ZPH2CSV converter was used.
      5. We'll come back to this again in the next call.
   10. I think the measurement_point section of the data model can be covered by a structured column name of the timeseries data? Yes we think this would be a good thing. It is not deviating from the data model as such, we are just not including that part of the data model in the header.
5. Column names: these could contain the minimal information so we can avoid having a very large header with measurement point information.
   1. Format: <sensor_type>_<serial_number>_<measurement_type>_<height>_<measurement_units>_<statistic_type>
      Example: lidar_1234_wind_speed_110_m/s_avg
   2. Continue this discussion on the next call.

Actions

• Think about the extra section to capture versions.
• As well as prepare for next call to discuss column name format and file name.

[stephenholleran]

31-Oct-2024, 3rd Meeting

Attendance:
Stephen Holleran (BrightWind), Jon Upton (Shell), Anselme Troiville (Green Rebel), Hanna Torrens-Spence (BrightWind), Jose Miguel Garro (Eolos), Ryan Scott (RWE), Felix Kelberlau (Fugro), Gibson Kersting (RWE) [8]

Minutes:

1. Quick intros (if needed).
2. Recap on previous call.
3. This isn't really a "daily" file format but a "data" file format. Be conscious of this when finalising the name for this.
4. Header section to capture OEM buoy specific details:
   1. Options:
      1. 'format_version' is the version of this file we are developing now. Alternatives: 'file_format_version', 'file_system_version'? Yes, we agree that this would be good to include. 'format_version' is good to use.
      2. 'software_version' could be the version of the floating lidar OEM pipeline to produce the file. Software version is too broad to know what it refers to. Also, narrowing down the usefulness of this format. Very difficult to version control the pipeline that generates this file. Not easy to maintain and so we decided not to include something like this.
      3. 'buoy_version' Type verification is a certain type of buoy with a certain type of lidar that gets type certification. 'fls_type' would be more appropriate. The serial number can allow the user to work out what fls type this is though. Therefore, not that useful so no need to include it.
      4. 'buoy_serial'? Yes, do include. Alt options: 'fls_serial_number', 'station_serial_number'. 'station_serial_number' is good to go with so there could be a buoy with no lidar (a basic metocean station). This 'station_serial_number' could also be put into the WRA Data Model under the 'measurement_location'. EDIT: Actually it can't as there are no date_from and date_to at this level to allow it to change over time.
      5. 'notes' could be useful to include. Notes are actually included at the measurement location level so any notes could be used there instead. Include it in the example.
   2. Should these be required? Yes. Good to include and is basic info so should be relatively easy to implement.
5. Column names:
   1. Suggested format: <sensor_type>_<serial_number>_<measurement_type>_<height_m>_<measurement_units>_<statistic_type>
      Example: lidar_1234_wind_speed_110_m/s_avg
   2. Where the values follow the data model type and enums.
   3. This can get very long especially with long serial numbers. When working with this data in Excel, Python or MATLAB it could be cumbersome. Also, when a lidar is swapped out the file would contain a whole load of different column names than the previous file that would all need to get mapped to the appropriate measurement point names. The only way to have shorter column names would be to include the 'measurement_point' section from the data model included in the header. EDIT: However, we also think a standard column name is worth pursuing so continued to develop it.
   4. Separator is an underscore however element names also use underscore. This can cause a problem with splitting out the data into individual components. Can we use a different separator or should we just handle the single underscores?
      1. Can't use a hyphen (-) as it can get mixed up with a negative height and dimensionless units.
      2. Can't use a period (.) as it can get mixed up with decimal heights and/or lat and long.
      3. Can't use a forward-slash (/) as it can get mixed up with units.
      4. A semicolon (;) can also be used as a delimiter so we should avoid that however, if the delimiter for the timeseries data is a comma (,) then we could use a semicolon.
      5. A colon (:) is an option.
      6. A double underscore is also an option. This is the preferred option instead of the single underscore.
      7. Examples:
         1. lidar:1234:wind_speed:110:m/s:avg
         2. lidar;1234;wind_speed;110;m/s;avg
         3. lidar__1234__wind_speed__110__m/s__avg
   5. Are all parts of the column name required?
      1. serial_number is probably the only one that may not be known. In the data model 'null' is allowed.
      2. They should all be required.

Actions

• [ ]

[stephenholleran]

14-Nov-2024, 4th Meeting

Attendance:
Stephen Holleran (BrightWind), Jon Upton (Shell), Gibson Kersting (RWE), Hanna Torrens-Spence (BrightWind), Jose Miguel Garro (Eolos), Michael Wiegele (RPS), Niko Simisiroglou (RWE), Philip Hargreaves (TGS) [8]

Minutes:

1. Quick intros (if needed).
2. Recap on previous call.

3. {
    "format_version": "1.0.0",
    "station_serial_number": "FLS 01",
    "name": "Site X",
    "latitude_ddeg": 53.5017,
    "longitude_ddeg": -19.0966,
    "measurement_station_type_id": "floating_lidar",
    "notes": "Some additional notes specific for this deployment.",
    "logger_main_config":
    [{
    	"logger_oem": "ZX Lidars",
    	"logger_model_name": "300M",
    	"logger_serial_number": "1234",
    	"logger_firmware_version": "v12345",
    	"logger_id": "1234",
    	"logger_name": "FLS1",
    	"offset_from_utc_hrs": 0.0,
    	"averaging_period_minutes": 10,
    	"timestamp_is_end_of_period": true,
    	"lidar_config":
    	{
    		"flow_corrections_applied": false,
    		"logger_stated_device_datum_plane_height_m": 2,
    		"logger_stated_device_orientation_deg": 3
    	}
    }, {
    	"logger_oem": "Nortek",
    	"logger_model_name": "Signature250",
    	"logger_serial_number": "1234567",
    	"offset_from_utc_hrs": 0.0,
    	"averaging_period_minutes": 10
    }]
   }

4. Column names:
   1. Suggested format: <sensor_type>__<serial_number>__<measurement_type>__<height_m>__<measurement_units>__<statistic_type>
      Examples:
      1. lidar__1234__wind_speed__110__m/s__avg
      2. lidar__1234__wind_speed__110__m/s__max
      3. adcp__4321__water_speed__-23__m/s__avg
      4. thermometer__null__water_temperature__-1__deg_C__avg
   2. Quality flags should be considered as a lidar could provide temperature and pressure values and a quality data column could relate to either the temperature or pressure as they would both be at the same height. Similar if it is from Airmar weather station sensor where it can output several measurement types where each could have a quality flag. The column name structure we have come up with would not be unique for each of the pressure, temperature, wind speed, wind direction measurements. lidar__1234__quality__1__%__avg is this for the temperature or pressure from the lidar which are both at 1 m height?
      1. lidar__1234__quality__1__%__avg__temperature tag on the end the measurement type the flag/quality column relates too?
      2. Maybe we could think about it like how Windographer flags data in that they duplicate the column name and tag on flag at the end. So each column name in the timeseries can have it's own flag e.g. thermometer__null__air_temperature__1__deg_C__avg__flag.
      3. Instead of tagging flag on the end, a possible solution is to allow a notes on the end. This is effectively a free field to allow the OEM to include whatever they want: <sensor_type>__<serial_number>__<measurement_type>__<height_m>__<measurement_units>__<statistic_type>__<notes>. This gives the OEM the option to use lidar__1234__quality__110__%__avg for the wind speed, vertical wind speed and wind direction values or use lidar__1234__air_temperature__1__deg_C__avg__flag for other sensors.
      4. It would be good to dwell on this and make a decision when more OEM representation is on the call.
   3. What about calculated column names? For example air density can be calculated by two different sensors.
      1. We could have in the sensor enum a calc option to indicate that this column of data is calculated from multiple other sensors or use other or null which are already options in the data model? Using calc might also solve another issue we have for soiling loss calculation in solar. Examples:
         1. calc__null__air_density__2__kg/m^3_avg
         2. other__null__air_density__2__kg/m^3_avg
         3. null__null__air_density__2__kg/m^3_avg
      2. Agreed to go with adding calc to the sensor_type enum.
      3. If there are 2 air density calculations derived at the same height from two different temperature and pressure sensors, how do we make them unique? Concatenate the serial numbers or just use a slightly different height?
         1. calc__temp1234_press4321__air_density__2__kg/m^3__avg
         2. calc__null__air_density__2.0__kg/m^3__avg
         3. calc__null__air_density__2.1__kg/m^3__avg
         4. Another option, if we allow 'notes' on the end as discussed above, could be calc__null__air_density__2.0__kg/m^3__avg__A and calc__null__air_density__2.0__kg/m^3__avg__B
         5. This is more of a choice for the FLS OEM as to which option they want to go with. They could use the <serial_number> to provide a derived identifier either instead of using the area.
   4. What about a battery voltage column names where we may have difficulty to distinguish which battery is associated with which logger/device/system?
      1. Could we use the serial number of the device that is linked to that battery e.g. use the lidar serial number for the lidar battery?
         1. voltmeter__1234__voltage__0__V__avg
      2. Again, maybe the field on the end can help distinguish which device/system this battery voltage relates too. Or again the <serial_number> could be used to distinguish.
      3. Only briefly touched on this in the call. Continue on next call.
      4. Also consider how we measure the fuel by possibly adding a fuel_level option in the measurement_type enum. This would also require a fuel_gauge in the sensor_type enum.
5. Meeting change date; 28th Nov is our Christmas party and Thanksgiving. Push it to the 5th Dec. and then break for Christmas?
   1. As we didn't get through all the issues with the column name we will pull forward the 28th to the 21st Nov and also have one on the 5th Dec to try and get 2 more in before we break for Christmas.

Actions

• [ ]

[stephenholleran]

21-Nov-2024, 5th Meeting

Attendance:
Stephen Holleran (BrightWind), Salma Yahiaoui (Akrocean), Anselme Troiville (Green Rebel), Jose Miguel Garro (Eolos), Ryan Scott (RWE), Matthew Linham (BP), Gibson Kersting (RWE) [7]

Minutes:

1. Quick intros (if needed).
2. Recap on previous call.
3. Column names:
   1. How to include quality flags as discussed in the previous meeting?
      1. The suggestion is to include 'notes' as an optional item at the end of the filename. Therefore a flag label could be added for each measurement column name. E.g. lidar__1234__air_temperature__1__deg_C__avg and lidar__1234__air_temperature__1__deg_C__avg__flag.
      2. <sensor_type>__<serial_number>__<measurement_type>__<height_m>__<measurement_units>__<statistic_type>__<notes>
      3. This 'notes' can also be used as an option for when there are duplication of measurements at the same height but for different systems, e.g. distinguish between battery for lidar or battery for navigation lights.
      4. 'Notes' are all over the data model already as optional fields so probably reasonable to give this option here too. It could also cover any unforeseen scenarios.
      5. We think adding the 'notes' is a good option to include. Another example for its use came up in the discussions was to use it to distinguish between different 'wave_period' that are calculated from the same sensor but using a different algorithm. The column name without the 'notes' optionality would be a duplicate so the free field text of the notes could be used to distinguish between the methods of calculation.
   2. Add fuel_level option to the measurement_type enum and fuel_gauge in the sensor_type enum? Yes, do this.
   3. Should we standardise the timestamp column name?
      1. Options could be:
      2. Timestamp
      3. timestamp
      4. DateTime
      5. date_time
      6. Yes. We will go with 'timestamp', all lower case.
      7. The timestamp format should be ISO 8601 with the 'T' required, similar to the WRA Data Model, but with no time zone. E.g. 2024-12-12T13:40:00. Agreed.
4. File Name and Extension:
   1. For file extension I would keep this simple and use a file extension of .csv even though it is technically not a csv? It could be .txt, .dat or ??? Yes, we should standardise the file extension. We decided on '.csv', lower case.

Actions

• [ ]

[felixkelberlau]

Thanks to all for working on this yesterday. The meeting time makes it difficult for me to always join the calls.

I think for the column header names the most important parameter should come first because in many situations the long names will be displayed shortened and in alphabetical order. Thus, I am wondering if the order with <sensor_type> first is the best. Is <measurement_type> first maybe advantageous?

Thinking of how measurement parameters are typically sorted in reporting, measurement plans and tender documents, it appears to me that wind data, current data, meteorological parameters and so on are grouped together. So, I think that <measurement_type> in the first position will lead to more intuitive sorting. It will also keep identical/redundant measurements closer together.

When working with vertical profile data, I find it very convenient when the same type of measurements from each height are grouped together instead of being spread out over the file. Thus, I would keep <measurement_type> and <statistic_type> together. In this way, we will get one block of all mean wind speeds and so on. I know that this is different from ZPH and STA output but would you agree that it is more practical when working with data?

I also think that it is customary to have the <meaurement_units> at the end of a parameter name.

So, to summarize I am suggesting the following order:
<measurement_type>__<statistic_type>__<height_m>__<sensor_type>__<serial_number>__<measurement_units>__<notes>

[stephenholleran]

@felixkelberlau thanks very much for this and sorry for not replying sooner.

I think these are great suggestions. I will include them for discussion on today's call.

[stephenholleran]

@felixkelberlau
Thanks again for your suggestion. I have been thinking about it in prep for todays call and I am wondering what your thoughts are on swapping the height and statistic type around?

So instead of:
wind_speed__avg__110__lidar__1234__m/s
wind_speed__avg__120__lidar__1234__m/s
wind_speed__sd__110__lidar__1234__m/s
wind_speed__sd__120__lidar__1234__m/s

It would be:
wind_speed__110__avg__lidar__1234__m/s
wind_speed__120__avg__lidar__1234__m/s
wind_speed__110__sd__lidar__1234__m/s
wind_speed__120__sd__lidar__1234__m/s

You still get your requirement to group all the mean wind speeds together but I am thinking that the height is very useful info to have up the pecking order.

[stephenholleran]

@felixkelberlau sorry, I was completely wrong in what I wrote above. The 'avg' wouldn't be grouped together.

[stephenholleran]

5-Dec-2024, 6th Meeting

Attendance:
Stephen Holleran (BrightWind), Anselme Troiville (Green Rebel), Jose Miguel Garro (Eolos), Salma Yahiaoui (Akrocean) [4]

Minutes:

1. Quick intros (if needed).
2. Recap on previous call.
3. Column names:
   1. Instead of this structure <sensor_type>__<serial_number>__<measurement_type>__<height_m>__<measurement_units>__<statistic_type>__<notes> Felix is suggesting <measurement_type>__<statistic_type>__<height_m>__<sensor_type>__<serial_number>__<measurement_units>__<notes>? Examples:
      1. wind_speed__avg__110__lidar__1234__m/s
      2. wind_speed__avg__120__lidar__1234__m/s
      3. wind_speed__sd__110__lidar__1234__m/s
      4. wind_speed__sd__120__lidar__1234__m/s
      5. air_density__avg__2__calc__null__kg/m^3
      6. air_temperature__avg__1__thermometer__null__deg_C
   2. Or we could swap the height and statistic_type around to bring up the height.
      1. wind_speed__110__avg__lidar__1234__m/s
      2. wind_speed__110__sd__lidar__1234__m/s
      3. wind_speed__120__avg__lidar__1234__m/s
      4. wind_speed__120__sd__lidar__1234__m/s
   3. Changing the ordering would split out the sensors from what measured them. For example, measurements coming from a lidar are all grouped together and ADCP are grouped together. However, we do understand where Felix is coming from and these will mostly be read by machines so it doesn't really matter what the order is. But it does help when doing some ad-hoc analysis in Excel or something so we are good to go with measurement_type first. We are ok with the statistic_type before the height so this wind_speed__avg__120__lidar__1234__m/s.
4. File Name and Extension:
   1. We decided that the file extension should be '.csv', lower case.
   2. Do we standardise the file name? It would be good to propose a best practice at least.
   3. Options could be:
      1. <station_serial_number>__<date_from>__<date_to>.csv
      2. <oem_name>__<station_serial_number>__<date_from>__<date_to>.csv
      3. <oem_name>__<station_serial_number>__<date_from>__<date_to>__<averaging_period_minutes>.csv
      4. <oem_name>__<station_serial_number>__<date_from>__<date_to>__<notes>.csv
      5. <oem_name>__<station_name>__<station_serial_number>__<date_from>__<date_to>__<notes>.csv
      6. <oem_name>__<station_name>__<station_serial_number>__<client_name>__<date_from>__<date_to>__<notes>.csv
      7. Fugro__Site X__FLS 01__2024-01-01_00_00_00__2024-01-31_23_50_00__LidarData.csv
      8. Eolos__Site X__FLS 01__2024-01-01_00_00_00__2024-01-01_23_50_00__10min.csv
      9. Do we include the client name? Client names can change and it really doesn't give the consumer any useful info to identify what measurement station this relates too. Don't want a very long name as, along with the folder path, we could exceed the Windows max number of characters. In saying that, it is mostly the folder path that causes the issues and not the filename and so we then think that option 'e' above is the one to go with <oem_name>__<station_name>__<station_serial_number>__<date_from>__<date_to>__<notes>.csv. We don't need the client as there may be more than one client and it has no impact on the data.
   4. Where date_from and date_to are the first and last timestamps found in the file? Yes
      1. Include the time or just the date? There may be multiple files that contain chunks of the same day and therefore would be a duplicate filename. Note; can't use ':' in a file name => 2024-01-01_235000 or 2024-01-01_23_50_00 or 2024-01-01_2350. We should include the time component to avoid duplicates. Other options are '20240101_235000', `2024-01-01_23-50-00'. We chose to go with '2024-01-01_23_50_00'.
      2. Should the date_to be optional? Maybe not as a daily file for the first of the month would have the same filename as a monthly file for that same month. No, we need a date_to.
      3. Could include 'to' between dates, e.g. 2024-01-01__to__2024-01-31, to make it more obvious what the dates mean? No, this is not necessary.
5. We will need to trial this in anger. How shall we go about doing that?
   2. SH to create a folder in the repo to capture all of the above decisions.
   3. Within this create a readme and a guidelines of use which would capture the best practice use of this.
   4. As well as a sample file, maybe a JSON Schema for the header too.
   5. All OEMs to send me (Stephen) a sample file as it exists at the moment. I will then pull out as many different column names and put them into this new format. This can then be used as a guide for the OEMs to try it out for themselves.
6. How is this to be packaged?
   1. Be part of the WRA Data Model repo where it lives in a dedicated folder or it's own separate repo? Can answer this at a later date.
7. Next meeting, 16th or 23rd January? Go with the 23rd to give me more time. Agenda will just be reviewing the sample file I created.

Actions

• [ ]

[stephenholleran]

23-Jan-2025, 7th Meeting

Attendance:
Stephen Holleran (BrightWind), Anselme Troiville (Green Rebel), Gibson Kersting (RWE), Jon Upton (Shell), Niko Simisiroglou (RWE), Léa Rolland (Akrocean), Scott Wylie (GreenRebel) [7]

Minutes:

1. Quick intros (if needed).
2. Recap on previous call.
3. SH putting together a sample file and creating the documentation. We will review this PR: Floating lidar file format #271
   1. Folder in the repo to capture all of the above discussions. OK to have a subfolder in the WRA Data Model repo and its name is ok.
   2. In the Intro, list the companies involved.
   3. Dates in file name, could we use 'T' instead of the underscore? Forgot that the colon (:) can't be used in the file name as Windows doesn't allow : and so therefore it can't follow ISO 8601 anyway. However it was felt that the T does offer a distinction between the date and time parts and is not confused with the underscore separator of the individual time parts therefore it would be good to use. E.g. 2024-01-01T00_00_00.
   4. Put explanation of the double underscore for the file naming.
   5. Outline that users should use Window compatible characters in the file name.
   6. Explain in the header part more where this JSON Schema came from i.e. from the Data Model with unnecessary parts deleted.
   7. In the header section we would probably get away with not explaining all terms as the users will be familiar with JSON Schema and we can link to the WRA Data Model docs.
   8. In the notes variable, specify not to use double underscores.
   9. Got through most of the example column names. Need to revisit 'packets in avg' counter or quality columns.

Actions

• Continue working on a first draft of the ReadMe.
• Link up with floating lidar OEMs to come up with unusual column names.

[stephenholleran]

06-Feb-2025, 8th Meeting

Attendance:
Stephen Holleran (BrightWind), Anselme Troiville (Green Rebel), Gibson Kersting (RWE), Salma Yahiaoui (Akrocean), Ryan Scott (RWE), Mona Senelier (Vaisala) [6]

Minutes:

1. Quick intros (if needed).
2. Recap on previous call.
3. SH putting together a sample file and creating the documentation. We will review this PR: Floating lidar file format #271
   1. Felix via email: "height_m: If not discussed before, I would like to encourage a discussion on whether the entries should have a leading zero, e.g., “80” or rather “080”. Without a leading zero alphabetical ordering ruins the order in ascending heights."
      1. Other examples e.g. -5 to -005, -5.5 to -005.5, 32 to 032.
      2. What would happen if we measure over 1,000 m? Could be possible, therefore there would be 4 digits required.
      3. We don't think this can be a requirement however what we can do is write a recommendation that using leading zeros will organise the column names better alphabetically. Give example of 4 digits.
   2. Felix via email: "measurement_units: Taking Matlab as an example, only letters, digits, and underscores are allowed as variable names. There are workarounds but it might be useful to accept this limitation and avoid “/” and “%” in our definition. So, is “ms” or “mps” maybe better than “m/s”?"
      1. SH; these units are an enum in the main data model so would be more difficult to change. We also try to keep to SI units as much as possible where "m/s" is the SI unit.
      2. Decided to keep them as is.
   3. Sensor can be a null error in the read me.
   4. voltage__avg__2__voltmeter__null__V__fog horn is ok to use to distinguish between voltage measured from a fog horn verses an aviation light or communications system.
   5. counter__count__120__lidar__1234__null__packets in avg and quality__quality__null__lidar__1234__%__proportion of packets with rain are ok examples of using the notes section to capture the 'packets' measured by ZX lidars.
   6. Anselme will come up with an example for an Aquadopp (a horizontal ADCP).
   7. Didn't get to discuss pitch, roll, yaw or heading.

Actions

• Continue working on a first draft of the ReadMe.
• Anselme will come up with an example for an Aquadopp (a horizontal ADCP).

[stephenholleran]

20-Feb-2025, 9th Meeting

Attendance:
Stephen Holleran (BrightWind), Jose Miguel Garro (Eolos), Anselme Troiville (Green Rebel), Scott Wylie (GreenRebel), Ryan Scott (RWE), Gibson Kersting (RWE), Jon Upton (Shell) [7]

Minutes:

1. Quick intros (if needed).
2. Recap on previous call.
3. SH putting together a sample file and creating the documentation. We will review this PR: Floating lidar file format #271. Mostly working on the Column Names.
   1. Anselme has come up with an example for an Aquadopp (a horizontal ADCP). An Aquadopp is similar to a scanning or nacelle mounted lidar in wind. The data model can't yet handle scanning lidars and so the following is an example of how this floating lidar file format may be able to handle this Aquadopp data.
      1. water_speed__avg__-001__adcp__1234__m/s__meas_distance_3_m
      2. water_speed__avg__-001__adcp__1234__m/s__meas_distance_4_m
      3. water_speed__avg__-001__adcp__1234__m/s__meas_distance_5_m
      4. water_speed__avg__-001__adcp__1234__m/s__meas_distance_6_m
      5. water_speed__avg__-001__adcp__1234__m/s__meas_distance_7_m
      6. This is a good temporary solution until the data model gets better and can handle scanning lidars or side mounted ADCPs. Strongly recommend OEMs to use this format.
   2. What about pitch, roll, yaw, heading or bearing?
      1. Sensor type? The sensor could be one of many e.g. ADCP, IMU, so the pitch, roll is of the device. The CF Conventions has "platform_roll" and it says it covers the instruments. However, in floating lidar terms the "platform" would generally mean the buoy itself. So a bottom or side mounted ADCP's roll would be different to the buoy's roll. It would be best to just use "roll".
      2. Measurement type? Need to add 'roll', 'pitch', 'yaw', 'heading'.
      3. Units? Have deg already.
   3. From Anselme; what about 'biofouling sensor'? Metal plates mounted on the buoy. Sent off to a lab to measure. Therefore it is a once off dataset. Not a timeseries dataset so therefore no need to capture.
   4. Micro-wave profiler? Essentially a temperature profiler in the atmosphere.
      1. Sensor type? Add ``microwave_temperature_profiler` (MTP) https://airbornescience.nasa.gov/instrument/Microwave_Temperature_Profiler
   5. Bird/bat monitoring (data would be animal count)? Jon to try and find out about the data.
   6. Hydro-phonic sensor (data would be sound files, e.g. from marine animals)? These are sound files and have their own format so not really appropriate for this file format.
   7. Turbidity sensor / water quality (dissolved oxygen)?
      1. Sensor type? Add nephelometer https://en.wikipedia.org/wiki/Nephelometer
      2. Measurement type? Add turbidity, refer to CF Conventions but drop the 'sea_water' as floating lidar can be placed in lakes and rivers.
      3. Units? Dimensionless
   8. Visibility
      1. Sensor type? visibility_sensor, do a bit more research
      2. Measurement type? CF Conventions has visibility_in_air. Stick with the CF Conventions in this case as turbidity covers visibility in water.
      3. Units? m, so we have this already.
   9. "Wave Spectra" talk about on next call.

Actions

• Continue working on a first draft of the ReadMe.

[felixkelberlau]

I have a comment regarding ii.)
pitch, roll, yaw (, heading, bearing) require careful definitions.

The coordinate system (E,N,D, relative to true north or platform axes) and counting order need to be defined (https://en.wikipedia.org/wiki/Euler_angles).

In addition, it must be clear how ten minute statistics are calculated. The arithmetic mean of the pitch or roll angle for an FLS will usually be close to zero and is an indicator for static tilt of the platform. The mean of their absolute value might be already more interesting as it describes the average inclination in each direction. Maybe good to also include the variance of the roll/pitch angles to show what proportion is static and dynamic? Especially for axis-symmetric systems the mean tilt angle (as the quadratic sum of roll and pitch before averaging) and its direction can be the most significant parameters of all.

I am not aware of any widely-accepted guidance on this and I believe that without definitions, each OEM will provide different orientation data.