You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
it('checks that metric carbon with aggregation == sum is properly spread over interpolated time points with custom upsampling resolution given',async()=>{
Copy file name to clipboardExpand all lines: src/if-run/builtins/time-sync/README.md
+28-5Lines changed: 28 additions & 5 deletions
Original file line number
Diff line number
Diff line change
@@ -9,9 +9,10 @@ Time sync standardizes the start time, end time and temporal resolution of all o
9
9
The following should be defined in the plugin initialization:
10
10
11
11
-`start-time`: global start time as ISO 8061 string
12
-
-`stop`: global end time as ISO 8061 string
12
+
-`end-time`: global end time as ISO 8061 string
13
13
-`interval`: temporal resolution in seconds
14
-
-`error-on-padding`: avoid zero/'zeroish' padding (if needed) and error out instead. `False` by defult.
14
+
-`allow-padding`: avoid zero/'zeroish' padding (if needed) and error out instead.
15
+
-`upsampling-resolution`: temporal resolution at which observations will be upsampled, in seconds. Defaults to 1.
15
16
16
17
#### Inputs:
17
18
@@ -28,7 +29,7 @@ A manifest file for a tree might contain many nodes each representing some diffe
28
29
We do this by implementing the following logic:
29
30
30
31
- Shift readings to nearest whole seconds
31
-
- Upsample the time series to a base resolution (1s)
32
+
- Upsample the time series to a base resolution.
32
33
- Resample to desired resolution by batching 1s entries
33
34
- Extrapolate or trim to ensure all time series share global start and end dates
34
35
@@ -39,6 +40,7 @@ The next section explains each stage in more detail.
39
40
##### Upsampling rules
40
41
41
42
A set of `inputs` is naturally a time series because all `observations` include a `timestamp` and a `duration`, measured in seconds.
43
+
42
44
For each `observation` in `inputs` we check whether the duration is greater than 1 second. If `duration` is greater than 1 second, we create N new `observation` objects, where N is equal to `duration`. This means we have an `observation` for every second between the initial timestamp and the end of the observation period. Each new object receives a timestamp incremented by one second.
Each `observation` actually includes many key-value pairs. The precise content of the `observation` is not known until runtime because it depends on which plugins have been included in the pipeline. Different values have to be treated differently when we upsample in time. The method we use to upsample depends on the `aggregation-method` defined for each key in `units.yml`.
@@ -151,12 +154,31 @@ For example, for `startTime = 2023-12-12T00:00:00.000Z` and `endTime = 2023-12-1
151
154
]
152
155
```
153
156
154
-
Note that when `error-on-padding` is `true` no padding is performed and the plugin will error out instead.
157
+
Note that when `allow-padding` is `true` no padding is performed and the plugin will error out instead.
155
158
156
159
##### Resampling rules
157
160
158
161
Now we have synchronized, continuous, high resolution time series data, we can resample. To achieve this, we use `interval`, which sets the global temporal resolution for the final, processed time series. `interval` is expressed in units of seconds, which means we can simply batch `observations` together in groups of size `interval`. For each value in each object we either sum, average or copy the values into one single summary object representing each time bucket of size `interval` depending on their `aggregation-method` defined in `aggregation` section in the manifest file. The returned array is the final, synchronized time series at the desired temporal resolution.
159
162
163
+
#### Setting a custom upsampling resolution
164
+
165
+
The model defaults to upsampling observations to a 1-second resolution. However, this can lead to unnecessary effort, as upsampling at a coarser resolution is often sufficient, provided it doesn't interfere with the accuracy of resampling. To optimize performance, we can set the `upsampling-resolution` parameter in the configuration to a more appropriate value. The chosen value should meet the following criteria :
166
+
167
+
- It should evenly divide all observation durations within the dataset.
168
+
- It must be a divisor of the `interval`.
169
+
- It should also divide any gaps between observations, as well as the start and end paddings.
170
+
171
+
For example, for `interval = 10` and this time-series
0 commit comments