Skip to content

feat: Add sync between TimeSeries, Video, Audio #7376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: develop
Choose a base branch
from

Conversation

cloudmark
Copy link

@cloudmark cloudmark commented Apr 15, 2025

… components

Fixes #4892: How can we label video and time series signals while watching them together?
Fixes #5359: Video/audio sync doesn't work

  • Add sync attribute to TimeSeries tag model to enable group-based synchronization
  • Compose TimeSeries with SyncableMixin for event broadcasting and handling
  • Implement sync event handlers for seek, play, and pause events
  • Add smooth view updates during audio/video playback
  • Prevent event flooding by suppressing sync events during playback
  • Maintain view window size during playback
  • Map media timeline to TimeSeries range (start/end points)

This enables synchronized playback and seeking between Video, Audio, and TimeSeries components when they share the same sync group. Multiple TimeSeries components can also sync with each other, allowing for coordinated view updates across different time series data. For example:

<View>
  <Video name="video" value="$video" sync="group_a"/>
  <Audio name="audio" value="$audio" sync="group_a"/>
  <TimeSeries name="accel_timeseries" value="$accel_data" sync="group_a">
    <Channel column="accel_x" strokeColor="#FF0000"/>
    <Channel column="accel_y" strokeColor="#00FF00"/>
  </TimeSeries>
  <TimeSeries name="gyro_timeseries" value="$gyro_data" sync="group_a">
    <Channel column="gyro_x" strokeColor="#0000FF"/>
    <Channel column="gyro_y" strokeColor="#FF00FF"/>
  </TimeSeries>
  
  <Choices name="choice" toName="video">
    <Choice value="synced"/>
  </Choices>
</View>

<!-- {
  "video": "/static/samples/opossum_snow.mp4",
  "audio": "/static/samples/game.wav",
  "accel_data": "/samples/time-series.csv?time=time&values=accel_x%2Caccel_y&sep=%2C&tf=%25Y-%25m-%25d+%25H%3A%25M%3A%25S",
  "gyro_data": "/samples/time-series.csv?time=time&values=gyro_x%2Cgyro_y&sep=%2C&tf=%25Y-%25m-%25d+%25H%3A%25M%3A%25S"
}
-->

… components

Fixes #4892: How can we label video and time series signals while watching them together?
Fixes HumanSignal#5359: Video/audio sync doesn't work

- Add sync attribute to TimeSeries tag model to enable group-based synchronization
- Compose TimeSeries with SyncableMixin for event broadcasting and handling
- Implement sync event handlers for seek, play, and pause events
- Add smooth view updates during audio/video playback
- Prevent event flooding by suppressing sync events during playback
- Maintain view window size during playback
- Map media timeline to TimeSeries range (start/end points)

This enables synchronized playback and seeking between Video, Audio, and TimeSeries
components when they share the same sync group. Multiple TimeSeries components can
also sync with each other, allowing for coordinated view updates across different
time series data. For example:

```
<View>
  <Video name="video" value="$video" sync="group_a"/>
  <AudioPlus name="audio" value="$audio" sync="group_a"/>
  <TimeSeries name="accel_timeseries" value="$accel_data" sync="group_a">
    <Channel column="accel_x" strokeColor="#FF0000"/>
    <Channel column="accel_y" strokeColor="#00FF00"/>
  </TimeSeries>
  <TimeSeries name="gyro_timeseries" value="$gyro_data" sync="group_a">
    <Channel column="gyro_x" strokeColor="#0000FF"/>
    <Channel column="gyro_y" strokeColor="#FF00FF"/>
  </TimeSeries>
</View>
```
Copy link

netlify bot commented Apr 15, 2025

👷 Deploy request for heartex-docs pending review.

Visit the deploys page to approve it

Name Link
🔨 Latest commit 345fa0f

Copy link

netlify bot commented Apr 15, 2025

👷 Deploy request for label-studio-docs-new-theme pending review.

Visit the deploys page to approve it

Name Link
🔨 Latest commit 345fa0f

@github-actions github-actions bot added the feat label Apr 15, 2025
Copy link

netlify bot commented Apr 15, 2025

Deploy Preview for label-studio-storybook ready!

Name Link
🔨 Latest commit 345fa0f
🔍 Latest deploy log https://app.netlify.com/sites/label-studio-storybook/deploys/680bd2df101e2c000845a894
😎 Deploy Preview https://deploy-preview-7376--label-studio-storybook.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@makseq makseq changed the title feat: Add sync between TimeSeries, Video, Audio, and other TimeSeries… feat: Add sync between TimeSeries, Video, Audio, and other TimeSeries components Apr 24, 2025
@makseq makseq changed the title feat: Add sync between TimeSeries, Video, Audio, and other TimeSeries components feat: Add sync between TimeSeries, Video, Audio Apr 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Video/audio sync doesn't work
1 participant