How to change filter_complex

[odi1n]

Please tell me how to implement the following.

I have a list of videos, they are of different sizes, and I need to overlay them on the main video. I composed the following FFmpeg command:

ffmpeg \
-i main_video.mp4 \
-f concat -safe 0 -i list.txt \
-filter_complex \
"[1:v]scale=1280x720:force_original_aspect_ratio=decrease,setsar=1[scaled]; \
 [scaled]concat=n=1:v=1:a=0[fg]; \
 [0:v]scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2,setsar=1[bg]; \
 [bg][fg]overlay=(W-w)/2:(H-h)/2:shortest=1[outv]; \
 [0:a]volume=0.5[bg_audio]; \
 [1:a]concat=n=1:v=0:a=1,volume=1.5[concat_audio]; \
 [bg_audio][concat_audio]amix=inputs=2:duration=longest:dropout_transition=2[a]" \
-map "[outv]" \
-map "[a]" \
-c:v libx264 -crf 18 -preset fast \
-c:a aac -b:a 192k \
output.mp4

After that, I wrote a Python script using the library:

input_concat = ffmpeg.input(
    filename="list.txt",
    f="concat",
    extra_options={"safe": 0},
)
scaled_video = input_concat.video.scale(
    w=1280,
    h=720,
    force_original_aspect_ratio="decrease",
).setsar(max=1)
foreground = ffmpeg.filters.concat(scaled_video, n=1, v=1, a=0)

main_video_input = ffmpeg.input(filename=main_video_path)

background = (
    main_video_input.video
    .scale(w=1920, h=1080, force_original_aspect_ratio="decrease")
    .pad(width=1920, height=1080, x="(ow-iw)/2", y="(oh-ih)/2")
    .boxblur(luma_radius=10)
    .setsar(max=1)
)

output_video = ffmpeg.filters.overlay(
    background,
    foreground.video(0),
    x="(W-w)/2", y="(H-h)/2",
    shortest=1,
    ts_sync_mode=1,
)

# Concatenate audio
background_audio = main_video_input.audio.volume(volume=0.5)
concat_audio = input_concat.audio.volume(volume=1.5)
output_audio = ffmpeg.filters.amix(
    background_audio,
    concat_audio,
    inputs=2,
    duration="longest",
    dropout_transition=2,
)

f = (
    ffmpeg.output(
        output_video,
        output_audio,
        filename=final_output_path,
        vcodec="libx264",
        acodec="aac",
        r=self.FPS,
        b=self.BITRATE,
        extra_options={
            "crf": 18,
            "preset": "fast",
        },
        t=10,
    )
    .overwrite_output()
)

f.run(
    overwrite_output=True,
    capture_stderr=True,
    capture_stdout=True,
)

As a result, I get the following FFmpeg command:

ffmpeg \
-y -t 10 \
-i test_video.mp4 \
-f concat -safe 0 -i list.txt \
-filter_complex \
"[0:v]scale=w=1920:h=1080:force_original_aspect_ratio=decrease[s0]; \
 [1:v]scale=w=1280:h=720:force_original_aspect_ratio=decrease[s1]; \
 [0:a]volume=volume=0.5[s2]; \
 [1:a]volume=volume=1.5[s3]; \
 [s0]pad=1920:1080:(ow-iw)/2:(oh-ih)/2[s4]; \
 [s1]setsar=1[s5]; \
 [s4]boxblur=10[s7]; \
 [s5]concat=n=1:v=1:a=0[s8]; \
 [s7]setsar=1[s9]; \
 [s2][s3]amix=inputs=2:duration=longest:dropout_transition=2[s6]; \
 [s9][s8]overlay=x=(W-w)/2:y=(H-h)/2:shortest=1[s10]" \
-map '[s10]' -map '[s6]' -r 25 \
-vcodec libx264 -b 10485760k -acodec aac -crf 18 \
-preset fast \
video_gen.mp4

However, it doesn’t work as intended—the overlaid videos last only 1 second and abruptly cut off. I figured out that the problem lies in the following: these lines need to be swapped:

[1:v]scale=w=1280:h=720:force_original_aspect_ratio=decrease[s1]; 
[0:v]scale=w=1920:h=1080:force_original_aspect_ratio=decrease[s0]; 

As soon as I swap them, everything starts working. How can I do this using the library?

Also, while working with the library, the following question arose.

As you can see from my command, I need to specify setsar=1, but in the library, the .setsar() method has several parameters: sar and max. How can I specify setsar=1?

Can I pass the entire foreground stream here instead of just video(0)?

output_video = ffmpeg.filters.overlay(
    background,
    foreground.video(0),
    x="(W-w)/2", y="(H-h)/2",
    shortest=1,
    ts_sync_mode=1,
)

I checked the issues and examples, and the input method used to have a safe parameter, but now it's missing. How can I specify it? Through extra_options?

[lucemia]

In FFmpeg, when you use the setsar filter with a single value like setsar=1, you’re setting the sample aspect ratio (SAR) to 1:1, indicating square pixels. This is because, in FFmpeg’s filter syntax, if you provide unnamed parameters, they are assigned based on the order of the filter’s declared options.

For the setsar filter, the parameters are declared in the following order:
1. sar: Sets the sample aspect ratio. 
2. max: Sets the maximum value for the numerator or denominator in the ratio.

Therefore, setsar=1 is interpreted as setting sar=1.

In a library context, such as when using a method like .setsar(), you would specify the parameter explicitly:

.setsar(sar=1)

This ensures clarity and aligns with the parameter’s intended use.

[lucemia]

I checked the issues and examples, and the input method used to have a safe parameter, but now it's missing. How can I specify it? Through extra_options?

Yes, you’re correct. In typed-ffmpeg, the safe parameter is not directly available in the input() function. To specify it, you should use the extra_options dictionary, as it’s a special input parameter.

Here’s how you can set it:

import ffmpeg

stream = ffmpeg.input(
    "files.txt",
    f="concat",
    extra_options={
        "safe": "0",
        "protocol_whitelist": "file,http,https,tcp,tls",
    },
)

In this example, safe is set to "0" to allow unsafe file paths, which is necessary when using absolute paths in your file list. The protocol_whitelist option specifies the allowed protocols for the input.

This approach aligns with how typed-ffmpeg handles special input options.

[odi1n]

Thank you very much!

I have this command, which I attached in the example above, and the problem is with filter-complex. If [0:v] is first — the main video — the overlay video lasts 1 second and then stops. When I make the first video concat - [1:v], everything works fine. Can I control this in the library?

[odi1n]

I found a solution, but it's a workaround. Method compile

...
    if vf_commands:
        vf_commands[:2] = vf_commands[1::-1] # add
        commands += ["-filter_complex", ";".join(vf_commands)]
...

[lucemia]

Can I pass the entire foreground stream here instead of just video(0)?

output_video = ffmpeg.filters.overlay(
    background,
    foreground.video(0),
    x="(W-w)/2", y="(H-h)/2",
    shortest=1,
    ts_sync_mode=1,
)

Yes, you can pass the entire foreground stream directly to the overlay filter instead of just foreground.video(0), provided that the stream contains only video data.

If you encounter any issues or need further assistance, feel free to ask!

[lucemia]

Thank you very much!

I have this command, which I attached in the example above, and the problem is with filter-complex. If [0:v] is first — the main video — the overlay video lasts 1 second and then stops. When I make the first video concat - [1:v], everything works fine. Can I control this in the library?

I’m trying to understand and reproduce the issue you mentioned. Could you clarify what you mean by “swap”?

Are you referring to just switching the order of the following two lines, without making any other changes?

ffmpeg \
-y -t 10 \
-i test_video.mp4 \
-f concat -safe 0 -i list.txt \
-filter_complex \
"[0:v]scale=w=1920:h=1080:force_original_aspect_ratio=decrease[s0]; \. <---line 6
 [1:v]scale=w=1280:h=720:force_original_aspect_ratio=decrease[s1]; \  <--- line 7
 [0:a]volume=volume=0.5[s2]; 
 [1:a]volume=volume=1.5[s3]; \
 [s0]pad=1920:1080:(ow-iw)/2:(oh-ih)/2[s4]; \
 [s1]setsar=1[s5]; \
 [s4]boxblur=10[s7]; \
 [s5]concat=n=1:v=1:a=0[s8]; \
 [s7]setsar=1[s9]; \
 [s2][s3]amix=inputs=2:duration=longest:dropout_transition=2[s6]; \
 [s9][s8]overlay=x=(W-w)/2:y=(H-h)/2:shortest=1[s10]" \
-map '[s10]' -map '[s6]' -r 25 \
-vcodec libx264 -b 10485760k -acodec aac -crf 18 \
-preset fast \
video_gen.mp4

So the only difference would be swapping line 6 and line 7? Please confirm, or let me know if there were any other modifications involved in the working version.

[odi1n]

So the only difference would be swapping line 6 and line 7? Please confirm, or let me know if there were any other modifications involved in the working version.

Yes, that's correct. If I leave everything as it is in this command, it doesn't work. When I swap the sixth and seventh lines, everything works.

[lucemia]

So the only difference would be swapping line 6 and line 7? Please confirm, or let me know if there were any other modifications involved in the working version.

Yes, that's correct. If I leave everything as it is in this command, it doesn't work. When I swap the sixth and seventh lines, everything works.

Sorry, I ran some experiments but couldn’t reproduce the issue on my end — the two commands generated identical outputs for me. That said, FFmpeg can behave differently depending on subtle factors, so I agree that anything can happen under the hood.

Could you let me know which FFmpeg version you’re using?
Also, if possible, please share the input video you used, or provide more details about the inputs — for example, the duration of main_video.mp4 and the contents or structure of list.txt.

[lucemia]

@odi1n would you please provide more hint for me to reproduce the issue?