Whisper timestamp fix

[RyanMetcalfeInt8]

Closes #1855

[RyanMetcalfeInt8]

Just added 1 more commit here that aligns the logic between static / dynamic whisper pipelines when it comes to overall produced tokens & max_new_tokens. This if statement was removed from dynamic whisper pipeline some time back, but not from static pipeline... which causes really long audio segments processed by NPU to stop before transcribing the entire duration.

[RyanMetcalfeInt8]

Hi @as-suvorov, could you take a look when you get a chance? thanks!

[as-suvorov]

Suggested change

	const float toffset = 0.f);
	const float time_offset = 0.f);

[as-suvorov]

Suggested change

	const float chunk_toffset = chunk_offset * chunk_length_in_seconds;
	const float chunk_time_offset = chunk_offset * chunk_length_in_seconds;

[as-suvorov]

same comment for whisper_pipeline_static.cpp

Suggested change

	const float chunk_length_in_seconds =
	const float frame_length_in_seconds =

[as-suvorov]

@RyanMetcalfeInt8 thanks for PR!
In the gh issue you wrote that you also addressed "offset drift" #1855 (comment). Could you please explain what is the fix?

[RyanMetcalfeInt8]

@RyanMetcalfeInt8 thanks for PR! In the gh issue you wrote that you also addressed "offset drift" #1855 (comment). Could you please explain what is the fix?

sure. Well in this case the 'offset drift' issue that I refer to is caused is within the workaround code submitted by author of that issue:

    auto output = pipeline.generate(input, config);

    // unwrap chunks, TODO: handle error case
    auto chunks = output.chunks.value();

    float offset = 0.0;
    float last_chunk_length = 0.0;

    for (const auto& chunk : chunks) {
        if (chunk.start_ts == 0.0) {
            // this almost works, but offset becomes too small after a while. 
            offset += last_chunk_length;
        }
        last_chunk_length = chunk.end_ts;
        float abs_start = chunk.start_ts + offset;
        float abs_end = chunk.start_ts + offset;
        std::cout << abs_start << " - " << abs_end << ": " << chunk.text << std::endl;
    }

In other words, it's not really a functional workaround as it assumes that all chunk segments will be contiguous with no gaps -- offset keeps getting accumulated by the last_chunk_length. So, the abs_start starts to drift left for each 'gap' (of silence) between each segment.

[as-suvorov]

@RyanMetcalfeInt8 thanks for PR! In the gh issue you wrote that you also addressed "offset drift" #1855 (comment). Could you please explain what is the fix?

sure. Well in this case the 'offset drift' issue that I refer to is caused is within the workaround code submitted by author of that issue:

    auto output = pipeline.generate(input, config);

    // unwrap chunks, TODO: handle error case
    auto chunks = output.chunks.value();

    float offset = 0.0;
    float last_chunk_length = 0.0;

    for (const auto& chunk : chunks) {
        if (chunk.start_ts == 0.0) {
            // this almost works, but offset becomes too small after a while. 
            offset += last_chunk_length;
        }
        last_chunk_length = chunk.end_ts;
        float abs_start = chunk.start_ts + offset;
        float abs_end = chunk.start_ts + offset;
        std::cout << abs_start << " - " << abs_end << ": " << chunk.text << std::endl;
    }

In other words, it's not really a functional workaround as it assumes that all chunk segments will be contiguous with no gaps -- offset keeps getting accumulated by the last_chunk_length. So, the abs_start starts to drift left for each 'gap' (of silence) between each segment.

What do you mean by gap between segments? How your implementation addresses this gaps? The code snippet and this PR implementations looks identical to me as chunk.end_ts in code snippet and segment_offset in whisper.cpp are basically the same offset. Maybe I'm missing something.
Maybe you have a sample to share so I can check this as well?

[RyanMetcalfeInt8]

What do you mean by gap between segments? How your implementation addresses this gaps? The code snippet and this PR implementations looks identical to me as chunk.end_ts in code snippet and segment_offset in whisper.cpp are basically the same offset. Maybe I'm missing something. Maybe you have a sample to share so I can check this as well?

So for the 'fix' in this PR, an absolute time is calculated for each iteration of this loop: for (size_t chunk_offset = 0; chunk_offset < input_features.n_frames; chunk_offset += segment_offset), given chunk_offset. Essentially, given chunk_offset, calculate an absolute time offset, which then is added to the start / end time for the segments generated.

This offset ends up being a little bit (or a lot in some cases) different than if you were to accumulate (chunk.end_ts - chunk.start_ts).

Sure, let me prepare a sample with some illustrations from the Audacity project where it's pretty clear what the difference is.