How can I benchmark performance of stumpy algorithms?

[yashssh]

Hi, I'm working on upgradingNumba to LLVM20 and right now we are in the middle of upstreaming changes. I want to measure the performance uplifts(if any) after all these LLVM upgrades. Is there a set of benchmarks for stumpy that I can leverage?
If not are there some good examples covering major algorithms that I can use to create some micro benchmarks?

[seanlaw]

Hi @yashssh! Welcome to the STUMPY community.

So, we don't have an active benchmark per se but our most common workhorse example is the following:

import numpy as np
import stumpy
import time

n = 100_000
T = np.random.rand(n)
m = 50

timing = []

for i in range(11):
    start = time.time()
    mp = stumpy.stump(T, m)
    timing.append(time.time() - start)

print(np,mean(timing[1:]))  # We skip the first run and compute the average time from the remaining runs

Depending on your your hardware resources, this should take 5-20 seconds to run on modern multithreaded hardware. Let me know if that helps and if you have any further questions. Please feel free to link to your repo if you'd like to discuss it there instead.

[yashssh]

Thank you for the example @seanlaw!

I'm also trying to run the tests using test.sh and I am seeing some assertion failures. However, the test run is aborted as soon as they run into a failure, is there a way to prevent test execution to abort if there is a failure?

[seanlaw]

No, I'm not aware of any way to ignore test failures. Are the tests failing due to your LLVM20 upgrades?

[yashssh]

Yes. Across all unit tests I am seeing 2 failures each in test_stimp.py and test_scrump.py. I am yet to investigate if this is an issue with LLVM, Numba or Stumpy.

Everything else seems to be working fine.

How can I create standalone reproducers for these failing tests?

FAILED tests/test_stimp.py::test_stimp_1_percent[T1] - AssertionError:
FAILED tests/test_stimp.py::test_stimp_1_percent_with_isconstant - AssertionError:
FAILED test_scrump.py::test_prescrump_self_join_larger_window_m_5_k_5[T_A0-T_B0] - AssertionError:
FAILED tests/test_scrump.py::test_prescrump_self_join_larger_window_m_5_k_5_with_isconstant[T_A0-T_B0] - AssertionError:

Let me know if you want to look at the error message.

[yashssh]

Nvm, attaching the failure logs
scrump.log
stimp.log

[seanlaw]

@yashssh This looks suspicious. What version of LLVM and Numba are you using? I would try using the current stable versions of LLVM and Numba first to see if the assert errors still persist.

Also, if you are running things on Linux, please install tbb >= 2019.5 for proper multi-threading.

[yashssh]

I am checking with LLVM 20 based Numba(that version is still not released, the pull requests for that change are still under review on llvmlite and Numba repositories). I did check with latest release of Numba (v0.61 based on LLVM15) and the failures don't persist.

I don't think I have tbb installed, is it necessary for this example?

[seanlaw]

I don't think I have tbb installed, is it necessary for this example?

Yes, for running in parallel with numba on Linux, tbb is necessary. Otherwise, it may not be thread safe. Your log files are full of tbb warnings, which should be taken seriously.

[yashssh]

Thanks for looking into it! I couldn't find an easy way to fix the tbb issue but I did try with omp and workqueue threading layers. The warnings disappear but I still see the assert failure with both. Attaching crash logs of the test executed with omp threading layer.

$ NUMBA_THREADING_LAYER='omp' python scrump.py

Traceback (most recent call last):
  File "/local/home/yashwants/work/logs/stumpy/scrump.py", line 207, in <module>
    test_prescrump_self_join_larger_window_m_5_k_5(test_data[0][0], test_data[0][1])
  File "/local/home/yashwants/work/logs/stumpy/scrump.py", line 204, in test_prescrump_self_join_larger_window_m_5_k_5
    npt.assert_almost_equal(ref_P, comp_P)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/local/home/yashwants/venv/perf-nvmain/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 573, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/local/home/yashwants/venv/perf-nvmain/lib/python3.10/site-packages/numpy/_utils/__init__.py", line 85, in wrapper
    return fun(*args, **kwargs)
  File "/local/home/yashwants/venv/perf-nvmain/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 1137, in assert_array_almost_equal
    assert_array_compare(compare, actual, desired, err_msg=err_msg,
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/local/home/yashwants/venv/perf-nvmain/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 779, in assert_array_compare
    flagged |= func_assert_same_pos(x, y,
  File "/local/home/yashwants/venv/perf-nvmain/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 745, in func_assert_same_pos
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 7 decimals

+inf location mismatch:
 ACTUAL: array([[inf, inf, inf, inf, inf],
       [inf, inf, inf, inf, inf],
       [inf, inf, inf, inf, inf]])
 DESIRED: array([[          inf,           inf,           inf,           inf,
                  inf],
       [3.3320009e-08,           inf,           inf,           inf,...

Is this enough to conclude it might not be a multi-threading issue?

[seanlaw]

The warnings disappear

That's a good sign (at least to eliminate problems).

Is this enough to conclude it might not be a multi-threading issue?

Yes, this likely means that LLVM20 based numba is doing something different/wrong

[yashssh]

I am starting to believe this issue is closely related to fast math optimisations. Is there a way to disable fast math optimisation before executing the tests? I can see the config variable config.STUMPY_FASTMATH_TRUE but I think that might require re-installing stumpy?

[seanlaw]

In case it matters, @NimaSarajpoor had resolved some recent issues with LLVM fastmath but this seems unrelated to those changes since the tests were passing.

I am starting to believe this issue is closely related to fast math optimisations. Is there a way to disable fast math optimisation before executing the tests?

The simplest way would be to modify config.py with:

    # "STUMPY_FASTMATH_TRUE": True,
    # "STUMPY_FASTMATH_FLAGS": {"nsz", "arcp", "contract", "afn", "reassoc"},
    # "STUMPY_FASTMATH_FASTMATH._ADD_ASSOC": True,
    "STUMPY_FASTMATH_TRUE": False,
    "STUMPY_FASTMATH_FLAGS": False,
    "STUMPY_FASTMATH_FASTMATH._ADD_ASSOC": False,

and then reinstall STUMPY:

python -m pip uninstall stumpy
python -m pip install .

Then, you should be able to run your tests as per usual.

[yashssh]

Awesome thank you!

Yes indeed, both tests pass after I disable fast math!

In case it matters, @NimaSarajpoor had resolved stumpy-dev/automate#3 with LLVM fastmath but this seems unrelated to those changes since the tests were passing.

I believe since the LLVM version I am testing with is 5 generations ahead it's fair to assume the meaning of some of these fast math flags must have diverged, might necessitate another similar debugging session when the time comes. I'll spend a bit more time on it if I can narrow down the root cause, else I'll try to return when Numba officially moves to LLVM 20+.

[seanlaw]

It's rather concerning that the LLVM fastmath flags continue to diverge and cause problems instead of maintaining backwards compatibility. Do the tests still fail if you do:

    "STUMPY_FASTMATH_TRUE": {"nsz", "arcp", "contract", "afn", "reassoc"},
    "STUMPY_FASTMATH_FLAGS": {"nsz", "arcp", "contract", "afn", "reassoc"},
    "STUMPY_FASTMATH_FASTMATH._ADD_ASSOC": {"nsz", "arcp", "contract", "afn", "reassoc"},

Considering the instability, I'm thinking that we may want to be more conservative with the fastmath flags that we use.

By the way, out of curiosity @yashssh, may I ask how you came upon the STUMPY package and what it is that you are trying to achieve?

[yashssh]

Apologies for the late reply @seanlaw , I got occupied with other things.

Do the tests still fail if you do:

Surprisingly not, the failures go away if I mention the flags explicitly like you asked. Looks like this might have something to do with newer fast math flags that were not available till LLVM15.

By the way, out of curiosity @yashssh, may I ask how you came upon the STUMPY package and what it is that you are trying to achieve?

I'm working on upgrading Numba to LLVM20 so trying to make sure if everything's working well and to collect some numbers around how much performance gains can dependencies expect. I was told STUMPY is a direct user of Numba so started here. If you can point me to other benchmarks/applications that use Numba heavily that would be great!

Thanks

[seanlaw]

Apologies for the late reply @seanlaw , I got occupied with other things.

No problem at all @yashssh! Thank you for supporting the numba/llvm community!

Surprisingly not, the failures go away if I mention the flags explicitly like you asked. Looks like this might have something to do with newer fast math flags that were not available till LLVM15.

Are you able to share any references to the "newer fastmath flags"? The numba docs doesn't say anything particularly useful regarding advanced flags and the referenced LLVM fastmath docs is limited too.

I really want to know the difference between the fastmath=True and fastmath={"nsz", "arcp", "contract", "afn", "reassoc"}. Originally, it sounded like the difference is ninf and nnz would be removed but based on whats you are seeing, there must be many more flags that aren't documented. In STUMPY, speed is important but accuracy/precision is even more important. Upgrading to LLVM20 should NOT be viewed ONLY from a performance standpoint. One must consider how many packages will be broken because the computed result will be completely wrong with incorrect output. STUMPY just happens to be meticulous in checking that the results are correct but it sounds like upgrading LLVM would break backwards compatibility and that is bad.

I'm working on upgrading Numba to LLVM20 so trying to make sure if everything's working well and to collect some numbers around how much performance gains can dependencies expect. I was told STUMPY is a direct user of Numba so started here. If you can point me to other benchmarks/applications that use Numba heavily that would be great!

May I ask "who" will be consuming these performance benchmarks and taking action on them? I sincerely do not mean to "shoot the messenger" but the performance gains mean nothing if STUMPY is broken with the new flags.

[yashssh]

Are you able to share any references to the "newer fastmath flags"? The numba docs doesn't say anything particularly useful regarding advanced flags and the referenced LLVM fastmath docs is limited too.

I am afraid I don't have an answer right now, I don't have very good grasp over the meaning of all the flags. It was just my guess that issue might be due to some of the newer fast math flags, but let me try spend some more time on it. Also, let me add @gmarkall @stuartarchibald here to answer what exactly is the difference between fastmath=True and fastmath={"nsz", "arcp", "contract", "afn", "reassoc"}.

May I ask "who" will be consuming these performance benchmarks and taking action on them?

For now these are solely for my own experiments to get an early feel on how things might look like with the latest LLVM version in Numba. If there are surprises we can raise them with Numba where either I or someone else can pick them up.

I sincerely do not mean to "shoot the messenger" but the performance gains mean nothing if STUMPY is broken with the new flags.

I understand your concerns but I am not sure if I agree with STUMPY might be broken sentiment. Numba is right now on LLVM15 and moving to LLVM20 will not only bring performance gains but also bring in all the bug fixes that went into the compiler. On top of that these upgrades are supposed to be incremental with multiple LLVM version supported at once and not in 1 go giving us time to fix any failures.

The other thing is LLVM itself describes fast math flags as enabler for "otherwise unsafe floating-point transformations", they aren't guaranteed to produce correct results and a couple of failures in fast math precision tests due to 5 generation worth of upgrades shouldn't be that alarming(not to discredit their importance or imply they should be ignored).

[seanlaw]

That's fair. Please let me know if there's anything I can do to help @yashssh

[yashssh]

I did a bit more research and below are my findings.
fastmath=True translates to fastmath={'fast'} i.e fastmath={'nnan', 'nsz', 'ninf' 'arcp', 'contract', 'afn', 'reassoc',}

When I get rid of ninf the tests pass so indeed your original guess is correct.

Now, I don't understand the motive behind 2 config variables inside Stumpy STUMPY_FASTMATH_TRUE and STUMPY_FASTMATH_FLAGS which are used interchangeably inside the codebase but defined to very different values.
STUMPY_FASTMATH_TRUE is defaulting to True which translates to {'nnan', 'nsz', 'ninf' 'arcp', 'contract', 'afn', 'reassoc',} and STUMPY_FASTMATH_FLAGS defaults to {"nsz", "arcp", "contract", "afn", "reassoc"} with 3 more fast flags present in former.

[seanlaw]

STUMPY_FASTMATH_TRUE is defaulting to True which translates to {'nnan', 'nsz', 'ninf' 'arcp', 'contract', 'afn', 'reassoc',} and STUMPY_FASTMATH_FLAGS defaults to {"nsz", "arcp", "contract", "afn", "reassoc"} with 3 more fast flags present in former.

Rather than 3 more fastmath flags, I think there are only 2 more flags present in the former (i.e., nnan and ninf), right? There are 7 flags when fastmath=True and 5 flags in fastmath={"nsz", "arcp", "contract", "afn", "reassoc"}. Having said that, what are the "newer fastmath flags" that you were referring to earlier that are not included in {"nsz", "arcp", "contract", "afn", "reassoc"} and may be causing the unit tests to fail?

Now, I don't understand the motive behind 2 config variables inside Stumpy STUMPY_FASTMATH_TRUE and STUMPY_FASTMATH_FLAGS which are used interchangeably inside the codebase but defined to very different values.

According to the LLVM fastmath documentation, when nnan and ninf are present (as is the case with fastmath=True), it is assumed that no the function arguments/results are NOT nan/inf. So, the STUMPY functions that do not have arguments/results with nan/inf will use STUMPY_FASTMATH_TRUE while all remaining STUMPY functions that may possibly have nan/inf arguments/results must use STUMPY_FASTMATH_FLAGS. Thus, the two config variables are not used "interchangeably" and, instead, they are used with a special purpose, which depends on whether or not nan/inf arguments/results are present in the function. Otherwise, the results cannot be guaranteed to be consistent with our unit tests.