[Mac OS] SDL2 applications crash if USB joystick attached (Speedlink Competition Pro)

[RobinSergeant]

After upgrading to a new Mac Mini M4 I encountered segmentation faults running MAME with my Speedlink Competition Pro joystick attached. The same joystick worked fine with my Fedora Linux machine and old Intel Mac.

After some investigation I've pinned this down to SDL and the following simple test application exhibits the same behaviour using SDL2 version 2.32.0:

int main()
{
  std::cout << "Calling SDL_init\n";
  if (SDL_Init(SDL_INIT_JOYSTICK) < 0)
  {
    std::cout << "Could not initialize\n";
    return 1;
  }
  else
  {
    std::cout << "init OK\n";
  }

  if (SDL_NumJoysticks() == 0)
  {
    std::cout << "No joysticks found\n";
    return 1;
  }
  std::cout << SDL_NumJoysticks() << " joysticks found" << std::endl;
}

With the joystick attached it crashes inside SDL_Init() due to what looks like memory corruption.

I also found that the Amiga emulator FS-UAE works fine. As this is bundled with a much older framework I tried using SDL2 version 2.0.20. With this old version both my test application and MAME work fine. Therefore, this looks like a possible regression error, but I don't know when it was introduced.

Please see original MAME issue report from 2023:

mamedev/mame#11568

Here is my test application stack trace in case that helps:

test(3025,0x1fade8840) malloc: Heap corruption detected, free list is damaged at 0x6000039f8060
*** Incorrect guard value: 36170086427328512
test(3025,0x1fade8840) malloc: *** set a breakpoint in malloc_error_break to debug
Process 3025 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x000000019143f720 libsystem_kernel.dylib`__pthread_kill + 8
libsystem_kernel.dylib`__pthread_kill:
->  0x19143f720 <+8>:  b.lo   0x19143f740    ; <+40>
    0x19143f724 <+12>: pacibsp 
    0x19143f728 <+16>: stp    x29, x30, [sp, #-0x10]!
    0x19143f72c <+20>: mov    x29, sp
Target 0: (test) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x000000019143f720 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x0000000191477f70 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x0000000191384908 libsystem_c.dylib`abort + 128
    frame #3: 0x000000019128de38 libsystem_malloc.dylib`malloc_vreport + 896
    frame #4: 0x00000001912b6458 libsystem_malloc.dylib`malloc_zone_error + 100
    frame #5: 0x00000001912a5774 libsystem_malloc.dylib`nanov2_guard_corruption_detected + 44
    frame #6: 0x00000001912a5734 libsystem_malloc.dylib`nanov2_allocate_outlined + 460
    frame #7: 0x00000001912a4468 libsystem_malloc.dylib`nanov2_calloc_type + 568
    frame #8: 0x000000019119698c libxpc.dylib`_xpc_alloc + 40
    frame #9: 0x000000019118308c libxpc.dylib`_xpc_dictionary_unpack_value_and_vend + 44
    frame #10: 0x000000019117d690 libxpc.dylib`_xpc_dictionary_look_up + 160
    frame #11: 0x0000000191183330 libxpc.dylib`vproc_swap_complex + 224
    frame #12: 0x00000001911831e0 libxpc.dylib`vproc_swap_string + 64
    frame #13: 0x0000000191524ea8 CoreFoundation`__CFXNotificationCenterSetupConnection + 88
    frame #14: 0x000000019151f2ec CoreFoundation`_CFXNotificationCenterCreate + 352
    frame #15: 0x0000000191524e40 CoreFoundation`__CFNotificationCenterGetDistributedCenter_block_invoke + 36
    frame #16: 0x00000001912c55b4 libdispatch.dylib`_dispatch_client_callout + 20
    frame #17: 0x00000001912c6e00 libdispatch.dylib`_dispatch_once_callout + 32
    frame #18: 0x0000000191524e18 CoreFoundation`CFNotificationCenterGetDistributedCenter + 116
    frame #19: 0x00000001926de9cc Foundation`+[NSDistributedNotificationCenter notificationCenterForType:] + 176
    frame #20: 0x00000001950abea0 AppKit`+[NSEvent initialize] + 56
    frame #21: 0x00000001910aabac libobjc.A.dylib`CALLING_SOME_+initialize_METHOD + 24
    frame #22: 0x00000001910aa854 libobjc.A.dylib`initializeNonMetaClass + 692
    frame #23: 0x00000001910c8a3c libobjc.A.dylib`initializeAndMaybeRelock(objc_class*, objc_object*, locker_mixin<lockdebug::lock_mixin<objc_lock_base_t>>&, bool) + 164
    frame #24: 0x00000001910a9f98 libobjc.A.dylib`lookUpImpOrForward + 304
    frame #25: 0x00000001910a9b84 libobjc.A.dylib`_objc_msgSend_uncached + 68
    frame #26: 0x00000001001933b0 KeyboardAndMouseSupport`-[GCKeyboardAndMouseManagerImpl initWithQueue:] + 392
    frame #27: 0x00000001aa73baa4 GameController`+[GCKeyboardAndMouseManager managerWithQueue:] + 104
    frame #28: 0x00000001aa7be85c GameController`-[_GCControllerManager(Legacy) _legacy_init] + 144
    frame #29: 0x00000001aa730760 GameController`-[_GCControllerManager init] + 272
    frame #30: 0x00000001aa7761f0 GameController`-[_GCControllerManagerAppClient init] + 52
    frame #31: 0x00000001aa730550 GameController`__38+[_GCControllerManager sharedInstance]_block_invoke + 48
    frame #32: 0x00000001912c55b4 libdispatch.dylib`_dispatch_client_callout + 20
    frame #33: 0x00000001912c6e00 libdispatch.dylib`_dispatch_once_callout + 32
    frame #34: 0x00000001aa73051c GameController`+[_GCControllerManager sharedInstance] + 80
    frame #35: 0x00000001aa75923c GameController`+[GCController controllers] + 44
    frame #36: 0x00000001004b1ea8 SDL2`___lldb_unnamed_symbol3276 + 128
    frame #37: 0x00000001004ddc3c SDL2`___lldb_unnamed_symbol3774 + 292
    frame #38: 0x000000010052c988 SDL2`___lldb_unnamed_symbol4907 + 504
    frame #39: 0x0000000100002f64 test`main at joystick.cpp:9:7
    frame #40: 0x00000001910f8274 dyld`start + 2840

[slouken]

Are you able to git bisect it on the SDL2 branch?

[RobinSergeant]

I guess I would need to keep trying different releases first to find out when it broke. So far all I've tried is 2.30.10, 2.32.0 and 2.0.20. I don't have time to look tonight but can try next week if nobody has tracked it down by then.

[slouken]

Thanks!

[RobinSergeant]

After trying various releases 2.26.5 is the last one that works so the bug seems to have been introduced in 2.27.1

[RobinSergeant]

@slouken I've managed to do a debug build using Xcode on the latest SDL2 branch. Does the following stack trace provide any clues? At least there are some line numbers now!

    frame #35: 0x00000001b2e8d23c GameController`+[GCController controllers] + 44
    frame #36: 0x000000010080a568 SDL2`IOS_JoystickInit at SDL_mfijoystick.m:871:42
    frame #37: 0x000000010085ab70 SDL2`SDL_JoystickInit at SDL_joystick.c:647:13
    frame #38: 0x00000001008e00b4 SDL2`SDL_InitSubSystem_REAL(flags=512) at SDL.c:319:17
    frame #39: 0x00000001008e08f0 SDL2`SDL_Init_REAL(flags=512) at SDL.c:394:12
    frame #40: 0x00000001007d2508 SDL2`SDL_Init_DEFAULT(a=512) at SDL_dynapi_procs.h:88:1
    frame #41: 0x00000001007c4430 SDL2`SDL_Init(a=512) at SDL_dynapi_procs.h:88:1
    frame #42: 0x00000001000030dc test`main at joystick.cpp:10:7

[slouken]

That crash is in Apple code, but I don't see any meaningful changes in the SDL GCController support between 2.26.5 and 2.28.0, so I'm not sure why the new code is crashing.

Is this the controller you have?
https://www.amazon.com/SPEEDLINK-SL-650212-BKRD-Competition-EXTRA-Joystick/dp/B07K9N36GG

[RobinSergeant]

Yes, that's the controller. Thanks for looking.

I guess it's some sort of memory corruption as occasionally my test app exits without crashing.

[slouken]

Can you use address sanitizer to track it down?
https://stackoverflow.com/questions/16130191/clang-address-sanitizer-on-os-x

[RobinSergeant]

Thanks, I'll try and do that at the weekend. I normally develop on Linux so haven't done much debugging on a Mac. It looks similar to Valgrind.

[RobinSergeant]

Address Sanitizer has found has found heap use after free which is probably the cause:

Calling SDL_init
=================================================================
==4614==ERROR: AddressSanitizer: heap-use-after-free on address 0x6020000016b0 at pc 0x0001031b6dd4 bp 0x00016d35eff0 sp 0x00016d35e7a0
WRITE of size 9 at 0x6020000016b0 thread T0
    #0 0x1031b6dd0 in memcpy+0x428 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x52dd0)
    #1 0x103c36e7c  (IOHIDLib:arm64e+0xee7c)
    #2 0x182985d3c in __CFMachPortPerform+0x124 (CoreFoundation:arm64e+0xaad3c)
    #3 0x182958e34 in __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__+0x38 (CoreFoundation:arm64e+0x7de34)
    #4 0x182958d54 in __CFRunLoopDoSource1+0x208 (CoreFoundation:arm64e+0x7dd54)
    #5 0x1829576b4 in __CFRunLoopRun+0x8c4 (CoreFoundation:arm64e+0x7c6b4)
    #6 0x182956730 in CFRunLoopRunSpecific+0x248 (CoreFoundation:arm64e+0x7b730)
    #7 0x104132440 in process_pending_events hid.c:510
    #8 0x104131adc in PLATFORM_hid_enumerate hid.c:529
    #9 0x104137698 in SDL_hid_enumerate_REAL SDL_hidapi.c:1329
    #10 0x1043b42e4 in HIDAPI_UpdateDeviceList SDL_hidapijoystick.c:1112
    #11 0x1043b4b58 in HIDAPI_IsDevicePresent SDL_hidapijoystick.c:1262
    #12 0x1044693e0 in GetDeviceInfo SDL_iokitjoystick.c:501
    #13 0x1044682fc in JoystickDeviceWasAddedCallback SDL_iokitjoystick.c:557
    #14 0x18638e728 in __IOHIDManagerDeviceApplier+0x24c (IOKit:arm64e+0x4b728)
    #15 0x18291f72c in __CFSetApplyFunction_block_invoke+0x18 (CoreFoundation:arm64e+0x4472c)
    #16 0x18291f548 in CFBasicHashApply+0x90 (CoreFoundation:arm64e+0x44548)
    #17 0x18291f484 in CFSetApplyFunction+0xdc (CoreFoundation:arm64e+0x44484)
    #18 0x18638eaf8 in __IOHIDManagerInitialEnumCallback+0x74 (IOKit:arm64e+0x4baf8)
    #19 0x1829588a0 in __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__+0x18 (CoreFoundation:arm64e+0x7d8a0)
    #20 0x182958834 in __CFRunLoopDoSource0+0xac (CoreFoundation:arm64e+0x7d834)
    #21 0x182958598 in __CFRunLoopDoSources0+0xf0 (CoreFoundation:arm64e+0x7d598)
    #22 0x182957134 in __CFRunLoopRun+0x344 (CoreFoundation:arm64e+0x7c134)
    #23 0x182956730 in CFRunLoopRunSpecific+0x248 (CoreFoundation:arm64e+0x7b730)
    #24 0x104468210 in ConfigHIDManager SDL_iokitjoystick.c:612
    #25 0x104467b08 in CreateHIDManager SDL_iokitjoystick.c:669
    #26 0x1044655b0 in DARWIN_JoystickInit SDL_iokitjoystick.c:683
    #27 0x1042dae88 in SDL_JoystickInit SDL_joystick.c:647
    #28 0x104481810 in SDL_InitSubSystem_REAL SDL.c:319
    #29 0x10448239c in SDL_Init_REAL SDL.c:394
    #30 0x10411e490 in SDL_Init_DEFAULT SDL_dynapi_procs.h:88
    #31 0x10410bb00 in SDL_Init SDL_dynapi_procs.h:88
    #32 0x102a9f0d8 in main joystick.cpp:13
    #33 0x1824f0270  (<unknown module>)

0x6020000016b0 is located 0 bytes inside of 9-byte region [0x6020000016b0,0x6020000016b9)
freed by thread T0 here:
    #0 0x1031b8d40 in free+0x98 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54d40)
    #1 0x104134d9c in PLATFORM_free_hid_device hid.c:181
    #2 0x104136eb0 in PLATFORM_hid_close hid.c:1177
    #3 0x104139a84 in SDL_hid_close_REAL SDL_hidapi.c:1573
    #4 0x1043bcf84 in HIDAPI_SetupDeviceDriver SDL_hidapijoystick.c:540
    #5 0x1043bb008 in HIDAPI_AddDevice SDL_hidapijoystick.c:958
    #6 0x1043b4540 in HIDAPI_UpdateDeviceList SDL_hidapijoystick.c:1124
    #7 0x1043b411c in HIDAPI_JoystickInit SDL_hidapijoystick.c:623
    #8 0x1042dae88 in SDL_JoystickInit SDL_joystick.c:647
    #9 0x104481810 in SDL_InitSubSystem_REAL SDL.c:319
    #10 0x10448239c in SDL_Init_REAL SDL.c:394
    #11 0x10411e490 in SDL_Init_DEFAULT SDL_dynapi_procs.h:88
    #12 0x10410bb00 in SDL_Init SDL_dynapi_procs.h:88
    #13 0x102a9f0d8 in main joystick.cpp:13
    #14 0x1824f0270  (<unknown module>)

previously allocated by thread T0 here:
    #0 0x1031b8fd0 in calloc+0x9c (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x54fd0)
    #1 0x104133574 in PLATFORM_hid_open_path hid.c:854
    #2 0x1041388e8 in SDL_hid_open_path_REAL SDL_hidapi.c:1463
    #3 0x1043bcc58 in HIDAPI_SetupDeviceDriver SDL_hidapijoystick.c:517
    #4 0x1043bb008 in HIDAPI_AddDevice SDL_hidapijoystick.c:958
    #5 0x1043b4540 in HIDAPI_UpdateDeviceList SDL_hidapijoystick.c:1124
    #6 0x1043b411c in HIDAPI_JoystickInit SDL_hidapijoystick.c:623
    #7 0x1042dae88 in SDL_JoystickInit SDL_joystick.c:647
    #8 0x104481810 in SDL_InitSubSystem_REAL SDL.c:319
    #9 0x10448239c in SDL_Init_REAL SDL.c:394
    #10 0x10411e490 in SDL_Init_DEFAULT SDL_dynapi_procs.h:88
    #11 0x10410bb00 in SDL_Init SDL_dynapi_procs.h:88
    #12 0x102a9f0d8 in main joystick.cpp:13
    #13 0x1824f0270  (<unknown module>)

SUMMARY: AddressSanitizer: heap-use-after-free (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x52dd0) in memcpy+0x428
Shadow bytes around the buggy address:
  0x602000001400: fa fa fd fa fa fa fd fa fa fa fd fa fa fa 03 fa
  0x602000001480: fa fa fd fd fa fa 03 fa fa fa 03 fa fa fa fd fd
  0x602000001500: fa fa fd fd fa fa fd fd fa fa fd fd fa fa 04 fa
  0x602000001580: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x602000001600: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
=>0x602000001680: fa fa fd fa fa fa[fd]fd fa fa fd fa fa fa fd fd
  0x602000001700: fa fa 00 fa fa fa fd fa fa fa 00 fa fa fa 00 00
  0x602000001780: fa fa 00 00 fa fa 00 00 fa fa fd fd fa fa fd fd
  0x602000001800: fa fa fd fd fa fa fd fd fa fa fd fa fa fa fd fd
  0x602000001880: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fa
  0x602000001900: fa fa fd fa fa fa fd fd fa fa fd fd fa fa fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb

[RobinSergeant]

@slouken while obviously not the correct solution commenting out the following line in SDL_config_macosx.h does fix the problem for my joystick:

#define SDL_JOYSTICK_HIDAPI 1

Building a release build with this config also stops MAME from crashing.

Do you have any idea why the IOKIT driver is trying to use a pointer allocated and freed by the HIDAPI driver?

[slouken]

SDL is registering a report callback for opened devices, and passing it a newly allocated input_report_buf. When it closes the device, it unregisters that callback, runs the run loop to make sure it's really unregistered, and then frees the buffer.

Apparently what's happening is that a report is still queued somewhere and the unregistered callback gets called with the freed buffer anyway. I think this is a bug in Apple's framework, but I'm not sure.

[slouken]

Out of curiosity, does this happen with SDL3 as well?

Here's the SDL3 version of your example:

#include <SDL3/SDL.h>

int main()
{
  int num_joysticks;

  std::cout << "Calling SDL_init\n";
  if (!SDL_Init(SDL_INIT_JOYSTICK))
  {
    std::cout << "Could not initialize\n";
    return 1;
  }
  else
  {
    std::cout << "init OK\n";
  }

  SDL_free(SDL_GetJoysticks(&num_joysticks));
  if (num_joysticks == 0)
  {
    std::cout << "No joysticks found\n";
    return 1;
  }
  std::cout << num_joysticks << " joysticks found" << std::endl;
  return 0;
}