Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HttpClient ObjectDisposed after SDK upgrade from 34.0.95 -> 34.0.113 #9039

Open
haavamoa opened this issue Jun 18, 2024 · 89 comments · May be fixed by #9789
Open

HttpClient ObjectDisposed after SDK upgrade from 34.0.95 -> 34.0.113 #9039

haavamoa opened this issue Jun 18, 2024 · 89 comments · May be fixed by #9789
Assignees
Labels
Area: HTTP Issues with sockets / HttpClient. regression

Comments

@haavamoa
Copy link

haavamoa commented Jun 18, 2024

Android framework version

net8.0-android

Affected platform version

Android SDK 34.0.113

Description

Hello, and thank you for this amazing project.

We have a mobile application running MAUI in hospitals in Norway, where Android is one of the most used platforms. We needed to deliver a new version this week. Last Wednesday / Thursday we noticed that we started getting sporadic http client disposal messages. After some time we figured out that the only changes to the app was the Android SDK version delivered by .NET . 34.0.113 was released, and we noticed that pinning to 34.0.95 fixed the issues.

After a lot of times, I've actually not been able to give you a reproducible project , and due to internal policies in my company I am unable to share my project.

But I can give you a brief explanation of the architecture surrounding http client in the project:

  1. We reuse our HttpClients as much as we can in between different pages.
  2. We add our own DelegatingHandlers to make sure we add ticket, add logging etc.
  3. We use Polly for timeouts.

The times I often see it is when I navigate between two pages that is using the same http client, but is using it against different request urls.

Here is the exception we keep seeing:

ObjectDisposed_Generic
ObjectDisposed_ObjectName_Name, Java.IO.InputStreamInvoker
System.ObjectDisposedException: ObjectDisposed_Generic
ObjectDisposed_ObjectName_Name, Java.IO.InputStreamInvoker
   at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable )
   at Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualVoidMethod(String , IJavaPeerable , JniArgumentValue* )
   at Java.IO.InputStream.Close()
   at Android.Runtime.InputStreamInvoker.Close()
   at System.IO.Stream.Dispose()
   at System.IO.BufferedStream.Dispose(Boolean )
   at System.IO.Stream.Close()
   at System.IO.Stream.Dispose()
   at System.Net.Http.StreamContent.Dispose(Boolean )
   at System.Net.Http.HttpContent.Dispose()
   at System.Net.Http.HttpResponseMessage.Dispose(Boolean )
   at Xamarin.Android.Net.AndroidHttpResponseMessage.Dispose(Boolean )
   at System.Net.Http.HttpResponseMessage.Dispose()
   at System.Net.Http.HttpClient.HandleFailure(Exception , Boolean , HttpResponseMessage , CancellationTokenSource , CancellationToken , CancellationTokenSource )
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage , HttpCompletionOption , CancellationTokenSource , Boolean , CancellationTokenSource , CancellationToken )

I understand that this a long shot as I do not have a reproducible project, but I hope that anyone in the team / community can reach out and see if they might have broken something, or have an idea on what we are potentially doing wrong.

Appreciate your time!

Steps to Reproduce

None unfortunately.

Did you find any workaround?

For now, we pin the SDK version by using the rollback feature of dotnet workload install, which saved our delivery, but we plan to upgrade every bits and pieces soon, so this will soon be an issue for us again.

Relevant log output

No response

@haavamoa haavamoa added the needs-triage Issues that need to be assigned. label Jun 18, 2024
@jpobst jpobst added regression Area: HTTP Issues with sockets / HttpClient. and removed needs-triage Issues that need to be assigned. labels Jun 18, 2024
@jpobst jpobst assigned simonrozsival and unassigned jpobst Jun 18, 2024
@jpobst
Copy link
Contributor

jpobst commented Jun 18, 2024

@simonrozsival Now that we are part of the dotnet GitHub organization, should I transfer HTTP issues like this to dotnet/runtime?

@simonrozsival
Copy link
Member

@jpobst this might be an issue in Xamarin.Android.Net.AndroidHttpResponseMessage.Dispose(Boolean ) so I would keep it in the android repo for now, until we can say that this issue doesn't belong here.

@jpobst jpobst added this to the .NET 8 Servicing milestone Jun 18, 2024
@haavamoa
Copy link
Author

haavamoa commented Jun 18, 2024

I forgot to mention another work around I've had success with:

  1. Catch the exception
  2. Recreate the RequestMessage
  3. Do the same HttpClient call

This might end up with throwing another dispose exception, or not. I've added a 5 second retry, which will end up with a success at some point during those 5 seconds.

If you are interested:
Here's my HttpClient helper class that gets run when an ObjectDisposedException occurs in a DelegateHandler:

internal static class SharedRetryHelper
{
    internal static Task<HttpResponseMessage> RetryDueToAndroidBug(this IHttpClient httpClient, HttpRequestMessage requestMessage, CancellationToken cancellationToken,ObjectDisposedException objectDisposedException)
    {
        var dateTimeNow = DateTime.Now;
        var now = new TimeOnly(dateTimeNow.Hour, dateTimeNow.Minute, dateTimeNow.Second);
        if (requestMessage.RequestUri == null) throw objectDisposedException;
        if (httpClient.DisposeInformation.LastTimeDisposed != null)
        {
            var nowTicks = DateTime.Now.Ticks;
            var lastTime = httpClient.DisposeInformation.LastTimeDisposed.Value;
            if (lastTime.Add(TimeSpan.FromSeconds(5)).Ticks > nowTicks) //If it's been more than 5seconds of disposal exceptions.
            {
                httpClient.DisposeInformation.LastTimeDisposed = null;
                Log(httpClient, requestMessage, "Got disposed exception on Android for more than 5 seconds. Will give up and throw exception.");
                throw objectDisposedException;
            }    
        }
        else
        {
            httpClient.DisposeInformation.LastTimeDisposed = now;
        }
        
        httpClient.DisposeInformation.TotalTimesObjectDisposed++;
        Log(httpClient, requestMessage, $"Got disposed exception on Android. Has been disposed {httpClient.DisposeInformation.TotalTimesObjectDisposed} times, last time was {httpClient.DisposeInformation.LastTimeDisposed.Value}:{httpClient.DisposeInformation.LastTimeDisposed.Value.Second}, will retry now at {now}:{now.Second}");
        if (requestMessage.Method == HttpMethod.Get)
        {
            return httpClient.Get(requestMessage.RequestUri.AbsoluteUri, cancellationToken);
        }
            
        if (requestMessage.Method == HttpMethod.Post)
        {
            return httpClient.Post(requestMessage.RequestUri.AbsoluteUri, requestMessage.Content, cancellationToken);
        }
            
        if (requestMessage.Method == HttpMethod.Put)
        {
            return httpClient.Put(requestMessage.RequestUri.AbsoluteUri, requestMessage.Content, cancellationToken);
        }
            
        if (requestMessage.Method == HttpMethod.Delete)
        {
            return httpClient.Delete(requestMessage.RequestUri.AbsoluteUri, cancellationToken);
        }

        throw objectDisposedException;
    }

    private static void Log(IHttpClient httpClient, HttpRequestMessage requestMessage, string message)
    {
#if __ANDROID__
        if (requestMessage.RequestUri != null)
        {
            Android.Util.Log.Debug("DME HttpClientAdapter",$"HttpClientName: {httpClient.Name} : {requestMessage.RequestUri.AbsoluteUri} : {message}");    
        }
#endif
    }

IHttpClient is just an abstraction on top of HttpClient in our project. The DispiseInformatin is just a simple class containing when it last got the bug and a counter of how many times it happened. This is convenient for debugging purposes.

@haavamoa
Copy link
Author

haavamoa commented Jun 18, 2024

@simonrozsival: Just shout out if you need me to test potential nightly builds or something in our project.

@simonrozsival
Copy link
Member

@haavamoa can you please share more information about the settings you're using with the client? I'm especially interested in automatic decompression, built-in authentication (Basic?, NTLM?), proxies, ... Also are you able to tell what happened with the request just before it threw the exception (redirect, 4xx error, 5xx error)? Does this happen for a specific HTTP method or does it happen both with and without any body sent to the server?

@haavamoa
Copy link
Author

Hi @simonrozsival.

We've seen this bug on both HttpClients that is using authentication, but also on HttpClients which his not. For the ones that do ; we are using built-in-authentication using Bearer Token from the Identity Model flow. We've not set any decompression or proxies to our HttpClients.

No requests have failed before we do the calls. One example is simply trying to reach a /status/ping endpoint which does nothing more than return 200 OK. This fail sporadicly. It happens for all kinds of HTTP methods, with or without bodies.

I've spent days trying to see the connection between the order of doing calls, or what happens to them before it fails, but I am unable to find a pattern at all. From my point of view it can happen to any call we do regardless of the situation, which I find too good to be true to be honest.

@simonrozsival
Copy link
Member

Thanks for the details, @haavamoa. I remember seeing reports of ObjectDisposedException being thrown before but we've never been able to reproduce it. I will try to repro this again to see if I can narrow down what could cause this exception.

If you think of any additional details (are you making requests in parallel or just one at a time, you're making requests always when the app is in foreground or if the app is in background, ...) or if you're able to repro it reliably, please let me know.

@simonrozsival
Copy link
Member

@haavamoa could you try building your app with <UseNativeHttpHandler>false</UseNativeHttpHandler>? That will internally change AndroidMessageHandler for SocketsHttpHandler and it might help you avoid this issue altogether.

@zachdean
Copy link

Hey @simonrozsival, I have started observing this exception randomly in our http requests (MAUI) after the tooling was upgraded to Android SDK 34.0.113. Looking through our logs I do not see any discernable patterns for when the failure happens.

@simonrozsival
Copy link
Member

@zachdean do you have any stacktraces you can share? Are you able to reproduce the exception in your app?

@kmiterror
Copy link

kmiterror commented Jun 21, 2024

Same happens for us after workload was updated to 34.0.113
34.0.95 works ok

As a workaround we will try this:

Make a file like this named workload.json:
{
"microsoft.net.sdk.android": "34.0.95/8.0.100"
}
Then dotnet workload update --from-rollback-file $(androidProjectFolder)/workload.json should install 34.0.95.

Stacktrace of the crash:

An error occured deserializing the response.
Refit.ApiException: An error occured deserializing the response.
 ---> System.ObjectDisposedException: ObjectDisposed_Generic
ObjectDisposed_ObjectName_Name, Java.IO.InputStreamInvoker
   at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable )
   at Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualVoidMethod(String , IJavaPeerable , JniArgumentValue* )
   at Java.IO.InputStream.Close()
   at Android.Runtime.InputStreamInvoker.Close()
   at System.IO.Stream.Dispose()
   at System.IO.BufferedStream.Dispose(Boolean )
   at System.IO.Stream.Close()
   at System.IO.Stream.Dispose()
   at System.IO.DelegatingStream.Dispose(Boolean )
   at System.IO.Stream.Close()
   at System.IO.StreamReader.Dispose(Boolean )
   at System.IO.StreamReader.Close()
   at Newtonsoft.Json.JsonTextReader.Close()
   at Newtonsoft.Json.JsonReader.Dispose(Boolean disposing)
   at Newtonsoft.Json.JsonReader.System.IDisposable.Dispose()
   at Refit.NewtonsoftJsonContentSerializer.<FromHttpContentAsync>d__4`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at Refit.RequestBuilderImplementation.<DeserializeContentAsync>d__15`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at Refit.RequestBuilderImplementation.<>c__DisplayClass14_0`2.<<BuildCancellableTaskFuncForMethod>b__0>d[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   Exception_EndOfInnerExceptionStack
   at Refit.RequestBuilderImplementation.<>c__DisplayClass14_0`2.<<BuildCancellableTaskFuncForMethod>b__0>d[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
--- End of stack trace from previous location ---
   at Polly.Timeout.AsyncTimeoutEngine.<ImplementationAsync>d__0`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at Polly.Timeout.AsyncTimeoutEngine.<ImplementationAsync>d__0`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at Polly.AsyncPolicy.<ExecuteAsync>d__21`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at Polly.Retry.AsyncRetryEngine.<ImplementationAsync>d__0`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at Polly.AsyncPolicy.<ExecuteAsync>d__21`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at Polly.AsyncPolicy.<ExecuteAsync>d__21`1[[System.DateTime, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()

@simonrozsival
Copy link
Member

@zachdean do you also use Polly?

@haavamoa
Copy link
Author

@simonrozsival , adding <UseNativeHttpHandler>false</UseNativeHttpHandler> did not fix the problem for me :(

@simonrozsival
Copy link
Member

simonrozsival commented Jun 25, 2024

@haavamoa can you share a stacktrace from the app with UseNativeHttpHandler=false?

@zachdean
Copy link

@simonrozsival we are not using poly, but we do have a retry mechanism that is in the request pipeline. We are injecting AndroidMessageHandler as the final request handler directly into the pipeline and the only thing special we do with it is set the automatic decompression new AndroidMessageHandler { AutomaticDecompression = DecompressionMethods.All, }. Here is the stack trace that I was seeing. The error was surfacing in almost every http request I do in the app and appeared to be random.

System.ObjectDisposedException: ObjectDisposed_Generic ObjectDisposed_ObjectName_Name, Java.IO.InputStreamInvoker
at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable)
at Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualVoidMethod(String, IJavaPeerable, JniArgumentValue*)
at Java.IO.InputStream.Close()
at Android.Runtime.InputStreamInvoker.Close()
at System.IO.Stream.Dispose()
at System.IO.BufferedStream.Dispose(Boolean)
at System.IO.Stream.Close()
at System.IO.Stream.Dispose()
at System.IO.Compression.DeflateStream.Dispose(Boolean)
at System.IO.Stream.Close()
at System.IO.Stream.Dispose()
at System.IO.Compression.GZipStream.Dispose(Boolean)
at System.IO.Stream.Close()
at System.IO.Stream.Dispose()
at System.Net.Http.StreamContent.Dispose(Boolean)
at System.Net.Http.HttpContent.Dispose()
at System.Net.Http.HttpResponseMessage.Dispose(Boolean)
at Xamarin.Android.Net.AndroidHttpResponseMessage.Dispose(Boolean)
at System.Net.Http.HttpResponseMessage.Dispose()
at System.Net.Http.HttpClient.HandleFailure(Exception, Boolean, HttpResponseMessage, CancellationTokenSource, CancellationToken, CancellationTokenSource)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage, HttpCompletionOption, CancellationTokenSource, Boolean, CancellationTokenSource, CancellationToken)
at Ramsey.Common.ApiClient.Core.ApiClient.SendAsync(HttpRequestMessage request, CancellationToken token)

@jonathanpeppers
Copy link
Member

jonathanpeppers commented Jun 27, 2024

So, we don't see any changes to AndroidMessageHandler here:

we noticed that pinning to 34.0.95 fixed the issues.

Is it possible this actually changed the runtime version?

If you have a rollback file like:

{
"microsoft.net.sdk.android": "34.0.95/8.0.100"
}

I don't actually know what it would choose to do with the runtime. Did it downgrade the runtime to 8.0.0?

If you have a .binlog of some of the builds, we could check.

@jonpryor
Copy link
Member

The commit diff between 34.0.95 and 34.0.113 is: 34.0.95...34.0.113

There is only one change to Mono.Android.dll: 0315e89, which doesn't directly touch HttpClient.

That's all I can say with any degree of confidence. :-)

I have the same question as @jonathanpeppers does.

@simonrozsival: meanwhile, within the dotnet/android change set, we also changed the dotnet/runtime that we use for unit tests in 784d320, for a runtime commit diff of: dotnet/runtime@62304a6...fd8f5b5

which does contain HttpClient-related changes…

I'm thus inclined to believe that something may have changed on the dotnet/runtime side.

@haavamoa
Copy link
Author

haavamoa commented Jun 28, 2024

I will see if I can provide a .binlog file.

@haavamoa
Copy link
Author

haavamoa commented Jun 28, 2024

Here's my binlog from the build where I was running the following command and setup:

> dotnet publish -bl:msbuild.binlog ConsumerApp.csproj -f net8.0-android -c Debug

> dotnet workload list                                                                                                             git:master

Installed Workload Id      Manifest Version       Installation Source
---------------------------------------------------------------------
ios                        17.2.8053/8.0.100      SDK 8.0.300        
maui                       8.0.40/8.0.100         SDK 8.0.300        
android                    34.0.95/8.0.100        SDK 8.0.300

@haavamoa
Copy link
Author

@simonrozsival , this is the stack trace of the error with the following setup:

<UseNativeHttpHandler>false</UseNativeHttpHandler>

and

> dotnet workload list 

Installed Workload Id      Manifest Version       Installation Source
---------------------------------------------------------------------
ios                        17.2.8053/8.0.100      SDK 8.0.300        
maui                       8.0.40/8.0.100         SDK 8.0.300        
android                    34.0.113/8.0.100       SDK 8.0.300        

Stack trace:

   at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable self) in /Users/runner/work/1/s/xamarin-android/external/Java.Interop/src/Java.Interop/Java.Interop/JniPeerMembers.cs:line 153
   at Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualVoidMethod(String encodedMember, IJavaPeerable self, JniArgumentValue* parameters) in /Users/runner/work/1/s/xamarin-android/external/Java.Interop/src/Java.Interop/Java.Interop/JniPeerMembers.JniInstanceMethods_Invoke.cs:line 57
   at Java.IO.InputStream.Close() in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/obj/Release/net8.0/android-34/mcw/Java.IO.InputStream.cs:line 116
   at Android.Runtime.InputStreamInvoker.Close() in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Android.Runtime/InputStreamInvoker.cs:line 62
   at System.IO.Stream.Dispose()
   at System.IO.BufferedStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.IO.Stream.Dispose()
   at System.Net.Http.StreamContent.Dispose(Boolean disposing)
   at System.Net.Http.HttpContent.Dispose()
   at System.Net.Http.HttpResponseMessage.Dispose(Boolean disposing)
   at Xamarin.Android.Net.AndroidHttpResponseMessage.Dispose(Boolean disposing) in /Users/runner/work/1/s/xamarin-android/src/Mono.Android/Xamarin.Android.Net/AndroidHttpResponseMessage.cs:line 42
   at System.Net.Http.HttpResponseMessage.Dispose()
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)

@jonathanpeppers
Copy link
Member

@haavamoa from your .binlog, I see the 8.0.6 runtime:

KnownRuntimePack
    Microsoft.NETCore.App
        TargetFramework = net8.0
        LatestRuntimeFrameworkVersion = 8.0.6
...
	Adding /usr/local/share/dotnet/x64/packs/Microsoft.NETCore.App.Runtime.Mono.android-arm64/8.0.6/runtimes/android-arm64/native/libmonosgen-2.0.so

In the case you've gotten it to work by installing with a rollback file, is your build using a different runtime?

UseNativeHttpHandler=false also would indicate this is a change in the dotnet/runtime (BCL), as the code for that doesn't live in the Android workload.

@simonrozsival
Copy link
Member

UseNativeHttpHandler=false also would indicate this is a change in the dotnet/runtime (BCL), as the code for that doesn't live in the Android workload.

@haavamoa @jonathanpeppers given the last stacktrace, it is not using the managed handler, but it is still using the native one (notice Xamarin.Android.Net.AndroidHttpResponseMessage). Are you instantiating AndroidMessageHandler directly, @haavamoa? If so, can you change it to either SocketsHttpHandler or HttpClientHandler (+ UseNativeHttpHandler=false)?

@ederbond
Copy link

@jonathanpeppers right now I'm kind of fully packed of work on my current Job so I can stop to create an end-to-end repro sample.
If you need to share it with other folks from the .NET runtime team, fell free to copy my project's source code to an internal private git repo of yours and just ask others to do not expose it publicly.

BTW where you able to reproduce the issue on your end?

@zuo99
Copy link

zuo99 commented Oct 24, 2024

I encountered the same issue, where this error occasionally occurs when downloading data

@zuo99
Copy link

zuo99 commented Oct 24, 2024

I tested it, and using this is OK.*.csproj add
PropertyGroup>
UseNativeHttpHandler>false</UseNativeHttpHandler
PropertyGroup>

@jonathanpeppers
Copy link
Member

@jonathanpeppers right now I'm kind of fully packed of work on my current Job so I can stop to create an end-to-end repro sample. If you need to share it with other folks from the .NET runtime team, fell free to copy my project's source code to an internal private git repo of yours and just ask others to do not expose it publicly.

BTW where you able to reproduce the issue on your end?

@ederbond I tried loading the "patient screen" in your app about 20 times on emulator and 20 times on a Pixel 7, but it seems like it worked every time. Does it matter if I have fast internet?

@andreashellquist
Copy link

I have the same issue here. I have a page loading a shopping cart. It works fine and I can reload it any number of times. But if I go to another shell tab and update my customer with a post request and return to my CartPage, application crashes because of ObjectDisposedException.

jonpryor added a commit that referenced this issue Nov 6, 2024
Context: #9039

While trying to further diagnose #9039, @jonpryor started fearing
that a GC bridge issue might be at play, but the current GC bridge
logging could use some improvements.

Update `OSBridge::take_weak_global_ref_jni()` to do two things:

 1. Set the `handle` and `handle_type` fields *before* deleting the
    original GREF.  This is for a hypothetical thread-safety issue
    wherein the `handle` field and the `Handle` property could return
    a *deleted* GREF before being updated to the "new" Weak GREF.

    We have not seen this as being an issue in practice, but it's
    something we noticed.

 2. Update the `from` parameter text to `_monodroid_weak_gref_new()`
    to look more like a stack trace.

    Previously, the GREF log would contain entries such as:

        monodroid-gref: -g- grefc 441 gwrefc 1 handle 0x4fda/G from thread 'finalizer'(17269)
        monodroid-gref: take_weak_global_ref_jni

    making me wonder "why is `take_weak_global_ref_jni` there?!"

    It's there because of the "from" parameter, to provide context.
    Update it to instead read:

        monodroid-gref: -g- grefc 441 gwrefc 1 handle 0x4fda/G from thread 'finalizer'(17269)
        monodroid-gref:    at [[gc:take_weak_global_ref_jni]]

    which makes it clearer what's intended.

Update `OSBridge::take_global_ref_jni()` in a similar manner.

Update other `_monodroid*_gref_*()` call sites to update *their*
`from` parameters accordingly.

Update `AndroidRuntime.FinalizePeer()` so that the messages it logs
are actually useful.  Currently they are:

	Finalizing handle 0x0/G

which leaves a lot to be desired.  Update it to instead read:

	Finalizing Instance.Type=Example.Whatever PeerReference=0x0/G IdentityHashCode=0x123456 Instance=0xdeadbeef

"Promote" `LOG_GC` to work in Release builds as well.  This means
that both:

	adb shell setprop debug.mono.log gc
	adb shell setprop debug.mono.gc 1

will work with `libmono-android.release.so`, i.e. Release builds
of apps.

Update `OSBridge::gc_cross_references()` log the `handle` field of
bridged instances.
jonpryor added a commit that referenced this issue Nov 6, 2024
Context: #9039

While trying to further diagnose #9039, @jonpryor started fearing
that a GC bridge issue might be at play, but the current GC bridge
logging could use some improvements.

Update `OSBridge::take_weak_global_ref_jni()` to do two things:

 1. Set the `handle` and `handle_type` fields *before* deleting the
    original GREF.  This is for a hypothetical thread-safety issue
    wherein the `handle` field and the `Handle` property could return
    a *deleted* GREF before being updated to the "new" Weak GREF.

    We have not seen this as being an issue in practice, but it's
    something we noticed.

 2. Update the `from` parameter text to `_monodroid_weak_gref_new()`
    to look more like a stack trace.

    Previously, the GREF log would contain entries such as:

        monodroid-gref: -g- grefc 441 gwrefc 1 handle 0x4fda/G from thread 'finalizer'(17269)
        monodroid-gref: take_weak_global_ref_jni

    making me wonder "why is `take_weak_global_ref_jni` there?!"

    It's there because of the "from" parameter, to provide context.
    Update it to instead read:

        monodroid-gref: -g- grefc 441 gwrefc 1 handle 0x4fda/G from thread 'finalizer'(17269)
        monodroid-gref:    at [[gc:take_weak_global_ref_jni]]

    which makes it clearer what's intended.

Update `OSBridge::take_global_ref_jni()` in a similar manner.

Update other `_monodroid*_gref_*()` call sites to update *their*
`from` parameters accordingly.

Update `AndroidRuntime.FinalizePeer()` so that the messages it logs
are actually useful.  Currently they are:

	Finalizing handle 0x0/G

which leaves a lot to be desired.  Update it to instead read:

	Finalizing Instance.Type=Example.Whatever PeerReference=0x0/G IdentityHashCode=0x123456 Instance=0xdeadbeef

"Promote" `LOG_GC` to work in Release builds as well.  This means
that both:

	adb shell setprop debug.mono.log gc
	adb shell setprop debug.mono.gc 1

will work with `libmono-android.release.so`, i.e. Release builds
of apps.

Update `OSBridge::gc_cross_references()` log the `handle` field of
bridged instances.
@mydaycare-carefree
Copy link

mydaycare-carefree commented Nov 6, 2024

We are also experiencing this issue without solid replication steps.

Please see https://stackoverflow.com/questions/79135514/httpresponsemessage-readasstringasync-objectdisposed-generic-exception/79153879#79153879 for more details.

In our case, a weaker connection produces more of this error. For comparison, we also have an older MAUI app, that's just a browser with our site loaded. This JS implementation (which does the exact same thing as our native MAUI app) never has any connection issues, running in the exact same environment.

We have also setup manual retries but this is impacting performance in a significant way.

Please let me know if I can help in anyway (i.e., testing)

jonpryor added a commit that referenced this issue Nov 6, 2024
Context: #9039

While trying to further diagnose #9039, @jonpryor started fearing
that a GC bridge issue might be at play, but the current GC bridge
logging could use some improvements.

Update `OSBridge::take_weak_global_ref_jni()` to do two things:

 1. Set the `handle` and `handle_type` fields *before* deleting the
    original GREF.  This is for a hypothetical thread-safety issue
    wherein the `handle` field and the `Handle` property could return
    a *deleted* GREF before being updated to the "new" Weak GREF.

    We have not seen this as being an issue in practice, but it's
    something we noticed.  If it did happen, the app would crash and
    a JNI error message would be present in `adb logcat`, a'la:

        JNI DETECTED ERROR IN APPLICATION: use of deleted global reference 0x3d86

 2. Update the `from` parameter text to `_monodroid_weak_gref_new()`
    to look more like a stack trace.

    Previously, the GREF log would contain entries such as:

        monodroid-gref: -g- grefc 441 gwrefc 1 handle 0x4fda/G from thread 'finalizer'(17269)
        monodroid-gref: take_weak_global_ref_jni

    making me wonder "why is `take_weak_global_ref_jni` there?!"

    It's there because of the "from" parameter, to provide context.
    Update it to instead read:

        monodroid-gref: -g- grefc 441 gwrefc 1 handle 0x4fda/G from thread 'finalizer'(17269)
        monodroid-gref:    at [[gc:take_weak_global_ref_jni]]

    which makes it clearer what's intended.

Update `OSBridge::take_global_ref_jni()` in a similar manner.

Update other `_monodroid*_gref_*()` call sites to update *their*
`from` parameters accordingly.

Update `AndroidRuntime.FinalizePeer()` so that the messages it logs
are actually useful.  Currently they are:

	Finalizing handle 0x0/G

which leaves a lot to be desired.  Update it to instead read:

	Finalizing Instance.Type=Example.Whatever PeerReference=0x0/G IdentityHashCode=0x123456 Instance=0xdeadbeef

Note that the `Instance` value is `RuntimeHelpers.GetHashCode()`,
which may not be immediately useful.

"Promote" `LOG_GC` to work in Release builds as well.  This allows:

	adb shell setprop debug.mono.gc 1

to work with `libmono-android.release.so`, i.e. Release builds
of apps.

Update `OSBridge::gc_cross_references()` log the `handle` field of
bridged instances when `debug.mono.gc` is set.

When the `debug.mono.log` system property contains `gref`, you can
then correlate the handle values; for example:

	I monodroid-gc: cross references callback invoked with 1 sccs and 0 xrefs.
	I monodroid-gc: group 0 with 1 objects
	I monodroid-gc:  obj 0x6f4f3d4fd0 [android_net8_hw::MainActivity] handle 0x39c6
	I monodroid-gref: +w+ grefc 18 gwrefc 1 obj-handle 0x39c6/G -> new-handle 0x8cf/W from thread 'finalizer'(18712)
	I monodroid-gref:    at [[gc:take_weak_global_ref_jni]]
	I monodroid-gref: -g- grefc 17 gwrefc 1 handle 0x39c6/G from thread 'finalizer'(18712)
	I monodroid-gref:    at [[gc:take_weak_global_ref_jni]]
	I monodroid-gref: +g+ grefc 18 gwrefc 1 obj-handle 0x8cf/W -> new-handle 0x39ce/G from thread 'finalizer'(18712)
	I monodroid-gref:    at [[gc:take_global_ref_jni]]
	I monodroid-gref: -w- grefc 18 gwrefc 0 handle 0x8cf/W from thread 'finalizer'(18712)
	I monodroid-gref:    at [[gc:take_global_ref_jni]]
	I monodroid-gc: GC cleanup summary: 1 objects tested - resurrecting 1.

shows that the `MainActivity` instance with handle `0x39c6` was
provided to the GC bridge, and it survived the Java-side GC and was
kept alive ("resurrected").
@jonathanpeppers
Copy link
Member

@ederbond send me his entire app, which we were able to reproduce the issue.

At this point:

  • We believe this is a "Mono GC bridge" bug. The Java-side Java.IO.InputStream is being GC'd on the Android side, and is not being kept around for the .NET side that is using it
  • We need a smaller repro than @ederbond's entire app. We don't think the Mono team will have an easy time finding the problem & fixing it with such a large repro.
  • I attempted to repro here: https://github.com/jonathanpeppers/android-gcrepro (but this app seems to work fine)

If someone is able to make a sample app reproducing this issue, this is what we need. Thanks!

@mydaycare-carefree
Copy link

So glad you were able to reproduce it! any chance there are some replication steps you can provide?

I should be able to strip away my business logic and give you a small solution. My issue is I can't consistently replicate locally (but happens in production 100 of times a day).

@jonathanpeppers
Copy link
Member

The sample makes web requests using AndroidMessageHandler, at some point throws on this line:

res = BaseInputStream.Read (buffer, offset, count);

BaseInputStream was disposed from the Java side -> GC bridge.

jonpryor added a commit that referenced this issue Nov 7, 2024
Context: #9039

While trying to further diagnose #9039, @jonpryor started fearing
that a GC bridge issue might be in play, but the current GC bridge
logging could use some improvements.

Update `OSBridge::take_weak_global_ref_jni()` to do three things:

 1. Set the `handle` and `handle_type` fields *before* deleting the
    original GREF.  This is for a hypothetical thread-safety issue
    wherein the `handle` field and the `Handle` property could return
    a *deleted* GREF before being updated to the "new" Weak GREF.

    We have not seen this as being an issue in practice, but it's
    something we noticed [^1].

 2. Update the `from` parameter text to `_monodroid_weak_gref_new()`
    to look more like a stack trace.
    
    Previously, the GREF log would contain entries such as:
    
        monodroid-gref: -g- grefc 441 gwrefc 1 handle 0x4fda/G from thread 'finalizer'(17269)
        monodroid-gref: take_weak_global_ref_jni
    
    making @jonpryor wonder "why is `take_weak_global_ref_jni` there?!"
    
    It's there because of the `from` parameter, to provide context.
    Update it to instead read:
    
        monodroid-gref: -g- grefc 441 gwrefc 1 handle 0x4fda/G from thread 'finalizer'(17269)
        monodroid-gref:    at [[gc:take_weak_global_ref_jni]]
    
    which makes it clearer what's intended.

 3. Explicitly log when an instance was collected by the Java GC when
    the `debug.mono.gc` system property is set:

        monodroid-gref: handle 0x80f/W; key_handle 0xde7924e; MCW type: `Javax.Net.Ssl.HttpsURLConnectionInvoker`: was collected by a Java GC


Update `OSBridge::take_global_ref_jni()` in a similar manner.

Update other `_monodroid*_gref_*()` call sites to update *their*
`from` parameters accordingly.

Update `AndroidRuntime.FinalizePeer()` so that the messages it logs
are actually useful.  Currently they are:

	Finalizing handle 0x0/G

which leaves a lot to be desired.  Update it to instead read:

	Finalizing Instance.Type=Example.Whatever PeerReference=0x0/G IdentityHashCode=0x123456 Instance=0xdeadbeef

Note that the `Instance` value is `RuntimeHelpers.GetHashCode()`,
which may not be immediately useful.

"Promote" `LOG_GC` to work in Release builds as well.  This allows:

	adb shell setprop debug.mono.gc 1

to work with `libmono-android.release.so`, i.e. Release builds
of apps.

Update `OSBridge::gc_cross_references()` log the `handle` and
`key_handle` fields of bridged instances when `debug.mono.gc` is set.

All together, if you set:

	adb shell setprop debug.mono.log gref
	adb shell setprop debug.mono.gc 1

then `adb logcat` may have output similar to:

	/*   1 */   I monodroid-gref: +g+ grefc 38 gwrefc 0 obj-handle 0x7213951025/I -> new-handle 0x431a/G from thread '(null)'(1)
	/*   2 */   D monodroid-gref:    at …
	/*   3 */   D monodroid-gref: handle 0x431a; key_handle 0xde7924e: Java Type: `com/android/okhttp/internal/huc/HttpsURLConnectionImpl`; MCW type: `Javax.Net.Ssl.HttpsURLConnectionInvoker`
	/*   4 */   …
	/*   5 */   I monodroid-gc: cross references callback invoked with 36 sccs and 0 xrefs.
	/*   6 */   I monodroid-gc: group 4 with 1 objects
	/*   7 */   I monodroid-gc:  obj 0x6f4ecd4a60 [Javax.Net.Ssl::HttpsURLConnectionInvoker] handle 0x431a key_handle 0xde7924e
	/*   8 */   …
	/*   9 */   I monodroid-gref: +w+ grefc 67 gwrefc 5 obj-handle 0x431a/G -> new-handle 0x80f/W from thread 'finalizer'(6441)
	/*  10 */   I monodroid-gref:    at [[gc:take_weak_global_ref_jni]]
	/*  11 */   I monodroid-gref: -g- grefc 66 gwrefc 5 handle 0x431a/G from thread 'finalizer'(6441)
	/*  12 */   I monodroid-gref:    at [[gc:take_weak_global_ref_jni]]
	/*  13 */   D monodroid-gref: handle 0x80f/W; key_handle 0xde7924e; MCW type: `Javax.Net.Ssl.HttpsURLConnectionInvoker`: was collected by a Java GC
	/*  14 */   I monodroid-gref: -w- grefc 35 gwrefc 31 handle 0x80f/W from thread 'finalizer'(6441)
	/*  15 */   I monodroid-gref:    at [[gc:take_global_ref_jni]]
	/*  16 */   …
	/*  17 */   D monodroid-gref: Finalizing Instance.Type=Javax.Net.Ssl.HttpsURLConnectionInvoker PeerReference=0x0/G IdentityHashCode=0xde7924e Instance=0xf7d6f98c

with:

  * Lines 1-3: Creation of a `Javax.Net.Ssl.HttpsURLConnectionInvoker`
    instance with `key_handle` 0xde7924e.

    The omitted stack trace on "line 2" can be used to determine
    where the instance was created.

  * Lines 5-15+: GC bridge runs containing above
    `HttpsURLConnectionInvoker` instance.

  * Line 7: provides `key_handle` for correlation purposes.
    (`obj 0x6f4ecd4a60` is the raw pointer value, for MonoVM
    developer purposes.)

  * Line 13: Explicit call-out that the `HttpsURLConnectionInvoker`
    was collected by a Java-side GC.

  * Line 17: Finalization message. `IdentityHashCode` value is the
    `key_handle` value used elsewhere.


[^1]: If it did happen, the app would crash and a JNI error message
      would be present in `adb logcat`, a'la:

          JNI DETECTED ERROR IN APPLICATION: use of deleted global reference 0x3d86
jonpryor added a commit to dotnet/java-interop that referenced this issue Nov 8, 2024
Context: dotnet/android#9039
Context: dotnet/android@32495f3

@jonpryor suspects that the `ObjectDisposedException` being thrown
within dotnet/android#9039 *may* be due to a GC-related bug.

A problem with diagnosing this is tracking object lifetimes: yes, an
`Android.Runtime.InputStreamInvoker` is throwing
`ObjectDisposedException`, but in local reproductions, there are
*multiple* `InputStreamInvoker` instances created!  Which one is
throwing?

A local answer to that was "Update `InputStreamInvoker.Read()` to log
`BaseInputStream.JniIdentityHashCode`", which *was* useful, but is
not a "scalable" solution.

Review all `throw new ObjectDisposedException()` calls within
`Java.Interop.dll`, and update all sites which use `IJavaPeerable`
to include the `JniIdentityHashCode` value in the exception message.
This would result in a message like:

	System.ObjectDisposedException: Cannot access disposed object with JniIdentityHashCode=0x12345678.
	Object name: 'Android.Runtime.InputStreamInvoker'.
	   at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable )
	   …
jonpryor added a commit to dotnet/java-interop that referenced this issue Nov 8, 2024
)

Context: dotnet/android#9039
Context: dotnet/android@32495f3

@jonpryor suspects that the `ObjectDisposedException` being thrown
within dotnet/android#9039 *may* be due to a GC-related bug.

A problem with diagnosing this is tracking object lifetimes: yes, an
`Android.Runtime.InputStreamInvoker` is throwing
`ObjectDisposedException`, but in local reproductions, there are
*multiple* `InputStreamInvoker` instances created!  Which one is
throwing?

A local answer to that was "Update `InputStreamInvoker.Read()` to log
`BaseInputStream.JniIdentityHashCode`", which *was* useful, but is
not a "scalable" solution.

Review all `throw new ObjectDisposedException()` calls within
`Java.Interop.dll`, and update all sites which use `IJavaPeerable`
to include the `JniIdentityHashCode` value in the exception message.
This would result in a message like:

	System.ObjectDisposedException: Cannot access disposed object with JniIdentityHashCode=0x12345678.
	Object name: 'Android.Runtime.InputStreamInvoker'.
	   at Java.Interop.JniPeerMembers.AssertSelf(IJavaPeerable )
	   …
@basvbeek
Copy link

basvbeek commented Dec 6, 2024

If someone is able to make a sample app reproducing this issue, this is what we need. Thanks!

Hi @jonathanpeppers, I also started running into this issue and I'm mainly seeing the error when downloading files. It feels like it's occurring more often when the downloading is relatively slow.

Attached is a small sample app which roughly simulates the downloading in my real app. It also contains a small web api project which delays the file download, to simulate the server being slow. I can pretty consistently reproduce the issue using this, typically under ~10 clicks on the button. I've been testing with a Pixel 5 - API 30 in the Android Emulator.

Hope it helps.

HttpClientIssue.zip

@DanielSCBO
Copy link

Any news? Update project to .NET 9 and last MAUI and erro continues

@jonathanpeppers
Copy link
Member

If someone is able to make a sample app reproducing this issue, this is what we need. Thanks!

Hi @jonathanpeppers, I also started running into this issue and I'm mainly seeing the error when downloading files. It feels like it's occurring more often when the downloading is relatively slow.

Attached is a small sample app which roughly simulates the downloading in my real app. It also contains a small web api project which delays the file download, to simulate the server being slow. I can pretty consistently reproduce the issue using this, typically under ~10 clicks on the button. I've been testing with a Pixel 5 - API 30 in the Android Emulator.

Hope it helps.

HttpClientIssue.zip

Thanks, it looks like I can repro something:

Image

@ilyahejdoktor
Copy link

Hi @jonathanpeppers
Any updates on the issue?
Or do you need any help from our side with testing?
Thank you

@jonathanpeppers
Copy link
Member

We have a repro now, but we think it is a problem in Mono's GC bridge (so dotnet/runtime issue). Still trying to find the right person to investigate, thanks.

@talk2stephan
Copy link

@jonathanpeppers @simonrozsival

When can we expect a fix for this issue? Our production rollout is being delayed because of this, and we are under significant pressure from our customer. Please provide an ETA or suggest a workaround, as this issue is critical and could have a severe impact on our existence.

@b12kab
Copy link

b12kab commented Feb 7, 2025

@talk2stephan Is it possible to use the workaround mentioned here for now?

@simonrozsival
Copy link
Member

@talk2stephan as @b12kab suggested, try switching to the SocketsHttpHandler:

  • change your project configuration by adding <UseNativeHttpHandler>false</UseNativeHttpHandler> to your project file(s)
  • if you're using AndroidMessageHandler directly anywhere in your app, replace it with SocketsHttpHandler

@ederbond
Copy link

ederbond commented Feb 8, 2025

@simonrozsival and @jonathanpeppers I'm still facing it very often.

It's impossible to ship and app to production with this bug.
People have reported this last June, Any ETA?

Microsoft Visual Studio Enterprise 2022 (64-bit) - Current
Version 17.12.4

Installed Workload Id      Manifest Version       Installation Source
---------------------------------------------------------------------------------
android                    35.0.39/9.0.100        SDK 9.0.100, VS 17.12.35707.178
ios                        18.2.9173/9.0.100      SDK 9.0.100, VS 17.12.35707.178
maccatalyst                18.2.9173/9.0.100      SDK 9.0.100, VS 17.12.35707.178
maui-windows               9.0.14/9.0.100         SDK 9.0.100, VS 17.12.35707.178

@haavamoa
Copy link
Author

haavamoa commented Feb 10, 2025

We run with the workaround @simonrozsival mention, which solves it. But we just notice that we had some exception handling for SSL exceptions which we want to present in a user friendly way on Android. That completely broke as we are unable to catch the same types of exceptions anymore. This made me worried that we might have other unknown code paths in our production app that might missbehave now.

So I agree, how is the progress for this issue?

@b12kab
Copy link

b12kab commented Feb 10, 2025

@ederbond If you are using the workaround, do you get many other networking related crashes as @haavamoa ?

We've not yet moved off of android 34.0.95 / net8 because of this, but will have to move up to .net 9 soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: HTTP Issues with sockets / HttpClient. regression
Projects
None yet