Skip to content

Add support to download only the updated ranges using Etags, keeping … #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

CNScott
Copy link

@CNScott CNScott commented Jun 25, 2024

…resiliency and updating to match current version

This is based almost entirely off of eduardo010174's work. I just added the resiliency support in from the newer version.

…resiliency and updating to match current version

This is based almost entirely off of eduardo010174's work. I just added the resiliency support in from the newer version.
@eduardo010174
Copy link

Don't have problem for me.
But it's an important feature.

Other solution it's generate on server side on file with indexed Etag's of all files.
If I have time, I will make on repository for get every possible Etag and store cache of the day for not lose more time on make fake queries.

@CNScott
I have docker image on github with compile binary ready for daily use.
I will update binary with you commit if not have problems for you.

@CNScott
Copy link
Author

CNScott commented Jun 26, 2024

Don't have problem for me. But it's an important feature.

Other solution it's generate on server side on file with indexed Etag's of all files. If I have time, I will make on repository for get every possible Etag and store cache of the day for not lose more time on make fake queries.

@CNScott I have docker image on github with compile binary ready for daily use. I will update binary with you commit if not have problems for you.

No problem from me. Thank you! We really needed this feature!

{
try
{
string json = JsonSerializer.Serialize(_etag);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the content, why not serialize like this, to make the output cleaner?

string json = JsonSerializer.Serialize(_etag, s_jsonOptions);

Using this:

private static JsonSerializerOptions s_jsonOptions = new() { Encoder = System.Text.Encodings.Web.JavaScriptEncoder.UnsafeRelaxedJsonEscaping };

var request = new HttpRequestMessage(HttpMethod.Get, requestUri);
if (!string.IsNullOrEmpty(_etag[i]))
{
request.Headers.Add("If-None-Match", _etag[i]);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use the built in string constants?

request.Headers.IfNoneMatch.Add(new EntityTagHeaderValue(_etag[i]));

@@ -250,7 +325,12 @@ private async Task<Stream> GetPwnedPasswordsRangeFromWeb(int i, bool fetchNtlm)
break;
}

return content;
if (data_updated && response.Headers.TryGetValues("Etag", out IEnumerable<string>? new_etag))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be clearer?

if (data_updated)
{
    _etag[i] = response.Headers.ETag?.Tag ?? "";
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants