Skip to content

Commit 8b3ddbf

Browse files
tonyofrancistonyofrancis
committed
Merge branch 'v3-develop' of https://github.com/tonyofrancis/Fetch into v3-develop
2 parents decb11d + 42381f8 commit 8b3ddbf

File tree

15 files changed

+140
-26
lines changed

15 files changed

+140
-26
lines changed

‎CHANGELOG

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,14 @@
1+
Version 3.0.8/ Android x version 3.1.2
2+
1. Fixed group error reporting bugs with FetchGroup.getProgress() method.
3+
2. Fixed network on main thread error.
4+
3. Improved network connection checks.
5+
4. Improvements and fixes to the StorageResolver when working with the Storage Access Framework.
6+
5. Files are now pre allocated on local storage on request enqueue. This prevents waste of data. See FetchConfiguration method `fun preAllocateFileOnCreation(preAllocateFile: Boolean)` to enable or disable this feature. On by default.
7+
6. Added new field segment to Downloader.Request. See java docs for details.
8+
7. Added new Fetch method `fun getContentLengthForRequests(requests:List<Request>, fromServer: Boolean, func: Func<List<Pair<Request,Long>>>, func2: Func<List<Pair<Request, Error>>>): Fetch`
9+
8. Performance improvements to the Parallel downloader mode.
10+
11+
112
Version 3.0.7/ Android x version 3.1.1
213
Added new pauseAll method on Fetch.
314
Added new resumeAll method on Fetch.

‎README.md

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11

22
[![Build Status](https://travis-ci.org/tonyofrancis/Fetch.svg?branch=v2)](https://travis-ci.org/tonyofrancis/Fetch)
3-
[ ![Download](https://api.bintray.com/packages/tonyofrancis/maven/fetch2/images/download.svg?version=3.0.7) ](https://bintray.com/tonyofrancis/maven/fetch2/3.0.7/link)
3+
[ ![Download](https://api.bintray.com/packages/tonyofrancis/maven/fetch2/images/download.svg?version=3.0.8) ](https://bintray.com/tonyofrancis/maven/fetch2/3.0.8/link)
44
[![Android Arsenal](https://img.shields.io/badge/Android%20Arsenal-Android%20Networking-blue.svg?style=flat)](https://android-arsenal.com/details/1/5196)
55
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/tonyofrancis/Fetch/blob/master/LICENSE)
66

@@ -50,11 +50,11 @@ How to use Fetch
5050
Using Fetch is easy! Just add the Gradle dependency to your application's build.gradle file.
5151

5252
```java
53-
implementation "com.tonyodev.fetch2:fetch2:3.0.7"
53+
implementation "com.tonyodev.fetch2:fetch2:3.0.8"
5454
```
5555
Androidx use:
5656
```java
57-
implementation "androidx.tonyodev.fetch2:xfetch2:3.1.1"
57+
implementation "androidx.tonyodev.fetch2:xfetch2:3.1.2"
5858
```
5959

6060
Next, get an instance of Fetch and request a download.
@@ -220,11 +220,11 @@ to use the OkHttp Downloader instead. You can create your own custom downloaders
220220
if necessary. See the Java docs for details.
221221

222222
```java
223-
implementation "com.tonyodev.fetch2okhttp:fetch2okhttp:3.0.7"
223+
implementation "com.tonyodev.fetch2okhttp:fetch2okhttp:3.0.8"
224224
```
225225
Androidx use:
226226
```java
227-
implementation "androidx.tonyodev.fetch2okhttp:xfetch2okhttp:3.1.1"
227+
implementation "androidx.tonyodev.fetch2okhttp:xfetch2okhttp:3.1.2"
228228
```
229229

230230
Set the OkHttp Downloader for Fetch to use.
@@ -246,11 +246,11 @@ If you would like to take advantage of RxJava2 features when using Fetch,
246246
add the following gradle dependency to your application's build.gradle file.
247247

248248
```java
249-
implementation "com.tonyodev.fetch2rx:fetch2rx:3.0.7"
249+
implementation "com.tonyodev.fetch2rx:fetch2rx:3.0.8"
250250
```
251251
Androidx use:
252252
```java
253-
implementation "androidx.tonyodev.fetch2rx:xfetch2rx:3.1.1"
253+
implementation "androidx.tonyodev.fetch2rx:xfetch2rx:3.1.2"
254254
```
255255

256256
RxFetch makes it super easy to enqueue download requests and query downloads using rxJava2 functional methods.
@@ -286,11 +286,11 @@ added in the coming days.
286286

287287
Start using FetchFileServer by adding the gradle dependency to your application's build.gradle file.
288288
```java
289-
implementation "com.tonyodev.fetch2fileserver:fetch2fileserver:3.0.7"
289+
implementation "com.tonyodev.fetch2fileserver:fetch2fileserver:3.0.8"
290290
```
291291
Androidx use:
292292
```java
293-
implementation "androidx.tonyodev.fetch2fileserver:xfetch2fileserver:3.1.1"
293+
implementation "androidx.tonyodev.fetch2fileserver:xfetch2fileserver:3.1.2"
294294
```
295295

296296
Start a FetchFileServer instance and add resource files that it can server to connected clients.
@@ -399,11 +399,11 @@ Fetch1 Migration
399399

400400
Migrate downloads from Fetch1 to Fetch2 using the migration assistant. Add the following gradle dependency to your application's build.gradle file.
401401
```java
402-
implementation "com.tonyodev.fetchmigrator:fetchmigrator:3.0.7"
402+
implementation "com.tonyodev.fetchmigrator:fetchmigrator:3.0.8"
403403
```
404404
Androidx use:
405405
```java
406-
implementation "androidx.tonyodev.fetchmigrator:xfetchmigrator:3.1.1"
406+
implementation "androidx.tonyodev.fetchmigrator:xfetchmigrator:3.1.2"
407407
```
408408

409409
Then run the Migrator.

‎fetch2/src/androidTest/java/com/tonyodev/fetch2/DownloadManagerInstrumentedTest.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ public void useAppContext() throws Exception {
7878
progessInterval, fetchLogger, networkInfoProvider, retryOnNetworkGain,
7979
downloadInfoUpdater, downloadManagerCoordinator,
8080
listenerCoordinator, serverDownloader, false, storageResolver,
81-
appContext, namespace, groupInfoProvider, FetchDefaults.DEFAULT_GLOBAL_AUTO_RETRY_ATTEMPTS);
81+
appContext, namespace, groupInfoProvider, FetchDefaults.DEFAULT_GLOBAL_AUTO_RETRY_ATTEMPTS, false);
8282
}
8383

8484
@After

‎fetch2/src/androidTest/java/com/tonyodev/fetch2/DownloadPriorityIteratorProcessorTest.java

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,8 @@ public void useAppContext() throws Exception {
7575
final DownloadManager downloadManager = new DownloadManagerImpl(client, concurrentLimit,
7676
progessInterval, fetchLogger, networkInfoProvider, retryOnNetworkGain,
7777
downloadInfoUpdater, downloadManagerCoordinator,
78-
listenerCoordinator, serverDownloader, false, storageResolver, appContext, namespace, groupInfoProvider, FetchDefaults.DEFAULT_GLOBAL_AUTO_RETRY_ATTEMPTS);
78+
listenerCoordinator, serverDownloader, false, storageResolver,
79+
appContext, namespace, groupInfoProvider, FetchDefaults.DEFAULT_GLOBAL_AUTO_RETRY_ATTEMPTS, false);
7980
priorityListProcessorImpl = new PriorityListProcessorImpl(
8081
new HandlerWrapper(namespace, null),
8182
new DownloadProvider(databaseManagerWrapper),

‎fetch2/src/androidTest/java/com/tonyodev/fetch2/FetchHandlerInstrumentedTest.java

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,8 @@ public void useAppContext() throws Exception {
9292
final DownloadManager downloadManager = new DownloadManagerImpl(client, concurrentLimit,
9393
progessInterval, fetchLogger, networkInfoProvider, retryOnNetworkGain,
9494
downloadInfoUpdater, downloadManagerCoordinator,
95-
listenerCoordinator, serverDownloader, false, storageResolver, appContext, namespace, groupInfoProvider, FetchDefaults.DEFAULT_GLOBAL_AUTO_RETRY_ATTEMPTS);
95+
listenerCoordinator, serverDownloader, false, storageResolver,
96+
appContext, namespace, groupInfoProvider, FetchDefaults.DEFAULT_GLOBAL_AUTO_RETRY_ATTEMPTS, false);
9697
priorityListProcessorImpl = new PriorityListProcessorImpl(
9798
handlerWrapper,
9899
new DownloadProvider(databaseManagerWrapper),

‎fetch2/src/main/java/com/tonyodev/fetch2/Fetch.kt

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -800,6 +800,22 @@ interface Fetch {
800800
* */
801801
fun getContentLengthForRequest(request: Request, fromServer: Boolean, func: Func<Long>, func2: Func<Error>?): Fetch
802802

803+
/**
804+
* Gets the content Length for each request in the passed in list. If the request or contentLength cannot be found in
805+
* the Fetch database(meaning Fetch never processed the request and started downloading it) -1 is returned.
806+
* However, setting fromServer to true will create a new connection to the server to get the connectLength
807+
* if Fetch does not already contain the data in the database for the request.
808+
* @param requests Request list. Can be a managed or un-managed list of requests. The requests are not stored in
809+
* the fetch database.
810+
* @param fromServer If true, fetch will attempt to get the ContentLength
811+
* from the server directly by making a network request. Otherwise no action is taken.
812+
* @param func callback which returns a list of all the success request pairs with their content length.
813+
* @param func2 callback used to return a list of all the request that error when trying to get their content length.
814+
* @throws FetchException if this instance of Fetch has been closed.
815+
* @return Instance
816+
* */
817+
fun getContentLengthForRequests(requests:List<Request>, fromServer: Boolean, func: Func<List<Pair<Request,Long>>>, func2: Func<List<Pair<Request, Error>>>): Fetch
818+
803819
/**
804820
* Gets the Server Response for the url and associated headers.
805821
* @param url the url. Cannot be null.

‎fetch2/src/main/java/com/tonyodev/fetch2/downloader/ParallelFileDownloaderImpl.kt

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -393,6 +393,11 @@ class ParallelFileDownloaderImpl(private val initialDownload: Download,
393393
if (downloadSpeedCheckTimeElapsed) {
394394
downloadSpeedStartTime = System.nanoTime()
395395
}
396+
try {
397+
Thread.sleep(progressReportingIntervalMillis)
398+
} catch (e: InterruptedException) {
399+
logger.e("FileDownloader", e)
400+
}
396401
}
397402
}
398403

@@ -421,7 +426,7 @@ class ParallelFileDownloaderImpl(private val initialDownload: Download,
421426
downloadBlock.downloadedBytes = fileSlice.downloaded
422427
downloadBlock.startByte = fileSlice.startBytes
423428
downloadBlock.endByte = fileSlice.endBytes
424-
val downloadRequest = getRequestForDownload(downloadInfo, fileSlice.startBytes + fileSlice.downloaded)
429+
val downloadRequest = getRequestForDownload(download = downloadInfo, rangeStart = fileSlice.startBytes + fileSlice.downloaded, segment = fileSlice.position + 1)
425430
var downloadResponse: Downloader.Response? = null
426431
var saveRandomAccessFile: RandomAccessFile? = null
427432
try {

‎fetch2/src/main/java/com/tonyodev/fetch2/downloader/SequentialFileDownloaderImpl.kt

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -342,7 +342,8 @@ class SequentialFileDownloaderImpl(private val initialDownload: Download,
342342
requestMethod = GET_REQUEST_METHOD,
343343
extras = initialDownload.extras,
344344
redirected = false,
345-
redirectUrl = "")
345+
redirectUrl = "",
346+
segment = 1)
346347
}
347348

348349
private fun getAverageDownloadedBytesPerSecond(): Long {

‎fetch2/src/main/java/com/tonyodev/fetch2/fetch/FetchImpl.kt

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1063,6 +1063,34 @@ open class FetchImpl constructor(override val namespace: String,
10631063
}
10641064
}
10651065

1066+
override fun getContentLengthForRequests(requests: List<Request>, fromServer: Boolean, func: Func<List<Pair<Request, Long>>>, func2: Func<List<Pair<Request, Error>>>): Fetch {
1067+
synchronized(lock) {
1068+
throwExceptionIfClosed()
1069+
handlerWrapper.executeWorkerTask {
1070+
val results = mutableListOf<Pair<Request, Long>>()
1071+
val results2 = mutableListOf<Pair<Request, Error>>()
1072+
for (request in requests) {
1073+
try {
1074+
results.add(Pair(request, fetchHandler.getContentLengthForRequest(request, fromServer)))
1075+
} catch (e: Exception) {
1076+
logger.e("Fetch with namespace $namespace error", e)
1077+
val error = getErrorFromMessage(e.message)
1078+
error.throwable = e
1079+
results2.add(Pair(request, error))
1080+
}
1081+
}
1082+
uiHandler.post {
1083+
func.call(results)
1084+
}
1085+
uiHandler.post {
1086+
func2.call(results2)
1087+
}
1088+
}
1089+
return this
1090+
}
1091+
}
1092+
1093+
10661094
override fun getServerResponse(url: String,
10671095
headers: Map<String, String>?,
10681096
func: Func<Downloader.Response>,

‎fetch2/src/main/java/com/tonyodev/fetch2/util/FetchUtils.kt

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,8 @@ fun getRequestForDownload(download: Download, requestMethod: String = GET_REQUES
5555
fun getRequestForDownload(download: Download,
5656
rangeStart: Long = -1,
5757
rangeEnd: Long = -1,
58-
requestMethod: String = GET_REQUEST_METHOD): Downloader.ServerRequest {
58+
requestMethod: String = GET_REQUEST_METHOD,
59+
segment: Int = 1): Downloader.ServerRequest {
5960
val start = if (rangeStart == -1L) 0 else rangeStart
6061
val end = if (rangeEnd == -1L) "" else rangeEnd.toString()
6162
val headers = download.headers.toMutableMap()
@@ -71,7 +72,8 @@ fun getRequestForDownload(download: Download,
7172
requestMethod = requestMethod,
7273
extras = download.extras,
7374
redirected = false,
74-
redirectUrl = "")
75+
redirectUrl = "",
76+
segment = segment)
7577
}
7678

7779
fun getServerRequestFromRequest(request: Request): Downloader.ServerRequest {
@@ -86,7 +88,8 @@ fun getServerRequestFromRequest(request: Request): Downloader.ServerRequest {
8688
fileUri = getFileUri(request.file),
8789
extras = request.extras,
8890
redirected = false,
89-
redirectUrl = "")
91+
redirectUrl = "",
92+
segment = 1)
9093
}
9194

9295
fun getCatalogServerRequestFromRequest(request: Request): Downloader.ServerRequest {
@@ -106,7 +109,8 @@ fun getCatalogServerRequestFromRequest(request: Request): Downloader.ServerReque
106109
fileUri = getFileUri(request.file),
107110
extras = request.extras,
108111
redirected = false,
109-
redirectUrl = "")
112+
redirectUrl = "",
113+
segment = 1)
110114
}
111115

112116
fun getPreviousSliceCount(id: Int, fileTempDir: String): Int {

‎fetch2core/src/main/java/com/tonyodev/fetch2core/Downloader.kt

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -172,7 +172,13 @@ interface Downloader<T, R> : Closeable {
172172
val redirected: Boolean,
173173

174174
/** redirect url*/
175-
val redirectUrl: String)
175+
val redirectUrl: String,
176+
177+
/**
178+
* If the request will be downloaded using the parallel download and the file slice count is greater the 1. This field will indicate
179+
* the request segment/part.
180+
* */
181+
val segment: Int)
176182

177183
/**
178184
* A class that contains the server response information used by Fetch

‎fetch2okhttp/src/main/java/com/tonyodev/fetch2okhttp/OkHttpDownloader.kt

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,8 @@ open class OkHttpDownloader @JvmOverloads constructor(
7272
requestMethod = oldRequest.requestMethod,
7373
extras = oldRequest.extras,
7474
redirected = true,
75-
redirectUrl = redirectUrl)
75+
redirectUrl = redirectUrl,
76+
segment = oldRequest.segment)
7677
}
7778

7879
override fun execute(request: Downloader.ServerRequest, interruptMonitor: InterruptMonitor): Downloader.Response? {

‎fetch2rx/src/main/java/com/tonyodev/fetch2rx/RxFetch.kt

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -514,6 +514,20 @@ interface RxFetch {
514514
* */
515515
fun getContentLengthForRequest(request: Request, fromServer: Boolean): Convertible<Long>
516516

517+
/**
518+
* Gets the content Length for each request in the passed in list. If the request or contentLength cannot be found in
519+
* the Fetch database(meaning Fetch never processed the request and started downloading it) -1 is returned.
520+
* However, setting fromServer to true will create a new connection to the server to get the connectLength
521+
* if Fetch does not already contain the data in the database for the request.
522+
* @param requests Request list. Can be a managed or un-managed list of requests. The requests are not stored in
523+
* the fetch database.
524+
* @param fromServer If true, fetch will attempt to get the ContentLength
525+
* from the server directly by making a network request. Otherwise no action is taken.
526+
* @throws FetchException if this instance of Fetch has been closed.
527+
* @return convertible containing the success and error result list.
528+
* */
529+
fun getContentLengthForRequests(requests:List<Request>, fromServer: Boolean): Convertible<Pair<List<Pair<Request, Long>>, List<Pair<Request, Error>>>>
530+
517531
/**
518532
* Gets the Server Response for the url and associated headers.
519533
* @param url the url. Cannot be null.

‎fetch2rx/src/main/java/com/tonyodev/fetch2rx/RxFetchImpl.kt

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -987,6 +987,32 @@ open class RxFetchImpl(override val namespace: String,
987987
}
988988
}
989989

990+
override fun getContentLengthForRequests(requests: List<Request>, fromServer: Boolean): Convertible<Pair<List<Pair<Request, Long>>, List<Pair<Request, Error>>>> {
991+
return synchronized(lock) {
992+
throwExceptionIfClosed()
993+
Flowable.just(Pair(requests, fromServer))
994+
.subscribeOn(AndroidSchedulers.from(handlerWrapper.getWorkTaskLooper()))
995+
.flatMap {
996+
throwExceptionIfClosed()
997+
val results = mutableListOf<Pair<Request, Long>>()
998+
val results2 = mutableListOf<Pair<Request, Error>>()
999+
for (request in requests) {
1000+
try {
1001+
results.add(Pair(request, fetchHandler.getContentLengthForRequest(request, fromServer)))
1002+
} catch (e: Exception) {
1003+
logger.e("RxFetch with namespace $namespace error", e)
1004+
val error = getErrorFromMessage(e.message)
1005+
error.throwable = e
1006+
results2.add(Pair(request, error))
1007+
}
1008+
}
1009+
Flowable.just(Pair(results as List<Pair<Request, Long>>, results2 as List<Pair<Request, Error>>))
1010+
}
1011+
.observeOn(uiScheduler)
1012+
.toConvertible()
1013+
}
1014+
}
1015+
9901016
override fun getServerResponse(url: String, headers: Map<String, String>?): Convertible<Downloader.Response> {
9911017
return synchronized(lock) {
9921018
throwExceptionIfClosed()

‎versions.gradle

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
ext {
2-
kotlin_version = '1.3.31'
2+
kotlin_version = '1.3.40'
33
okhttp_version = '3.12.3'
44
okhttp_url_version = '3.11.0'
55
android_support_version = '28.0.0'
@@ -13,10 +13,10 @@ ext {
1313
library_target_version = 28
1414
library_build_tools_version = "28.0.3"
1515
gradle_tools_version = '3.4.1'
16-
rxJava2_version = "2.2.8"
16+
rxJava2_version = "2.2.10"
1717
rxAndroid2_version = "2.1.1"
1818
timber_version = "4.7.1"
1919
novoda_bintray_version = "0.9"
20-
library_version = "3.0.7"
21-
library_version_code = 70
20+
library_version = "3.0.8"
21+
library_version_code = 71
2222
}

0 commit comments

Comments
 (0)