millironx/nf-configs
1
0
Fork 0
mirror of https://github.com/MillironX/nf-configs.git synced 2024-11-24 17:19:54 +00:00
Code Issues Releases Packages Activity

Merge pull request #417 from Sage-Bionetworks-Workflows/bgrande/sage-improve-downloads

Browse source 
Improve reliability of file transfers with retries and reduced concurrency
This commit is contained in:
Bruno Grande 2022-09-26 10:24:14 -07:00 committed by GitHub
commit 57f63add02
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 4 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
conf/sage.config
View file

@ -29,9 +29,12 @@ aws {
region = "us-east-1" region = "us-east-1"
client { client {
uploadChunkSize = 209715200 uploadChunkSize = 209715200
uploadMaxThreads = 4
} }
batch { batch {
maxParallelTransfers = 1 maxParallelTransfers = 1
maxTransferAttempts = 5
delayBetweenAttempts = '120 sec'
} }
} }
executor { executor {

1
docs/sage.md
View file

@ -8,6 +8,7 @@ This global configuration includes the following tweaks:
- Enable retries by default when exit codes relate to insufficient memory - Enable retries by default when exit codes relate to insufficient memory
- Allow pending jobs to finish if the number of retries are exhausted - Allow pending jobs to finish if the number of retries are exhausted
- Increase the amount of time allowed for file transfers - Increase the amount of time allowed for file transfers
- Improve reliability of file transfers with retries and reduced concurrency
- Increase the default chunk size for multipart uploads to S3 - Increase the default chunk size for multipart uploads to S3
- Slow down job submission rate to avoid overwhelming any APIs - Slow down job submission rate to avoid overwhelming any APIs
- Define the `check_max()` function, which is missing in Sarek v2 - Define the `check_max()` function, which is missing in Sarek v2