millironx/nf-configs
1
0
Fork 0
mirror of https://github.com/MillironX/nf-configs.git synced 2024-11-14 05:33:09 +00:00
Code Issues Releases Packages Activity

Merge pull request #417 from Sage-Bionetworks-Workflows/bgrande/sage-improve-downloads

Browse source 
Improve reliability of file transfers with retries and reduced concurrency
This commit is contained in:
Bruno Grande 2022-09-26 10:24:14 -07:00 committed by GitHub
commit 57f63add02
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 4 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
conf/sage.config
View file

@ -29,9 +29,12 @@ aws {
region = "us-east-1"
client {
uploadChunkSize = 209715200
uploadMaxThreads = 4
}
batch {
maxParallelTransfers = 1
maxTransferAttempts = 5
delayBetweenAttempts = '120 sec'
}
}
executor {

1
docs/sage.md
View file

@ -8,6 +8,7 @@ This global configuration includes the following tweaks:
- Enable retries by default when exit codes relate to insufficient memory
- Allow pending jobs to finish if the number of retries are exhausted
- Increase the amount of time allowed for file transfers
- Improve reliability of file transfers with retries and reduced concurrency
- Increase the default chunk size for multipart uploads to S3
- Slow down job submission rate to avoid overwhelming any APIs
- Define the `check_max()` function, which is missing in Sarek v2