AWS error when trying to run pipeline

[rolayoalarcon]

Hi! First of all, I have to say congrats on the pipeline! Seems very cool. I was very excited to use it. However, I get an error that I don't understand when using it.

I execute the following command (Docker and Singularity not available
nextflow run nf-core/chipseq --input design.csv --genome GRCh38 -profile conda

I get the following error

N E X T F L O W ~ version 21.04.0
Launching nf-core/chipseq [awesome_venter] - revision: 6924b66 [master]
The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: VQ29H8S12AEMRPC5; S3 Extended Request ID: pPuNXfvXsgXJlMOj5vwGbDgxaQDqCTbJCz48u8KMt37WgGcy60gPok/c9HKMPylhGlQxn4zZBik=)

-- Check script '/home/roberto_olayo/.nextflow/assets/nf-core/chipseq/main.nf' at line: 171 or see '.nextflow.log' file for more details

I did a few tests and I get this message even when -profile docker, so I don't think it's a conda thing. Also, the genome version is not the issue. Finally -profile test,conda works fine.

I've attached the log file in case it helps. As far as I can see it creates the aws session to handle a file. I recently cancelled my aws account; is there a way to avoid this?
nextflow_log.txt

Thanks for taking a look!

[klkeys]

did you verify that the pipeline can see valid AWS credentials? you need something to enable S3 command line access.

if you aren't running the pipeline on AWS itself then my guess is that Nextflow could not pull GRCh38 files from the iGenomes bucket on S3. if you canceled your AWS account then that might be a problem.

you could test this by downloading the relevant files and pointing Nextflow to local counterparts; see here.

[JoseEspinosa]

I close the issue since it has be already answered ☝️

[Aravind-sunda]

I'm experiencing an issue with my pipeline run. When I start the ChIP-seq pipeline, it takes about 40 minutes to begin. The Nextflow log repeatedly shows lines like:

Aug-05 14:41:47.955 [main] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:41:49.051 [main] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:41:51.191 [main] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:41:52.251 [main] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:41:54.420 [S3TransferManager-1] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:41:55.515 [S3TransferManager-1] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:41:57.661 [S3TransferManager-1] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:41:58.716 [S3TransferManager-2] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:42:00.719 [S3TransferManager-1] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:42:01.787 [S3TransferManager-2] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:42:03.791 [S3TransferManager-1] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:42:04.860 [S3TransferManager-3] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:42:06.863 [S3TransferManager-1] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:42:07.931 [S3TransferManager-3] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:42:09.935 [S3TransferManager-4] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:42:11.004 [S3TransferManager-1] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:42:13.007 [S3TransferManager-4] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:42:14.076 [S3TransferManager-1] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:42:16.208 [S3TransferManager-1] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:42:17.276 [S3TransferManager-5] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access Aug-05 14:42:19.278 [S3TransferManager-1] DEBUG n.c.aws.util.S3CredentialsProvider - No AWS credentials available - falling back to anonymous access

I suspect this is related to the lengthy startup time for the pipeline. Could you suggest a way to avoid this issue?