Chris Pruett's picture
Chris Pruett - Tue, 2016/01/26 - 22:10

TKLBAM backups to S3 are simply not working for me.  I would appreciate some guidance.

Here's what happens when I try the backup to S3:

Uploading s3://s3.amazonaws.com/tklbam-ezpl7752hiwwe27a/duplicity-full.20160126T191540Z.vol1.difftar.gpg to STANDARD Storage

Upload 's3://s3.amazonaws.com/tklbam-ezpl7752hiwwe27a/duplicity-full.20160126T191540Z.vol1.difftar.gpg' failed (attempt #1, reason: error: [Errno 104] Connection reset by peer)

... over and over until it gives up.  It does not appear to have even created the tklbam bucket.

What I've done:

I have a local install of a TKL image on a VM.  Works great. 

I have an AWS account and I believe I've set up the role/property stuff up properly.  I've initialized TKLBAM with the API key.  I even attached the AmazonS3FullAccess policy to the turnkeyhub role and it still failed:

Failed to create bucket (attempt #1) 'tklbam-ezpl7752hiwwe27a' failed (reason: error: [Errno 104] Connection reset by peer)
...

I installed s3cmd on the machine in question as was able to create buckets and poke around from the command line, so I know that the machine can theoretcially access S3.  There's not a firewall in the way or something.

I'm able to use TKLBAM to backup to local storage.

 

Forum: 
Support
Tags: 
tklbam
amazon s3
Jeremy Davis's picture

Jeremy Davis - Thu, 2016/01/28 - 02:32 
turnkey-version
apt-get update && apt-cache policy tklbam

I'm not an expert in these things but AFAIK "Connection reset by peer" is essentially the server refusing the connection (cancelling it). My guess at possible causes:

  • a networking issue between your server and Amazon
  • an issue with your IAMs role
  • some weird TKLBAM edge case

    When you connected to AWS S3 did you use the same IAMs role that was setup for the Hub? If not then my guess is something wrong with your IAMs role and the Hub authentication setup. I recommend that you log into AWS console and remove all the IAMs stuff you added for TKLBAM. Then log back into the Hub, remove what you have added there too and start again. You just need to copy paste from the Hub into your AWS console. The instructions should be fairly good but if you get stuck please ask.

    • Chris Pruett's picture

    Chris Pruett - Wed, 2016/01/27 - 18:33

    Thanks for helping.  Working through your suggestions:

    root@azure ~# turnkey-version

turnkey-gitlab-14.0-jessie-amd64

    I updated all the packages and rebooted the server. Tried again, no change.

    I redid and checked the role/policy stuff.  Hub shows green on the connectivity checks. (I actually cannot find the detailed AWS setup instructions on the hub that I originally followed.  Do you have a link?)

    Deleted /var/lib/tklbam and ran tklbam-init and then tklbam-backup.

    This time it managed to create a bucket, confirming access to S3, but the connection was still reset.

    Agree that this looks like a networking issue.  I will try doing some larger data transfers with s3cmd.

     

    Jeremy Davis's picture

    Jeremy Davis - Thu, 2016/01/28 - 03:24
    I just had a dig around in the Hub myself and you're right, I can't see how to re-enter the AWS info again... Although if TKLBAM managed to create a bucket this time that would suggest that authorisation is not the issue (so perhaps just leave that be for now).

    Also it might be worth testing from the commandline inside your appliance. I.e. install awscli. Official AWS-cli install docs Arguably easier instructions.

    Chris Pruett's picture

    Chris Pruett - Thu, 2016/01/28 - 23:19

    Tested some data transfers using s3cmd.  No problems. ('azure' is the TKL VM that cannot do TKLBAM backups to S3)

    root@azure ~# s3cmd put /mnt/amethyst_storage/backups/azure/duplicity-full.20160123T202128Z.vol1.difftar.gpg s3://tklbam-zg5ykrw7a4znn3j2/test.gpg
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
/mnt/amethyst_storage/backups/azure/duplicity-full.20160123T202128Z.vol1.difftar.gpg -> s3://tklbam-zg5ykrw7a4znn3j2/test.gpg  [1 of 1]
 9391284 of 9391284   100% in   10s   915.66 kB/s  done

root@azure ~# s3cmd ls s3://tklbam-zg5ykrw7a4znn3j2
2016-01-28 21:07   9391284   s3://tklbam-zg5ykrw7a4znn3j2/test.gpg

root@azure ~# s3cmd del s3://tklbam-zg5ykrw7a4znn3j2/test.gpg
File s3://tklbam-zg5ykrw7a4znn3j2/test.gpg deleted

root@azure ~# s3cmd ls s3://tklbam-zg5ykrw7a4znn3j2

     

    So I can clearly access and transfer files to/from S3.  I even used one of the TKLBAM backup files for test data.  That rules out network connectivity

    I suppose that s3cmd is different than duplicity/TKLBAM. I need to figure out how to debug duplicity/TKLBAM...

    Jeremy Davis's picture

    Jeremy Davis - Fri, 2016/01/29 - 05:07
    I'm not familiar with s3cmd (I've only used awscli) but it certainly does appear to rule out network connectivity issues.

    Can you (or did you) use the Hub IAMs user that you set up for the Hub?

    If you did and that worked then it pretty much proves that its a TKLBAM/Hub bug of some sort.

    OTOH if you didn't use the same credentials, it still doesn't rule out an issue with the IAMs role in the Hub. TBH I still think that's the cause (mostly because no one else is having this issue right now; at least no one has reported it). Although I acknowledge the fact that TKLBAM was (at least at one point) able to create an S3 bucket does make it seem tenuous...

    If possible (and if you haven't already), can you please try authenticating s3cmd with the Hub's IAMs user . If that's not possible then try aws-cli (I'm 99% sure that you can use an IAMs role with that).

    If you can confirm that you can access and write to your S3 bucket using the Hub's IAMs user then please send me an email via the Hub's feedback (log into the Hub and click the "feedback" top row, towards the left). I'll then escalate this to Liraz (the TKLBAM Daddy).

    Chris Pruett's picture

    Chris Pruett - Fri, 2016/01/29 - 20:00

    I was using my own credentials with s3cmd.  I'll try to use the Hub's IAMs user with s3cmd and/or awscli and report back.

    Thanks for the suggestions.

    Add new comment