Scott C's picture
Scott C - Thu, 2011/11/10 - 00:40

My TKBLM Dashboard is not showing the latest backups. Shows only the backup from 11/8/2011 and none from 11/9/2011. Is there an issue anyone is aware of?

Forum: 
Support
Scott C's picture

Scott C - Thu, 2011/11/10 - 22:59

Looks like it is now updated today, and shows the ones that were missing from yesterday. Might have just been a glitch on TKLBAM yesterday.


L. Arnold's picture

L. Arnold - Fri, 2011/11/11 - 00:08

i don't see the Backups from 11-10-11 AM on my Hub Account. Sounds like they are taking.  My question is what this does to the incremental backup system if one of the images is missing.

I don't think you can delete one image from your backup set and i don't know what is controlling duplicity, though i imagine it is the TKL install itself rather than the hub.

if TKLBAM thinks there is a backup, what will it do when it does not find it during restore?

Jeremy Davis's picture

Jeremy Davis - Fri, 2011/11/11 - 01:06

Are all your backups now listed on TKLBAM too? If so then all good.

In answer to your question, if an incremental backup is missing from the chain then that is very bad! The backup can not be restored unless all of the links in the chain (incremental backups) exist from the incremental backup you are trying to restore, all the way back to the preceeding full backup. That is why it is important that the chain isn't too long.

By default TKLBAM is set to monthly full backups (with no incrementals). I turn daily incrementals on, but set full backups to run every 2 weeks (see here). That way the chain is only ever 13 links long (13 incremental daily backups + 1 full backup = 14 days worth of backups). If you wanted hourly incrementals, personally I'd be looking at full backups every 12 hours. I'm not sure whether I'm being overly paranoid, but bottom line is that the shorter the chain the better IMO. I think it's also good practice to test your backups once every few full backs (so for my setup ideally I'd do it once every month or 2).

I'm not 100% sure what happened to the Hub (re not showing up all backups) but AFAIK the Hub is only displaying info, the important part is that TKLBAM itself is seeing all the incremental backups. Which from my understanding it did.

And yes Duplicity is controled by the local instance of TKLBAM (not the Hub). AFAIK, as long as your local TKLBAM knows what's going on, and the backups exist in your S3 container (assuming you are using the default TKLBAM storage option) then all will be well.

Jeremy Davis's picture

Jeremy Davis - Fri, 2011/11/11 - 04:39

I just ran a (full) backup (on a new machine) and it showed up in the Hub straight away. So not at all clear what/where the problem is (I haven't tried doing a new backup of an existing server yet).

I'm hoping one of the core devs will be along shortly to try to have a look at this. Perhaps even to clarify whether or not this is a major issue (ie backups not propagating to Hub and therefore dificult to restore if need be - and they need to get straight onto it) or only a minor issue (ie backups still ok, but just Hub not updating properly - but will catchup soon and/or they will fix when they have time).

[update] I suspect that this is related to this bug registered recently on the bug tracker.

Alon Swartz's picture

Alon Swartz - Fri, 2011/11/11 - 11:46

We've been performing some Hub maintenance these last couple of days, so the TKLBAM/Hub cache sync was probably not being triggered. The way it works is as a backup is completed, TKLBAM sends a trigger to the Hub to update its cache. Due to the maintenance the trigger was probably not being handled.

Just to be clear, there is no need for concern. If the TKLBAM backup completed successfully then the backup itself was successfull. The Hub's cache is just probably out of date. Once the issue resolved all backup sessions will be displayed.

Sorry for the late reply and reporting the issue!

Alon Swartz's picture

Alon Swartz - Fri, 2011/11/11 - 12:24

I just flushed the trigger queue so all backups should be updated. If your backups aren't updated, please send feedback via the Hub and I'll manually trigger the cache update for your account.

[correction] the queue is still being processed, so give it a couple of minutes before reporting out-of-sync backup accounts.

Jeremy Davis's picture

Jeremy Davis - Sat, 2011/11/12 - 19:01

I forgot to post back that I successfully restored a backup that was not listed correctly on the Hub. It was a fresh backup that I did and the Hub reported (actually still reports) "First backup in progress..." with a filesize of 0 bytes. But i successfuly restored it (migrated it) to a new appliance - no worries.

Actually there were a few things I found during this process, but they are irrelevant to this issue and I have lodged (non-urgent) bug reports.

Andre Derraik's picture

Andre Derraik - Fri, 2011/11/11 - 14:11

Hi everyone!

You're doing a Great Job!! Congratulations.

But, until now, no updates.

I made 3 backups today, thinking I missing somthing. Until to search the forum for support.

And found this.

When the Hub updates, I'll feedback.

Thanks.

[Adding Info]

[desenv@desenv:~] sudo tklbam list
[sudo] password for desenv: 
# ID  SKPP  Created     Updated     Size (MB)  Label
   3  No    2011-06-03  2011-10-28  524.42     DeepLook
[desenv@desenv:~] sudo tklbam-status
TKLBAM:  Backup ID #3, Updated Fri 2011-11-11 09:41

Andre Derraik's picture

Andre Derraik - Fri, 2011/11/11 - 15:19

Now the HUB update...




	

		88.6 KB (inc)

	

		Fri, 11 Nov. 2011 11:11:54 UTC





	

		368.7 KB (inc)

	

		Fri, 11 Nov. 2011 11:11:30 UTC





	

		256.1 MB (full)

	

		Fri, 11 Nov. 2011 11:11:21 UTC





	Thanks.
          


          

              

      






  

      

    L. Arnold's picture  

  


  

    
    


    

      L. Arnold - Fri, 2011/11/11 - 16:50    


    

      

	Thank you Alon for performing your magic! 



	Jeremy, Alon and Team.  So if we want to 'shorten" the number of links can we do that at any point?



	More to the question though, how do we "stop" taking Backups to one Backup id, and start issuing to a new id?  Or even better, how might we toggle to a different Backup id, in the case for instance of a Quarter or Year End Closing that we want to have access to but not load data onto on a regular basis?



	Backups are great, but whether via TKLBAM or, with a program like Retrospect, keeping track of it all is quite a chore and I am sure the back end data mapping is absolutely Byzantine.
          


          

              

      








  

      

    Jeremy Davis's picture  

  


  

    
    


    

      Jeremy Davis - Sat, 2011/11/12 - 00:27    


    

      

	But then you end up storing more data. Both methods have advantages and disadvantages, so like most things in life, there's always a trade off. Full backups take up more storage space and take longer to perform. Incremental backups are much faster to perform but are somewhat vulnerable in as much as if one gets corrupted, or goes missing then the chain back to the full backup is lost.



	My personal take on that (as I said in my post above) is to make full backups more frequently thus reducing the number of incrementals between them. You don't need to do anything more than that. You don't need to start taking backups to a new ID to reduce the chain length.



	But if I get you correctly, you want to 'start again' without all of the old data, but most of the config stuff? That should be doable, but will involve a bit of work. Creating a new ID is as simple as migrating your data to a new server and doing a full backup - a new ID is automatically created. But if you wished to purge all old info, then you would have to manually do that prior to running your first backup. If you document the steps you take, you may be able turn this into a script. So this is the workflow I would suggest to acheive what (I think) you're after:


    
	
  1. 
		Run a full backup of Server A (your current primary appliance).
    2. 
	
  2. 
		Restore the backup to Server B (your new appliance).
    3. 
	
  3. 
		Manually remove all data that is no longer needed. Document this step carefully.
    4. 
	
  4. 
		Run a full backup of Server B (which now your new current primary appliance) - the new backup ID is automatically created.
    5. 
	
  5. 
		As a final step I would recommend changing the name of the old backup so it is clear what it is (eg add the financial year that it applies to).
    6. 

          


          

              

      







  

      

    Scott C's picture  

  


  

    
    


    

      Scott C - Fri, 2011/11/11 - 19:09    


    

      

	All good for me now as well. Thanks!
              

          
        

          


          

              

      






  

      

    Jeremy Davis's picture  

  


  

    
    


    

      Jeremy Davis - Sat, 2011/11/12 - 20:58    


    

      

	I made 2 backups yesterday (my time - +11 GMT) within about an hour of one another.



	The first one, the Hub states occured "Fri, 11 Nov. 2011 02:11:36 UTC"



	The second that occurred within an hour (or perhaps 2 at the outside) still says "First backup in progress..." but it took about 2 minutes!



	TBH I'm not too fussed personally because I know that the data is there and I have confirmed that backups that the Hub states haven't completed restore fine. But still I could imagine that it may be a little unnerving for some!
          


          

              

      








  

      

    Alon Swartz's picture  

  


  

    
    


    

      Alon Swartz - Sun, 2011/11/13 - 14:48    


    

      

	The queue got "stuck" again, argh! I just "flushed" the queue and its running all the updates as I write this. We are still looking into what is causing this, and will issue a fix ASAP. 



[update]: I'm almost certain the issue has been solved for good. The backlog of tasks are running so it should take a couple of minutes to get through all of them. If anyone still has an issue later in the day please let me know...
          


          

              

      






  

      

    Jeremy Davis's picture  

  


  

    
    


    

      Jeremy Davis - Sun, 2013/07/21 - 05:26    


    

      
IIRC correctly it was only the Hub that was stuck, not the actual backups themselves... If you have a look at your available backups from the appliance itself (either within Webmin or from the commandline) hopefully the real situation should be apparent...


Also you could inform the devs via the Hub feedback (blue button on the left hand side when logged into the Hub).
          


          

              

      





      
Add new comment