Christoph Knittel's picture
Christoph Knittel - Tue, 2014/01/21 - 18:52

Hi all,

I just discovered that, in the OpenLDAP appliance, the ldap database (located in /var/lib/ldap) is not backed up by tklbam.

I consider this a hazard, as I would definitely expect an OpenLDAP appliance to back up its ldap database by default.

Adding /var/lib/ldap to /etc/tklbam/overrides fixes the problem, but I am not sure whether it is ok to backup the DB while slapd is running.

Christoph

Forum: 
Support
Tags: 
tklbam
ldap
backup
openldap
Jeremy Davis's picture

Jeremy Davis - Wed, 2014/03/12 - 13:00

If you use the '-s' switch (for simulate) it will show you everything that will be backed up...

If you have some suggestions on what should be being backed up but isn't please feel free to detail it.

Christoph Knittel's picture

Christoph Knittel - Tue, 2014/03/11 - 08:48

In the meantime I found that it is definitely not ok to backup or restore /var/lib/ldap without shutting down slapd first.

Therefore I added a tklbam hook /etc/tklbam/hooks.d/ldap as follows:

#!/bin/bash

tklbam_operation=$1
tklbam_state=$2

if [ "$tklbam_state" == "pre" ]; then
    service slapd stop
elif [ "$tklbam_state" == "post" ]; then
    service slapd restart
fi
Jeremy Davis's picture

Jeremy Davis - Wed, 2014/03/12 - 13:15

Thanks for posting this. Sounds like it should be included in the TKLBAM OpenLDAP profile.

Also I have just raised a bug on the TKL Isue Tracker.

Christoph Knittel's picture

Christoph Knittel - Sun, 2014/03/16 - 11:17

Howard, thanks for the hint! Unfortunately I am not an expert on OpenLDAP or BerkeleyDB.

In the OpenLDAP admin guide, it says

19.4. Migration

The simplest steps needed to migrate between versions or upgrade, depending on your deployment type are:

  1. Stop the current server when convenient 
  2. slapcat the current data out 
  3. Clear out the current data directory (/usr/local/var/openldap-data/) leaving DB_CONFIG in place 
  4. Perform the software upgrades
  5. slapadd the exported data back into the directory
  6. Start the server

So I guess

  • this procedure could be used in the tklbam pre/post hooks and
  • this would the safest way to go (especially in the case of restoring to a new instance of the OpenLDAP appliance that has a newer version of slapd).

Do you have any sample backup/restore scripts based on slapcat/slapadd and covering all the dbs?

Eric's picture

Eric - Tue, 2014/05/06 - 22:27

The LDAP schema's and index definitions are normally restored because they are stored in "/etc/ldap/slapd.d/" (and below).

To backup the LDAP data fields I use the (modified example) script "/etc/tklbam/hooks.d/ldap" (Note: /root/ldap/backup.ldif wil be normally backuped and restored):

#!/bin/bash -e
op=$1
state=$2
if [ "$state" = "pre" ]; then
    service slapd stop
elif [ "$state" = "inspect" ]; then
    if [ "$op" = "restore" ]; then
    elif [ "$op" = "backup" ]; then
        slapcat -l /root/ldap/backup.ldif
    fi
elif [ "$state" = "post" ]; then
    service slapd restart
else
    echo "bad hook invocation"
fi

I manually restore the LDAP data fields because a clean appliance doesn't have the ldap hook script with the following script:

#!/bin/bash
service slapd stop
rm -f /var/lib/ldap/*
slapadd -F /etc/ldap/slapd.d/ -l /root/ldap/backup.ldif
chown openldap:openldap /var/lib/ldap/*
service slapd start

Regards

Eric's picture

Eric - Thu, 2014/05/08 - 01:12

Thanks to point this out, I have changed the scripts as follows and it works great.

Hook:

#!/bin/bash -e
op=$1
state=$2
if [ "$state" = "pre" ]; then
    service slapd stop
elif [ "$state" = "inspect" ]; then
    if [ "$op" = "restore" ]; then
        echo "Only manual restore"
    elif [ "$op" = "backup" ]; then
        slapcat -n0 -l /root/ldap/config.ldif
        slapcat -l /root/ldap/data.ldif
    fi
elif [ "$state" = "post" ]; then
    service slapd restart
else
    echo "bad hook invocation"
fi

Restore

#!/bin/bash
service slapd stop
# config
rm -rf /etc/ldap/slapd.d/*
slapadd -n0 -F /etc/ldap/slapd.d/ -l /root/ldap/config.ldif
chown -R openldap:openldap /etc/ldap/slapd.d/*
# data
rm -rf /var/lib/ldap/*
slapadd -F /etc/ldap/slapd.d/ -l /root/ldap/data.ldif
chown -R openldap:openldap /var/lib/ldap/*
service slapd start

 

Jeremy Davis's picture

Jeremy Davis - Thu, 2014/05/08 - 02:06

I have added additional notes to the bug report referring back to this thread.

Christoph Knittel's picture

Christoph Knittel - Thu, 2014/05/08 - 08:48

Hi guys,

Thanks a lot for your work! I would have looked into this myself after Howard's comments, but didn't find the time. I will test the latest version of Eric's scripts and let you know my findings.

Christoph

Eric's picture

Eric - Thu, 2014/05/08 - 13:10

Using my scripts I discover that the first time I make a backup the config.ldif and data.ldif files doesn't appear in the backup! In following backup's they are in the backup.

This means, to my opinion, that the check of files to be backuped is done before calling the hook scripts. I don't know where to change the order.

Christoph Knittel's picture

Christoph Knittel - Fri, 2014/05/09 - 12:15

I can confirm the issue that Eric mentions above. It can be resolved by performing the backup in the operation=backup / state=pre phase.

Also, in Eric's script, an error would occur if the backup directory didn't exist yet.

Here is my latest version of the script with backup and restore working perfectly for me: :-)

#!/bin/bash

tklbam_operation=$1
tklbam_state=$2

ldap_backup_dir=/root/ldap_backup

if [ "$tklbam_operation" == "backup" ] && [ "$tklbam_state" == "pre" ]; then
    service slapd stop

    mkdir -p $ldap_backup_dir
    slapcat -n0 -l $ldap_backup_dir/config.ldif
    slapcat -l $ldap_backup_dir/data.ldif

    service slapd start
fi

if [ "$tklbam_operation" == "restore" ] && [ "$tklbam_state" == "post" ]; then
    service slapd stop

    # restore config
    rm -rf /etc/ldap/slapd.d/*
    slapadd -n0 -F /etc/ldap/slapd.d/ -l $ldap_backup_dir/config.ldif
    chown -R openldap:openldap /etc/ldap/slapd.d/*

    # restore data
    rm -rf /var/lib/ldap/*
    slapadd -F /etc/ldap/slapd.d/ -l $ldap_backup_dir/data.ldif
    chown -R openldap:openldap /var/lib/ldap/*

    service slapd start
fi
Eric's picture

Eric - Sun, 2014/05/11 - 21:26

I can confirm that the script above works, thanks Christoph.

I have noticed errors from the "libnss-ldapd" package during backup, so I stop/start also the "nslcd" service.

Another point I found is that during "--simulate" of tklbam-backup and tklbam-restore the script is normally executed :-( Is it possible that the script is aware of the "--simulate" option?

Liraz Siri's picture

Liraz Siri - Sat, 2014/06/07 - 20:39
Thanks to everyone who helped develop and test this hook. Awesome work guys.

I monkey patched the TKLBAM profiles for OpenLDAP that live on the Hub to include this hook so that that slapd migration happens out of the box and you don't need to install hooks to /etc/tklbam/hooks.d

It's been possible to do this since TKLBAM v1.4. From the RelNotes-1.4: 

* Cryptographically signed profile-level hooks are now supported. It's a
  generic mechanism but the original motivation for this was to allow us
  to automatically fix migrations between two versions of an appliance
  (E.g., TurnKey Redmine 12 and TurnKey Redmine 13)
Redmine was the first and only use case up until now. Thanks for helping me add a second one!

So that I don't have to monkey patch tklbam profiles for future versions I added the final version of the hook by Christoph to tklbam-profiles:

https://github.com/turnkeylinux/tklbam-profiles/commit/d12462fa18d2f721a...

Jeremy Davis's picture

Jeremy Davis - Sun, 2014/06/08 - 02:33

I just closed the issue on the tracker.

Christoph Knittel's picture

Christoph Knittel - Sun, 2014/06/08 - 17:53

Thanks a lot!

I just retested backup and restore to a new cloud server without my script present in /etc/tklbam/hooks.d, and everything worked fine. :-)

Jeremy Davis's picture

Jeremy Davis - Fri, 2014/06/13 - 02:30

Great news! Thanks for the confirmation. :)

Add new comment