Upgrading from 2.7.x to 3.1.x

This section is for administrators who have an older 2.7.x version of Karaage already installed, and wish to upgrade to 3.1.x.

These instructions are also usable for upgrades from 2.6.x, however Debian/jessie is a prerequisite.

If you don’t already have Karaage installed and wish to get started for the first time, please see the Getting Started section.

Warning

This is a non-trivial upgrade with non-trivial database migrations. The database migrations will take some time to complete and could fail or cause data loss. It is recommended that you copy your production data and run the database migrations on a test system, as shown below.

Assumptions

Assumptions made by this documentation. Other configurations are possible, however you will have to adapt from this documentation.

  • You have a cluster running Slurm or PBS that is already up and running.
  • You want to OpenLDAP with ppolicy support, on the same server.
  • You have a Debian Wheezy server already setup for Karaage.
  • You will be upgrading all components on a single system, called production.

Backup production box

Warning

Any changes to database while after the backup, but before the upgrade is finished, may be lost. It is recommended to shutdown Karaage before running the final run to ensure important changes are not made.

Backup various things on production box. Some of the following steps aren’t strictly required, however recommended to guarantee no data lose will occur during the upgrade.

  1. Backup /etc/karaage and everything under this directory.

  2. Backup /etc/ldap or /etc/openldap and everything under this directory.

  3. Dump mysql database on production box.

    mysqldump karaage > karaage.sql
    

    You will use this file in the next step.

  4. Dump LDAP database on production box.

    slapcat > karaage.ldif
    
  5. Copy these files off the production box to somewhere secure.

Operating System

Ensure production box is running Debian jessie with all applicable updates. You will not be able to install Karaage3 if you are running anything older then Wheezy. It is also suggested (but not required) that you upgrade to the latest 2.7 Karaage here, as this will make downgrading easier in case of problems upgrading to 3.0.

Check MySQL server config

  1. (optional) Create a /root/.my.cnf file containing:

    [client]
    user            = root
    password        = XXXXXXXX
    
  2. Create a /etc/mysql/conf.d/karaage.cnf file containing:

    [mysqld]
    character_set_server=utf8
    default-storage-engine = innodb
    sql_mode = STRICT_ALL_TABLES
    
    [client]
    default-character-set = utf8
    

    Note: these settings may affect other applications that use this database.

  3. Restart mysql server to load config:

    service mysql restart
    

Installing Karaage3

Warning

Double check what you are doing to ensure you don’t accidentally delete production data.

Warning

You must have sufficient disk space on the production box to create a copy of the Karaage database.

Note

The migrations do not make any invasive changes to LDAP in any way until you run migrate_ldap, manually, by hand. This can be done after testing everything to make sure it is working fine. This means you don’t need to copy LDAP data, however making a backup is still recommended.

If any institute references a group by gidNumber that does not exist, and cannot be found by name either, it will automatically be created. Not having a group is an error, even for Karaage 2.7.x.

Note

If you do encounter any problems performing database migrations, please file a bug report, so the problem can be fixed. Bug reports can be submitted at github.

  1. Create a copy of the karaage database on your production mysql server:

    1. Create a new user and database for karaage3:

      mysql> create database karaage3;
      mysql> CREATE USER 'karaage3'@'localhost' IDENTIFIED BY 'YYYYYYYY';
      mysql> GRANT ALL PRIVILEGES ON karaage3.* TO 'karaage3'@'localhost';
      

      You will use the values you set, later, in the karaage settings. Do not use YYYYYYYY on a production system.

      It is recommended that the user karaage3 should not have any access to the karaage database, to avoid the potential for mistakes being made.

    2. Copy mysql to new database on production box:

      mysql karaage3 < karaage.sql
      
    3. Check data in karaage3 looks correct, before proceeding.

  2. Check mysql data is correct.

    1. Connect to mysql database:

      mysql karaage3
      
    2. Ensure all tables in database are using innodb and utf8 encoding:

      mysql> use karaage3
      mysql> show table status
      
    3. If not all tables are using innodb and utf8, this can be corrected with the following script:

      #!/bin/sh
      set -x
      
      DBNAME="karaage3"
      USERNAME="karaage3"
      PASSWORD="YYYYYYYY"
      
      for i in $(mysql -B -N -e "SHOW TABLES" -u$USERNAME -p$PASSWORD $DBNAME); do
          mysql -B -N -e "SET foreign_key_checks = 0; ALTER TABLE $i ENGINE=InnoDB;" -u$USERNAME -p$PASSWORD $DBNAME
          mysql -B -N -e "SET foreign_key_checks = 0; ALTER TABLE $i CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci" -u$USERNAME -p$PASSWORD $DBNAME
      done
      
    4. The above can also be done manually. For every table that is not innodb, convert it with:

      mysql> ALTER TABLE table_name ENGINE=InnoDB;
      

      For every table that is not utf8, convert it with:

      mysql> ALTER TABLE table_name CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci
      

      Some of these conversions may take some time to complete, especially for the cpujob table.

    5. Ensure mysql data is correct without any obvious signs of problems.

  3. (optional) If uncertain of upgrade, setup a test system. Read the steps in Getting Started. Do not set up any data stores yet. These steps will instruct you to install a mysql database, which isn’t essential, however does help ensure karaage is working correctly at this point.

  4. Otherwise, install the latest version of karaage3:

    apt-get install karaage3
    apt-get install python-mysqldb
    
  5. Check the DATABASES, LDAP, MACHINE_CATEGORY_DATASTORES and KG27_DATASTORE settings in /etc/karaage3/settings.py. The mysql server should refer to the new database you created above, the LDAP server should refer to your production server.

    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'NAME': 'karaage3',
            'USER': 'karaage3',
            'PASSWORD': 'YYYYYYYY',
            'HOST': '',
            'PORT': '',
            'ATOMIC_REQUESTS': True,
            'OPTIONS': {
                'sql_mode': 'STRICT_ALL_TABLES'
            },
        }
    }
    
    LDAP = {
        'default': {
            'ENGINE': 'tldap.backend.fake_transactions',
            'URI': 'ldap://production',
            'USER': 'cn=admin,dc=example,dc=org',
            'PASSWORD': 'XXXXXXXX',
            'REQUIRE_TLS': False,
            'START_TLS ': False,
            'TLS_CA' : None,
        }
    }
    
    MACHINE_CATEGORY_DATASTORES = {
        'ldap' : [
            {
                'DESCRIPTION': 'LDAP datastore',
                'ENGINE': 'karaage.datastores.ldap.MachineCategoryDataStore',
                'LDAP': 'default',
                'ACCOUNT': 'karaage.datastores.ldap_schemas.openldap_account',
                'GROUP': 'karaage.datastores.ldap_schemas.openldap_account_group',
                'PRIMARY_GROUP': "institute",
                'DEFAULT_PRIMARY_GROUP': "dummy",
                'HOME_DIRECTORY': "/home/%(uid)s",
                'LOCKED_SHELL': "/usr/local/sbin/locked",
                'NUMBER_SCHEME': 'default',
                'LDAP_ACCOUNT_BASE': 'ou=Accounts,dc=example,dc=org',
                'LDAP_GROUP_BASE': 'ou=Groups,dc=example,dc=org',
            },
        ],
        'dummy' : [
        ],
    }
    
    KG27_DATASTORE = {
        'DESCRIPTION': 'LDAP datastore',
        'ENGINE': 'karaage.datastores.ldap.MachineCategoryDataStore',
        'LDAP': 'default',
        'ACCOUNT': 'karaage.datastores.ldap_schemas.openldap_kg27',
        'GROUP': 'karaage.datastores.ldap_schemas.openldap_kg27_group',
        'PRIMARY_GROUP': "institute",
        'DEFAULT_PRIMARY_GROUP': "dummy",
        'HOME_DIRECTORY': "/home/%(uid)s",
        'LOCKED_SHELL': "/usr/local/sbin/locked",
        'NUMBER_SCHEME': 'default',
        'LDAP_ACCOUNT_BASE': 'ou=People,dc=example,dc=org',
        'LDAP_GROUP_BASE': 'ou=Groups,dc=example,dc=org',
    }
    

    The values given for LDAP_ACCOUNT_BASE and LDAP_GROUP_BASE in KG27_DATASTORE should correspond with your existing LDAP data.

    Warning

    You must have LDAP configured correctly before you proceed to the migration step. The migrations reference LDAP information if it is available. If it is not available, the database may end up with incorrect information.

  6. (optional) If you require people to be recorded in LDAP:

    GLOBAL_DATASTORES = [
          {
                'DESCRIPTION': 'LDAP datastore',
                'ENGINE': 'karaage.datastores.ldap.GlobalDataStore',
                'LDAP': 'default',
                'PERSON': 'karaage.datastores.ldap_schemas.openldap_person',
                'GROUP': 'karaage.datastores.ldap_schemas.openldap_person_group',
                'NUMBER_SCHEME': 'global',
                'LDAP_PERSON_BASE': 'ou=People,dc=example,dc=org',
                'LDAP_GROUP_BASE': 'ou=People_Groups,dc=example,dc=org',
          },
    ]
    
  7. Migrate DB tables.

    If you have upgraded Django to 1.7 you will need to run south migrations first. Running this command is perfectly safe from Django 1.6, however not required. This must be run before running Django 1.7 migrations however.

    apt-get install virtualenv         # Debian Jessie only
    apt-get install python-virtualenv  # Debian Wheezy only
    kg-migrate-south
    

    Regardless of what version of Django you have installed, continue and run the native migrations.

    kg-manage migrate
    
  8. Check mysql data is correct.

    1. Connect to mysql database:

      mysql karaage3
      
    2. Ensure all tables in database are using innodb and utf8 encoding:

      mysql> use karaage3
      mysql> show table status
      
    3. Ensure mysql data is correct without any obvious signs of problems.

  9. If you have any other datastores, configure them now. See Data Stores.

  10. Restart karaage processes.

    service apache2 start
    service karaage3-celery start
    
  11. You can run the migrate_ldap command to check what LDAP changes are required:

    kg-manage migrate_ldap --delete --dry-run
    

    If you are happy, repeat without the --dry-run option.

    You may want to consider not using the --delete option if you are still using the old data.

    This need to be run before the upgrade can be considered complete, however most things should still work without doing this change.

  12. Test. You should now be able to go to http://hostname/kgadmin/. Apply any local customizations you need to have a fully operational system now. Do not continue if you are not completely happy.

Final server configuration

Up to this point you have not made any changes to your production server, apart from installing the newer packages. If you are not happy with the upgrade, you can revert to your production system. After you pass this point, it will still be possible to revert, only it will be slightly harder, as you have to undo the following steps. Once you proceed past this point, reverting may require restoring LDAP from the backup.

  1. Run the following commands to check what LDAP changes are required:

    kg-manage migrate_ldap --dry-run
    

    Make the LDAP changes if happy:

    kg-manage migrate_ldap
    
  2. If required, install Karaage plugins. See Plugins for more information.

  3. Test production box and make sure everything is working.

Other changes

Check that remote services that access Karaage work. For example, with torque_submitfilter, you need to update the Karaage URL to /karaage/.

You may need to update PBS/slurm logging to talk correctly to Karaage. Only do this if kg-pbs-logger was previously configured. Karaage no longer requires a dedicated account for kg-pbs-logger. Rather it uses the machine entry. For every machine:

  1. (optional) Delete the dedicated account that was previously used for authenticating this system.

  2. If using MAM, edit the MACHINE_CATEGORY_DATASTORES setting in /etc/karaage3/settings.py:

    {
        'DESCRIPTION': 'MAM datastore',
        'ENGINE': 'karaage.datastores.mam.MamDataStore',
        ...
    }
    
  3. Navigate to machine entry in admin website.

  4. Click password button to reset the password.

  5. Update /etc/karaage/pbs-logger.cfg and for ws_username use the machine name in karaage, and for ws_password use the password obtained in the previous step.

  6. You may need to reconfigure LDAP clients.

    • If people require access to service even if they don’t have an account (rare).

      ou=People,dc=example,dc=org and ou=People_Groups,dc=example,dc=org

    • If people only should get access with an account (most common):

    ou=Accounts,dc=example,dc=org and ou=People_Groups,dc=example,dc=org