Skip to main content

On-Prem Deployment using Filesystem

Note

Chef Automate 4.10.1 released on 6th September 2023 includes improvements to the deployment and installation experience of Automate HA. Please read the blog to learn more about key improvements. Refer to the pre-requisites page (On-Premises, AWS) and plan your usage with your customer success manager or account manager.

Note

  • If the user chooses backup_config as file_system in config.toml backup is already configured during the deployment, and in that case the below steps are not required. If backup_config is left blank, then the configuration needs to be configured manually.

Overview

A shared file system is always required to create OpenSearch snapshots. To register the snapshot repository using OpenSearch, it is necessary to mount the same shared filesystem to the exact location on all master and data nodes. Register the location (or one of its parent directories) in the path.repo setting on all master and data nodes.

Setting up the backup configuration

Configuration in OpenSearch Node

  • Mount the shared file system to the base mount path which is mentioned in backup_mount on all OpenSearch and Frontend servers.

Note

  • /mnt/automate_backups is the default value for the backup_mount, which is also used in this document page as reference backup path.
  • While using file_system as backup type, the uid of hab user should be same across all the remote nodes. The same will be verified during verify check before deployment
  • Do not modify/delete any file manually inside the backup_mount directory

Apply the following steps on all of the OpenSearch server node

  • Create an OpenSearch sub-directory and set permissions (only if the network mount is correctly mounted).

    sudo mkdir /mnt/automate_backups/opensearch
sudo chown hab:hab /mnt/automate_backups/opensearch/

Configuration for OpenSearch Node from Bastion Host

Configure the OpenSearch path.repo setting by following the steps given below:

  • Create a .toml (say os_config.toml) file in the Bastion host and copy the following template with the path to the repo.

      [path]
  # Replace /mnt/automate_backups with the backup_mount config found on the Bastion host in /hab/a2_deploy_workspace/a2ha.rb
  repo = "/mnt/automate_backups/opensearch"

  • Following command will add the configuration to the OpenSearch node.

      chef-automate config patch --opensearch <PATH TO OS_CONFIG.TOML>
Healthcheck commands

  • Following command can be run at the bastion node

    chef-automate status --opensearch

  • Following command can be run in the OpenSearch node

    hab svc status (check whether OpenSearch service is up or not)

curl -k -X GET "<https://localhost:9200/_cat/indices/*?v=true&s=index&pretty>" -u admin:admin (Another way to check is to check whether all the indices are green or not)

# Watch for a message about OpenSearch going from RED to GREEN
journalctl -u hab-sup -f | grep 'automate-ha-opensearch'

Configuration for Automate Node from Provision Host

  • Configure Automate to handle External OpenSearch Backups.

  • Create an automate.toml file on the provisioning server using the following command:

    touch automate.toml

    Add the following configuration to automate.toml on the provisioning host:

    [global.v1.external.opensearch.backup]
enable = true
location = "fs"

[global.v1.external.opensearch.backup.fs]
# The `path.repo` setting you've configured on your OpenSearch nodes must be a parent directory of the setting you configure here:
path = "/mnt/automate_backups/opensearch"

[global.v1.backups.filesystem]
path = "/mnt/automate_backups/backups"

  • Patch the automate.toml config to trigger the deployment from the provision host.

    chef-automate config patch --fe automate.toml

Backup and Restore

Backup

To create the backup, by running the backup command from bastion. The backup command is as shown below:

chef-automate backup create

Restore

Pre-Restore Validation

Run the restore command with the –verify-restore-config flag to validate the configuration settings before initiating the restore process. To perform the pre-check, run the following command from the bastion host:

chef-automate backup restore <BACKUP-ID> --yes -b /mnt/automate_backups/backups --verify-restore-config

The verification process ensures that the backup and restore configurations are correct and identifies potential issues so they can be addressed in advance.

Run Restore

To restore backed-up data of the Chef Automate High Availability (HA) using External File System (EFS), follow the steps given below:

  • Check the status of Automate HA Cluster from the bastion nodes by executing the chef-automate status command.

  • Execute the restore command from bastionchef-automate backup restore <BACKUP-ID> --yes -b /mnt/automate_backups/backups --airgap-bundle </path/to/bundle>.

Note

  • If you are restoring the backup from an older version, then you need to provide the --airgap-bundle </path/to/current/bundle>.
  • Large Compliance Report is not supported in Automate HA

Troubleshooting

Follow the steps below if Chef Automate encounters an error during data restoration.

  1. Check the Chef Automate status.

    chef-automate status

  2. Check the status of your Habitat service on the Automate node.

    hab svc status

  3. If the deployment services are not healthy, reload them.

    hab svc load chef/deployment-service

  4. Check the status of the Automate node, and then attempt to run the restore command from the bastion host.

  5. To change the base_path or path, follow the steps below for performing a backup.

    • File System

      • During deployment, the backup_mount is default set to /mnt/automate_backups.
      • The deployment process will automatically apply the updated path if you update the backup_mount value in the config.toml file before deployment.
      • If the backup_mount value is changed after deployment (e.g., to /bkp/backps), you must manually patch the configuration on all frontend and backend nodes.
      • Update the FE nodes using the template below. To update the configuration, use the command chef-automate config patch fe.toml --fe.
         [global.v1.backups]
      [global.v1.backups.filesystem]
         path = "/bkp/backps"
   [global.v1.external.opensearch.backup]
      [global.v1.external.opensearch.backup.fs]
         path = "/bkp/backps"
      • Update the OpenSearch nodes using the template provided below. Use the chef-automate config patch os.toml --os command to update the Opensearch node configs.
      [path]
   repo = "/bkp/backps"

      • Run the curl request against one of the Automate frontend nodes.

        curl localhost:10144/_snapshot?pretty

        • If the response is an empty JSON object {}, no changes are required to the snapshot settings in the OpenSearch cluster.

        • If you see a JSON response similar to the example below, check that the backup_mount setting is correctly configured. Use the location value in the response to verify. It should start with /bkp/backps.

        {
 "chef-automate-es6-event-feed-service" : {
 "type" : "fs",
 "settings" : {
 "location" : "/mnt/automate_backups/opensearch/automate-elasticsearch-data/chef-automate-es6-event-feed-service"
       }
    },
 "chef-automate-es6-compliance-service" : {
 "type" : "fs",
 "settings" : {
 "location" : "/mnt/automate_backups/opensearch/automate-elasticsearch-data/chef-automate-es6-compliance-service"
       }
    },
 "chef-automate-es6-ingest-service" : {
     "type" : "fs",
 "settings" : {
 "location" : "/mnt/automate_backups/opensearch/automate-elasticsearch-data/chef-automate-es6-ingest-service"
       }
    },
 "chef-automate-es6-automate-cs-oc-erchef" : {
 "type" : "fs",
 "settings" : {
 "location" : "/mnt/automate_backups/opensearch/automate-elasticsearch-data/chef-automate-es6-automate-cs-oc-erchef"
       }
    }
 }
        • If the prefix in the location value does not match the backup_mount, the existing snapshots must be deleted. Use the script below to delete the snapshots from the one of the Automate frontend nodes.
           snapshot=$(curl -XGET http://localhost:10144/_snapshot?pretty | jq 'keys[]')
   for name in $snapshot;do
       key=$(echo $name | tr -d '"')
      curl -XDELETE localhost:10144/_snapshot/$key?pretty
   done
        • The above script requires jq to be installed, You can install it from the airgap bundle. To locate the jq package, run the command below on one of the Automate frontend nodes.
        ls -ltrh /hab/cache/artifacts/ | grep jq

-rw-r--r--. 1 ec2-user ec2-user  730K Dec  8 08:53 core-jq-static-1.6-20220312062012-x86_64-linux.hart
-rw-r--r--. 1 ec2-user ec2-user  730K Dec  8 08:55 core-jq-static-1.6-20190703002933-x86_64-linux.hart
        • If multiple versions of jq are available, install the latest one. Use the command below to install the jq package on one of the Automate frontend nodes.
        hab pkg install /hab/cache/artifacts/core-jq-static-1.6-20190703002933-x86_64-linux.hart -bf

    • Object Storage

      • During deployment, the backup_config should be set to object_storage.
      • To use object_storage, we use the following template during deployment.
         [object_storage.config]
    google_service_account_file = ""
    location = ""
    bucket_name = ""
    access_key = ""
    secret_key = ""
    endpoint = ""
    region = ""
      • If you configured it before deployment, then you are all set.
      • If you want to change the bucket or base_path, use the following template for Frontend nodes.
      [global.v1]
  [global.v1.external.opensearch.backup.s3]
      bucket = "<BUCKET_NAME>"
      base_path = "opensearch"
   [global.v1.backups.s3.bucket]
      name = "<BUCKET_NAME>"
      base_path = "automate"

      • You can assign any value to the base_path variable. The base_path configuration is required only for the Frontend nodes.

      • Use the command chef-automate config patch frontend.toml --fe to apply the above template and update the configuration.

      • Use the following curl request to validate the configuration.

        curl localhost:10144/_snapshot?pretty

      • If the response is an empty JSON object ({}), the configuration is valid.

      • If the response contains a JSON output similar to the example below, it should have the correct value for the base_path.

        {
    "chef-automate-es6-event-feed-service" : {
      "type" : "s3",
      "settings" : {
        "bucket" : "MY-BUCKET",
        "base_path" : "opensearch/automate-elasticsearch-data/chef-automate-es6-event-feed-service",
        "readonly" : "false",
        "compress" : "false"
      }
    },
    "chef-automate-es6-compliance-service" : {
      "type" : "s3",
      "settings" : {
        "bucket" : "MY-BUCKET",
        "base_path" : "opensearch/automate-elasticsearch-data/chef-automate-es6-compliance-service",
        "readonly" : "false",
        "compress" : "false"
      }
    },
    "chef-automate-es6-ingest-service" : {
      "type" : "s3",
      "settings" : {
        "bucket" : "MY-BUCKET",
        "base_path" : "opensearch/automate-elasticsearch-data/chef-automate-es6-ingest-service",
        "readonly" : "false",
        "compress" : "false"
      }
    },
    "chef-automate-es6-automate-cs-oc-erchef" : {
      "type" : "s3",
      "settings" : {
        "bucket" : "MY-BUCKET",
        "base_path" : "opensearch/automate-elasticsearch-data/chef-automate-es6-automate-cs-oc-erchef",
        "readonly" : "false",
        "compress" : "false"
      }
    }
}
        • If the base_path value does not match, you must delete the existing snapshots. Please take a look at the File System troubleshooting steps for guidance.
Edit this page on GitHub

Thank you for your feedback!

×