Gluster 4.x and GD2 (GlusterD2) support

[sammcj]

Gluster 4.x has been in pre-release for some time and ended up being release last week (14/03/2018).

It would be great to see support for Gluster 4.x in this module so people can start upgrading.

• Announcement
• Release Notes
• Quick Start Guide
• Upgrade Guide
• More about Gluster D2

[sammcj]

FYI Gluster 4.1.0 4.1.2 has now been released.

[bronislav]

@sammcj Since gluster 4.0 and 4.1 already available in http://mirror.centos.org/centos/7/storage/x86_64/ you should be able to install these versions without (any?) changes in the module. Highly possible that other things could be broken.

[sammcj]

Gluster 4 onwards is a complete rewrite (in go) from the ground up. Gluster is now mainly API driven and uses different command line tools so I’d say the module needs a rewrite / major update to support it. I’d offer to give it a crack but I’m unfortunately really bogged down with another major project at the moment. https://fosdem.org/2018/schedule/event/gluster4_and_gd2/attachments/slides/2419/export/events/attachments/gluster4_and_gd2/slides/2419/gluster4_fosdem2018.pdf

…

-- Sam McLeod

On 14 Jun 2018, at 07:01, Anton Ilin ***@***.***> wrote: @sammcj Since gluster 4.0 and 4.1 already available in http://mirror.centos.org/centos/7/storage/x86_64/ you should be able to install these versions without (any?) changes in the module. Highly possible that other things could be broken. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[sammcj]

FYI - just a heads up, Gluster 3.1.3 is no longer available and has been completely purged from the Gluster repos, the official word is to please use Gluster 4 (4.1 now I guess).

A side note, we've tested clients running Gluster Client 4.1 and they can connect to Gluster 3.1.3.

But be ware - Gluster 3 is dying very quickly!

[ekohl]

In #170 I'm removing 3.8 repo support. Because I don't use gluster I don't have time to rewrite to Gluster 4.

[sammcj]

FYI Gluster 4.1.2 is now current stable and LTS.

I've tested this module with Gluster 4 and it works, but it definitely does not work with GlusterD2 (GD2).

---

Below is a heap of information about how I performed the upgrade and what our settings are in case it helps anyone.

FYI to upgrade from Gluster 3.13 to 4.1.2 we performed the following (one node at a time):

• verified client op-version was already at 4.1.2
• snapshotted int-gluster-01/02/03 nodes
• disabled puppet on int-gluster-01/02/03 nodes

puppet agent --disable gluster_upgrade

• merged 4.1.2 upgrade branch

systemctl stop glusterd
killall glusterfs glusterfsd glusterd

puppet agent --enable; puppet agent --test

systemctl stop glusterd
killall glusterfs glusterfsd glusterd

yum upgrade -y

systemctl stop glusterd
killall glusterfs glusterfsd glusterd
systemctl start glusterd

for i in `gluster volume list`; do gluster volume heal $i; done

Test that healing was finished:

for i in `gluster volume list`; do gluster volume heal $i info; done

• Repeat for nodes 02, then 03
• Then once completed and healthy upgrade the op-version:

gluster volume set all cluster.op-version 40100

• Turn off usage of MD5 checksums during healing, this enables running Gluster on FIPS compliant systems, as per: https://gluster.readthedocs.io/en/latest/Upgrade-Guide/upgrade_to_4.1/#post-upgrade-steps

# gluster volume set test_storage fips-mode-rchecksum

    'test_storage':
      'env': 'test'
      'volume_options':
        'storage.fips-mode-rchecksum': 'true'

and just to cleanup for good measure as puppet is now managing this in the module:

yum remove -y centos-release-gluster31{2,3}

And just FYI our gluster host profile looks a bit like this:

# htcp is the recommended TCP congestion control for Gluster
  # This is used in hiera with net.ipv4.tcp_congestion_control: htcp in gluster_host.eyaml
  kmod::load { 'tcp_htcp': }

  # https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Linux%20Kernel%20Tuning/
  sysctl { 'vm.vfs_cache_pressure': value => '150' }

  class { '::gluster::repo':
    release => $gluster_version,
  }

  class { '::gluster::install':
    server => true,
    client => true,
    repo   => true,
  }

  file { '/etc/glusterfs/glusterd.vol':
    ensure  => present,
    content => template('profiles/services/gluster/glusterd.vol.erb'),
    owner   => root,
    group   => root,
    mode    => '0644',
    require => Class[::gluster::install],
  }

  gluster::peer { $pool_members:
    pool    => $pool, # read from the included gluster::pool profile
    require => [ Class[::gluster::service],Class[::gluster::install] ],
  }

  #Set log level to WARNING or higher
  file_line {'glusterd_log_warning':
    ensure  => present,
    line    => 'GLUSTERD_LOGLEVEL="WARNING"',
    after   => '#GLUSTERD_LOGLEVEL="NORMAL"',
    path    => '/etc/sysconfig/glusterd',
    require => Class[::gluster::install],
    notify  => Service['glusterd'],
  }

  file_line {'glusterfsd_log_warning':
    ensure  => present,
    line    => 'GLUSTERFSD_LOGLEVEL="WARNING"',
    after   => '#GLUSTERFSD_LOGLEVEL="NORMAL"',
    path    => '/etc/sysconfig/glusterfsd',
    require => File_line['glusterd_log_warning'],
    notify  => Service['glusterd'],
  }

  file_line {'gluster_log_warning':
    ensure  => present,
    line    => 'LOG_LEVEL="WARNING"',
    after   => 'GLUSTERD_LOGLEVEL="WARNING"',
    path    => '/etc/sysconfig/glusterd',
    require => File_line['glusterd_log_warning'],
    notify  => Service['glusterd'],
  }

  class { '::gluster::service':
    ensure  => running,
    require => [File_line['gluster_log_warning'],File['/etc/glusterfs/glusterd.vol']],
  }

  # TODO Temporary workaround until https://github.com/voxpupuli/puppet-gluster/issues/136 is fixed
  exec { 'enable_global_multiplex':
    command => '/sbin/gluster volume set all cluster.brick-multiplex disable',
    user    => 'root',
    unless  => '/sbin/gluster volume get all cluster.brick-multiplex|/bin/grep disable',
  }

where glusterd.vol.erb is:

volume management
    type mgmt/glusterd
    option working-directory /var/lib/glusterd
    option transport-type socket
    option transport.socket.keepalive-time 10
    option transport.socket.keepalive-interval 2
    option transport.socket.read-fail-log off
    option rpc-auth-allow-insecure on
    option ping-timeout 15
    option event-threads 8
    option transport.address-family inet6
end-volume

and our hiera data looks a bit like this:

sysctl::base::values:
  'net.ipv4.tcp_keepalive_time':
    ensure: present
    value: '900'
    comment: 'Keepalives necessary so open-but-idle TCP connections are not blocked by the firewall after 30 mins'
  'net.ipv4.tcp_keepalive_intvl':
    value: '300'
    ensure: present
    comment: 'Keepalives necessary so open-but-idle TCP connections are not blocked by the firewall after 30 mins'
  'net.ipv4.tcp_keepalive_probes':
    ensure: present
    value: '30'
    comment: 'Keepalives necessary so open-but-idle TCP connections are not blocked by the firewall after 30 mins'
  'net.ipv4.tcp_congestion_control':
    ensure: present
    value: 'htcp'
  'net.ipv4.tcp_max_syn_backlog':
    ensure: present
    value: '300000'
    comment: 'Increase syn backlogs for gluster'
  'net.ipv4.ip_local_port_range':
    ensure: present
    value: '32768	65535'
    comment: 'Increase local port range for gluster'
  'net.core.somaxconn':
    ensure: present
    value: '65535'
    comment: 'Increase kernel socket backlog limit for gluster'
  'net.core.rmem_max':
    ensure: present
    value: '16777216'
    comment: 'Increase replication performance'
  'net.core.wmem_max':
    ensure: present
    value: '16777216'
    comment: 'Increase replication performance'
  'net.ipv4.tcp_slow_start_after_idle':
    ensure: present
    value: '0'
    comment: 'Disable TCP slow start after idle for gluster performance'
  'net.ipv4.tcp_fin_timeout':
    ensure: present
    value: '5'
    comment: 'Decrease tcp TIME_WAIT timeouts to improve gluster performance with large numbers of connections'
  'net.core.netdev_max_backlog':
    ensure: present
    value: '300000'
    comment: 'increase maximum size of receive queue'
  'net.ipv4.tcp_rmem':
    ensure: present
    value: '4096	87380	16777216'
    comment: 'increase maximum network buffers'
  'net.ipv4.tcp_wmem':
    ensure: present
    value: '4096	65536	16777216'
    comment: 'increase maximum network buffers'

# profiles::services::gluster::volumes:
#   # Defaults
profiles::services::gluster::volumes::defaults::brick_mountpoint: '/mnt/gluster-storage'
profiles::services::gluster::volumes::defaults::replica: 3
profiles::services::gluster::volumes::defaults::arbiter: 1
profiles::services::gluster::volumes::defaults::volume_options:
  # Failover clients after 15 seconds of a server being unavailable
  'network.ping-timeout': '15'
  'cluster.lookup-optimize': 'true'
  'cluster.readdir-optimize': 'true'
  'cluster.use-compound-fops': 'true'
  'cluster.eager-lock': 'true'
  'cluster.favorite-child-policy': 'size' # http://docs.gluster.org/en/latest/Troubleshooting/heal-info-and-split-brain-resolution/#5-automagic-unsplit-brain-by-ctimemtimesizemajority
  'performance.client-io-threads': 'true'
  'performance.stat-prefetch': 'true' # Speeds up operations like ls -la
  # 'performance.parallel-readdir': 'true'
  'performance.write-behind-window-size': '512MB'
  'performance.cache-refresh-timeout': '60'
  'performance.cache-max-file-size': '3MB'
  'performance.cache-min-file-size': '2KB'
  'performance.md-cache-timeout': '300' # w/ cache-invalidation increased from 30
  'performance.io-thread-count': '16'
  'performance.cache-invalidation': 'true'
  'features.cache-invalidation': 'true'
  'features.cache-invalidation-timeout': '300' # w/ cache-invalidation increased from 30
  'diagnostics.brick-log-level': 'ERROR'
  'diagnostics.client-log-level': 'ERROR'
  'server.event-threads': '4'
  'client.event-threads': '4'
  'transport.address-family': 'inet'
  'transport.listen-backlog': '2048'
  'performance.cache-size': '1GB'
  'network.inode-lru-limit': '50000' # https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/sect-directory_operations
  'server.outstanding-rpc-limit': '256'
  'performance.rda-cache-limit': '1GB'

profiles::services::gluster::volumes::pool: "%{alias('profiles::services::gluster::host::pool')}"
profiles::services::gluster::volumes::default_volume_options: "%{alias('profiles::services::gluster::volumes::defaults::volume_options')}" # per-vol options merged into defaults
profiles::services::gluster::volumes::brick_mountpoint: "%{alias('profiles::services::gluster::volumes::defaults::brick_mountpoint')}" # use the defaults
profiles::services::gluster::volumes::replica: "%{alias('profiles::services::gluster::volumes::defaults::replica')}" # use the defaults
profiles::services::gluster::volumes::arbiter: "%{alias('profiles::services::gluster::volumes::defaults::arbiter')}" # use the defaults
profiles::services::gluster::volumes::pool_members: "%{alias('profiles::services::gluster::host::pool_members')}"
profiles::services::gluster::volumes::pools:
  'int-gluster-pool-01':
    'sam':
      'env': 'sam'
      'volume_options':
        'storage.fips-mode-rchecksum': 'true'
        'performance.cache-size': '2GB'
        'performance.write-behind-window-size': '1GB'

To pass the above volumes in hiera we have a profile that looks a bit like:

class profiles::services::gluster::volumes (
    $arbiter,
    $brick_mountpoint,
    $default_volume_options,
    $pool,
    $pools,
    $replica,
    ) {

  $pool_volumes_hash = $pools[$pool]
  $gluster_volumes = keys($pool_volumes_hash)
  $pool_members = hiera('profiles::services::gluster::host::pool_members')

  $gluster_volumes.each |String $volume_name| {
    # Merge default volume options with custom volume options
    $combined_volume_options = merge($pool_volumes_hash[$volume_name]['volume_options'],$default_volume_options)
    validate_hash($combined_volume_options)

    # Convert volume options from a hash to an array for Gluster
    $volume_options_array = join_keys_to_values($combined_volume_options, ': ')
    validate_array($volume_options_array)

    $brick_location = "${brick_mountpoint}/${volume_name}"
    validate_string($brick_location)

    # Clean brick names to be supported by Gluster
    $bricks = regsubst($pool_members, '^(.*)', "\1:${brick_location}")
    validate_array($bricks)

    # Create Gluster volumes
    # e.g. /mnt/gluster-storage/test_storage/
    file { $brick_location:
      ensure  => directory,
      mode    => '0755',
      require => Class['profiles::os::redhat::centos7::lvm'],
    }
    ->gluster::volume { $volume_name:
        replica => $replica,
        arbiter => $arbiter,
        options => $volume_options_array,
        bricks  => $bricks,
        require => [ Class[::gluster::service],Class[::gluster::install], File[$brick_location] ],
      }
  }
}

and each node will have something like this in it's hiera definitions:

# Remember to update the cluster.op-version after upgrading gluster, e.g:
# gluster volume set all cluster.op-version 40100
profiles::services::gluster::host::gluster_version: '4.1'
profiles::services::gluster::host::pool: 'int-gluster-pool-01'
profiles::services::gluster::host::pool_members:
  - 'int-gluster-01.cool.story.bro'
  - 'int-gluster-02.cool.story.bro'
  - 'int-gluster-03.cool.story.bro'

[af6140]

Looks like no active development going on for this module.

[bronislav]

@af6140 GlusterD2 in under active development and wasn't considered production ready when I checked it couple month ago.