diff --git a/CHANGELOG.md b/CHANGELOG.md index 4476a80..b2d71fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,22 @@ This file is used to list changes made in each version of the aix cookbook. +## 2.3.1 (2018-05-16) + +- Don't .to_s strings +- Use the chef shell_out mixin vs. mixlib_shellout +- Properties that are Strings default to nil so we can remove the nil defaults +- Remove empty load_current_value method which isn't doing anything +- Load current resource properly instead of tracking state in properties +- Remove redundant default_action +- Use new_resource instead of @new_resource in several resources +- Fix strings in the examples to properly interpolate +- Fix error message to actually output the error +- Update the rubocop rules we disable +- Resolve FC019 warnings in the examples +- Disable FC0113 for now since we still support Chef 12 +- Fix loading current_resource in the inittab resource + ## 2.3.0 (2018-05-10) - Resolve incompatibilities with Chef 13 & 14 diff --git a/README.md b/README.md index 1291ea8..86bde58 100644 --- a/README.md +++ b/README.md @@ -506,6 +506,11 @@ In some cases a metadata operation is performed to discover the oslevel build nu The location directory is automatically created if it does not exist. +The following user process resource limits are recommended to run a suma recipe: +- fsize = -1 +- nofiles = 20000 (or greater) +- data = -1 + The NIM lpp_source resource is automatically created if needed. It meets the following requirement. Name contains build number and ends with the type of resource: - 7100-04-00-0000-lpp_source @@ -650,6 +655,7 @@ If space is needed, filesystem is automatically extended by increment of 100MB. - `filesets` - filter on fileset name - `csv` - custom apar csv file - `path` - directory where the report is saved +- `force` - if true, installed interim fixes will be automatically removed (default: false) - `clean` - clean temporary files and remove nim lpp_source resource (default: true) - `verbose` - save and display the report in verbose mode (default: false) - `check_only` - generate report only, no fixes are downloaded nor installed (default: false) @@ -717,7 +723,7 @@ action_list: * altdisk_copy: will make an alternate rootvg on an available disk * update: install software on VIOSes from existing NIM lpp source * altdisk_cleanup: to remove the altinst_rootvg - + #### Properties - `lpp_source` - Name of NIM lpp_source resource to install @@ -776,6 +782,87 @@ aix_nimviosupdate "check VIOSes tuples, make alternate rootvg, update VIOSes, re end ``` +### viosupgrade + +Use viosupgrade to upgrade a VIOS or a couple of VIOSes by installing the specifed image from the NIM server. +Performs the operations of backing up the virtual and logical configuration data, installing the specified image, +and restoring the virtual and logical configuration data of the Virtual I/O Server (VIOS). +Each action from the action list can be executed independently or together: check, altdisk_copy, validate, upgrade, altdisk_cleanup +action_list: + * check: verify the redundancy of the 2 VIOSes for the upgrade + * altdisk_copy: will make an alternate rootvg on an available disk + * validate: will make a verification about upgrade operation + * upgrade: install image on VIOSes from existing NIM mksysb iamge and restoring the virtual and logical configuration data + * altdisk_cleanup: to remove the altinst_rootvg + +#### Properties + +- `targets` - Comma separated list of single or dual VIOSes to manage in a tuple format (required) +- `ios_mksysb_name` - IOS_MKSYSB resource name on the NIM Master server for the specified VIOS installation +- `viosupgrade_type` - Instalation type - values: bosinst (installation on rootvg) or altdisk (installation on an alternative disk) +- `installdisks` - List of hdisks for the installation (used only for an altdisk installation) +- `altdisks` - List of hdisks on which the alternate disk copy will be created. A disk is automatically searched when no disk is specified for a VIOS +- `action_list` - Comma separated list of actions to perform on VIOSes(default: "validate,upgrade") possible values: check, altdisk_copy, validate, upgrade, altdisk_cleanup +- `resources` - List of configuration resources to be applied after the installation. Per VIOSes. +- `common_resources` - List of configuration resources to be applied after the installation. For all VIOSes. +- `preview` - (values: yes or no) (default: no) To select the install preview mode - validate operation. +- `viosupgrade_alt_disk_copy` -(values: yes or no) (default: no) - yes to use viosupgrade command to make an alternate disk copy (altdisks property will be used for the disks) + +#### Actions + +- `upgrade` - check, altdisk copy, validate, upgrade, cleanup depending on the action_list parameter. + +#### Examples + +```ruby +aix_viosupgrade "check the vios redundancy" do + targets "(vios1,vios2),(vios3,vios4)" + action_list "check" + action :upgrade +end + +aix_viosupgrade "build an alternate rootvg" do + targets "(vios1,vios2),(vios3,vios4),(vios5)" + altdisks "(hdisk1,hdisk2)(hdisk1,)()" + action_list "altdisk_copy" + action :upgrade +end + +aix_viosupgrade 'viosupgrade VIOSES on current rootvg' do + targets '(vios1,vios1),(vios3,vios4)' + ios_mksysb_name 'ios_1844B_72M' + viosupgrade_type 'bosinst' + common_resources 'master_net_conf' + action_list 'validate,upgrade' +end + +aix_viosupgrade 'viosupgrade VIOSES on alternate disks ' do + targets '(vios1,vios1),(vios3,vios4)' + ios_mksysb_name 'ios_1844B_72M' + viosupgrade_type 'altdisk' + installdisks '(hdisk2,hdisk2),(hdisk0,hdisk4)' + common_resources 'master_net_conf' + action_list 'validate,upgrade' +end + +aix_viosupgrade 'viosupgrade VIOSES on current rootvg using viosupgrade for building altinst_rootvg' do + targets '(vios1,vios1),(vios3,vios4)' + ios_mksysb_name 'ios_1844B_72M' + viosupgrade_type 'bosinst' + altdisks '(hdisk1,hdisk1),(hdisk2,hdisk2)' + common_resources 'master_net_conf' + action_list 'validate,upgrade' + viosupgrade_alt_disk_copy 'yes' +end + +aix_viosupgrade "remove alternate rootvg" do + targets "(vios1,vios2),(vios3,vios4),(vios5)" + action_list "altdisk_cleanup" + action :update +end + +``` + ### niminit Use niminit to configure the nimclient package. This will look if /etc/niminfo exists and create it if it does not exist. You can the use nimclient provider after niminiting the client. diff --git a/docs/Chef_Automate_infrastructure_updates_in_NIM_environment_v0.4.pdf b/docs/Chef_Automate_infrastructure_updates_in_NIM_environment_v0.4.pdf new file mode 100644 index 0000000..74316f6 Binary files /dev/null and b/docs/Chef_Automate_infrastructure_updates_in_NIM_environment_v0.4.pdf differ diff --git a/docs/Chef_Automate_vios_updates_in_NIM_environment_v1.1.pdf b/docs/Chef_Automate_vios_updates_in_NIM_environment_v1.1.pdf new file mode 100644 index 0000000..3982d99 Binary files /dev/null and b/docs/Chef_Automate_vios_updates_in_NIM_environment_v1.1.pdf differ diff --git a/docs/Chef_Automate_vios_viosupgrade_in_NIM_environment v0.1.pdf b/docs/Chef_Automate_vios_viosupgrade_in_NIM_environment v0.1.pdf new file mode 100644 index 0000000..bf6e1b2 Binary files /dev/null and b/docs/Chef_Automate_vios_viosupgrade_in_NIM_environment v0.1.pdf differ diff --git a/examples/vios_upgrade.rb b/examples/vios_upgrade.rb new file mode 100644 index 0000000..a7690e3 --- /dev/null +++ b/examples/vios_upgrade.rb @@ -0,0 +1,22 @@ +# upgrade the VIOSES using viosupgrade command + +Chef::Recipe.send(:include, AIX::PatchMgmt) + +aix_viosupgrade 'viosupgrade VIOSES on alternate disks ' do + targets '(vios1,vios1),(vios3,vios4)' + ios_mksysb_name 'ios_1844B_72M' + viosupgrade_type 'altdisk' + installdisks '(hdisk2,hdisk2),(hdisk0,hdisk4)' + common_resources 'master_net_conf' + action_list 'validate,upgrade' + +end + +# aix_viosupgrade 'viosupgrade VIOSES on current rootvg' do + # targets '(vios1,vios1),(vios3,vios4)' + # ios_mksysb_name 'ios_1844B_72M' + # viosupgrade_type 'bosinst' + # common_resources 'master_net_conf' + # action_list 'validate,upgrade' +# end + diff --git a/libraries/patch_mgmt.rb b/libraries/patch_mgmt.rb index 44c7f1b..d032d32 100644 --- a/libraries/patch_mgmt.rb +++ b/libraries/patch_mgmt.rb @@ -63,7 +63,7 @@ def free_vios_disks(vios) disks = {} cmd_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{vios} \"/usr/ios/cli/ioscli lspv -free\"" - Open3.popen3({ 'LANG' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| stderr.each_line do |line| STDERR.puts line log_info("[STDERR] #{line.chomp}") @@ -229,6 +229,9 @@ class NimRemoveError < NimError class NimHmcInfoError < StandardError end + class NimCecsInfoError < StandardError + end + class NimLparInfoError < StandardError end @@ -241,12 +244,24 @@ class NimAltDiskInstallTimedOut < StandardError class ViosCmdError < StandardError end + class ViosUpgradeQueryError < StandardError + end + class AltDiskFindError < StandardError end class AltDiskCleanError < StandardError end + class UnMirrorError < StandardError + end + + class MirrorError < StandardError + end + + class CmdError < StandardError + end + class SpLevel include Comparable attr_reader :aix @@ -357,7 +372,7 @@ def duration(d) def metadata suma_s = @suma_s + ' -a Action=Metadata' log_debug("SUMA metadata operation: #{suma_s}") - exit_status = Open3.popen3({ 'LANG' => 'C' }, suma_s) do |_stdin, stdout, stderr, wait_thr| + exit_status = Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, suma_s) do |_stdin, stdout, stderr, wait_thr| stdout.each_line do |line| log_info("[STDOUT] #{line.chomp}") end @@ -376,7 +391,7 @@ def preview suma_s = @suma_s + ' -a Action=Preview' log_debug("SUMA preview operation: #{suma_s}") do_not_error = false - exit_status = Open3.popen3({ 'LANG' => 'C' }, suma_s) do |_stdin, stdout, stderr, wait_thr| + exit_status = Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, suma_s) do |_stdin, stdout, stderr, wait_thr| stdout.each_line do |line| @dl = Regexp.last_match(1).to_f / 1024 / 1024 / 1024 if line =~ /Total bytes of updates downloaded: ([0-9]+)/ @downloaded = Regexp.last_match(1) if line =~ /([0-9]+) downloaded/ @@ -408,7 +423,7 @@ def download download_failed = 0 download_skipped = 0 puts "Start downloading #{@downloaded} fixes (~ #{@dl.to_f.round(2)} GB) to '#{@dl_target}' directory." - exit_status = Open3.popen3({ 'LANG' => 'C' }, suma_s) do |_stdin, stdout, stderr, wait_thr| + exit_status = Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, suma_s) do |_stdin, stdout, stderr, wait_thr| thr = Thread.new do start = Time.now loop do @@ -448,6 +463,7 @@ def download # N I M # ################# class Nim + require 'date' include AIX::PatchMgmt def exist?(resource) @@ -493,7 +509,7 @@ def perform_async_customization(lpp_source, clients) log_debug("NIM asynchronus cust operation: #{nim_s}") puts "\nStart updating machine(s) '#{clients}' to #{lpp_source}." do_not_error = false - exit_status = Open3.popen3({ 'LANG' => 'C' }, nim_s) do |_stdin, stdout, stderr, wait_thr| + exit_status = Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, nim_s) do |_stdin, stdout, stderr, wait_thr| stdout.each_line do |line| do_not_error = true if line =~ /Either the software is already at the same level as on the media, or/ log_info("[STDOUT] #{line.chomp}") @@ -514,7 +530,7 @@ def perform_sync_customization(lpp_source, clients) log_debug("NIM synchronous cust operation: #{nim_s}") puts "Start updating machine(s) '#{clients}' to #{lpp_source}." do_not_error = false - exit_status = Open3.popen3({ 'LANG' => 'C' }, nim_s) do |_stdin, stdout, stderr, wait_thr| + exit_status = Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, nim_s) do |_stdin, stdout, stderr, wait_thr| stdout.each_line do |line| print "\033[2K\r#{line.chomp}" if line =~ /^Filesets processed:.*?[0-9]+ of [0-9]+/ print "\033[2K\r#{line.chomp}" if line =~ /^Finished processing all filesets./ @@ -536,7 +552,7 @@ def perform_efix_customization(lpp_source, client, filesets = 'all') nim_s = "/usr/sbin/nim -o cust -a lpp_source=#{lpp_source} -a filesets='#{filesets}' #{client}" log_debug("NIM install efixes cust operation: #{nim_s}") puts "Start patching machine(s) '#{client}'." - exit_status = Open3.popen3({ 'LANG' => 'C' }, nim_s) do |_stdin, stdout, stderr, wait_thr| + exit_status = Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, nim_s) do |_stdin, stdout, stderr, wait_thr| thr = Thread.new do loop do print '.' @@ -568,7 +584,7 @@ def perform_efix_vios_customization(lpp_source, vios, _filesets = 'all') nim_s = "/usr/sbin/nim -o updateios -a preview=no -a lpp_source=#{lpp_source} #{vios}" log_debug("NIM updateios operation: #{nim_s}") puts "Start patching machine(s) '#{vios}'." - exit_status = Open3.popen3({ 'LANG' => 'C' }, nim_s) do |_stdin, stdout, stderr, wait_thr| + exit_status = Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, nim_s) do |_stdin, stdout, stderr, wait_thr| thr = Thread.new do loop do print '.' @@ -596,6 +612,168 @@ def perform_efix_vios_customization(lpp_source, vios, _filesets = 'all') raise NimCustError, "Error: Command \"#{nim_s}\" returns above error!" unless exit_status.success? end + # ----------------------------------------------------------------- + # Get packaging date from fileset + # + # return packaging date formatted as following + # 20180809050125 for "Thu Aug 9 05:01:25 CDT 2018" + # raise CmdError in case of error + # ----------------------------------------------------------------- + def get_pkg_date(lpp_source_dir, fileset) + pkg_date = '' + cmd_s = "/usr/sbin/emgr -d -e #{lpp_source_dir}/#{fileset} -v3 | /bin/grep -w 'PACKAGING DATE' | /bin/cut -c16-" + log_debug("get_pkg_date: #{cmd_s}") + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stderr.each_line do |line| + STDERR.puts line + log_info("[STDERR] #{line.chomp}") + end + unless wait_thr.value.success? + stdout.each_line { |line| log_info("[STDOUT] #{line.chomp}") } + raise CmdError, "Error: Command \"#{cmd_s}\" returns above error!" + end + + stdout.each_line do |line| + log_info("[STDOUT] #{line.chomp}") + # match "Thu Aug 9 05:01:25 CDT 2018" + next unless line =~ /^\D+\s(\D+)\s(\d+)\s(\d+):(\d+):(\d+)\s\D+(\d+)$/ + date_h = Regexp.last_match(3) + date_m = Regexp.last_match(4) + date_s = Regexp.last_match(5) + dp = Date.parse(line) + pkg_date = dp.year.to_s + format('%02d', dp.mon) + format('%02d', dp.mday) + date_h + date_m + date_s + end + end + log_debug("get_pkg_date for: #{lpp_source_dir}/#{fileset} => #{pkg_date}") + pkg_date + end + + # ----------------------------------------------------------------- + # Sort fileset list by packaging date + # + # return sorted list of fileset + # ----------------------------------------------------------------- + def efix_sort_by_packaging_date(lpp_source_dir, filesets) + pkg_date_h = {} + efixes_t = [] + filesets.each do |fileset| + begin + pkg_date_h[fileset] = get_pkg_date(lpp_source_dir, fileset) + rescue CmdError => e + log_debug("efix_sort_by_packaging_date -> get_pkg_date Error: #{e}") + end + end + pkg_date_h.sort_by { |_, date| date }.each do |key, _| + efixes_t << key + end + efixes_t.reverse! + end + + # ----------------------------------------------------------------- + # name : get_fileset_files_loc + # param : input:lpp_source_dir:string + # param : input:fileset:string + # + # return array of packaging names + # description : get package names impacted from specific fileset + # raise CmdError in case of error + # ----------------------------------------------------------------- + def get_fileset_files_loc(lpp_source_dir, + fileset) + log_debug("Into get_fileset_files_loc: lpp_source_dir=#{lpp_source_dir}, fileset=#{fileset}") + loc_files = [] + cmd_s = "/usr/sbin/emgr -d -e #{lpp_source_dir}/#{fileset} -v3 | /bin/grep -w 'LOCATION:' | /bin/cut -c17-" + log_info("get_fileset_files_loc: #{cmd_s}") + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stderr.each_line do |line| + log_debug("[STDERR] #{line.chomp}") + end + unless wait_thr.value.success? + stdout.each_line { |line| log_debug("[STDOUT] #{line.chomp}") } + raise CmdError, "Error: Command \"#{cmd_s}\" returns above error!" + end + stdout.each_line do |line| + log_debug("[STDOUT] #{line.chomp}") + next unless line.include?('/') + loc_files << line.strip + end + end + loc_files.uniq! + log_info("get_fileset_files_loc for: #{lpp_source_dir}/#{fileset} => #{loc_files}") + loc_files + end + + # ----------------------------------------------------------------- + # name : get_efix_files_loc + # param : input:lpp_source_dir:string + # param : input:filesets:array + # + # return hash table locked files (key = fileset) + # description : get impacted location files from fileset list + # ----------------------------------------------------------------- + def get_efix_files_loc(lpp_source_dir, + filesets) + log_info("Into get_efix_files_loc: lpp_source_dir=#{lpp_source_dir}, fileset=#{filesets}") + loc_files_h = {} + filesets.each do |fileset| + begin + loc_files_h[fileset] = get_fileset_files_loc(lpp_source_dir, fileset) + rescue CmdError => e + log_info("get_efix_files_loc -> get_fileset_files_loc Error: #{e}") + end + end + loc_files_h + end + + # ----------------------------------------------------------------- + # Get the list of the cec defined on the nim master and gat their serial number. + # + # return a dic with cecs info: + # the name of the cec objects defined on the nim master + # and their associated CEC serial number value + # raise NimCecsInfoError in case of error + # ----------------------------------------------------------------- + def get_cecs_info + info_hash = {} + obj_key = '' + cmd_s = '/usr/sbin/lsnim -t cec -l' + log_debug("get_cecs_info: #{cmd_s}") + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stderr.each_line do |line| + STDERR.puts line + log_info("[STDERR] #{line.chomp}") + end + unless wait_thr.value.success? + stdout.each_line { |line| log_info("[STDOUT] #{line.chomp}") } + raise NimCecsInfoError, "Error: Command \"#{cmd_s}\" returns above error!" + end + + stdout.each_line do |line| + log_info("[STDOUT] #{line.chomp}") + # lpar name + if line =~ /^(\S+):/ + obj_key = Regexp.last_match(1) + info_hash[obj_key] = {} + next + end + # cec serial number + if line =~ /^\s+serial\s+=\s+(.*)$/ + info_hash[obj_key]['serial'] = Regexp.last_match(1) + next + end + end + end + + log_info('CECs information:') + info_hash.keys.each do |key| + log_info(key.to_s) + info_hash[key].keys.each do |k| + log_info(" #{k}: #{info_hash[key][k]}") + end + end + info_hash + end + # ----------------------------------------------------------------- # Get the hmc info on the nim master # @@ -607,7 +785,7 @@ def get_hmc_info obj_key = '' cmd_s = '/usr/sbin/lsnim -t hmc -l' log_debug("get_hmc_info: #{cmd_s}") - Open3.popen3({ 'LANG' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| stderr.each_line do |line| STDERR.puts line log_info("[STDERR] #{line.chomp}") @@ -673,7 +851,7 @@ def get_nim_clients_info(lpar_type) obj_key = '' cmd_s = "/usr/sbin/lsnim -t #{lpar_type} -l" log_debug("get_nim_clients_info: '#{cmd_s}'") - Open3.popen3({ 'LANG' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| stderr.each_line do |line| STDERR.puts line log_info("[STDERR] #{line.chomp}") @@ -693,8 +871,7 @@ def get_nim_clients_info(lpar_type) end # Cstate if line =~ /^\s+Cstate\s+=\s+(.*)$/ - cstate = Regexp.last_match(1) - info_hash[obj_key]['cstate'] = cstate + info_hash[obj_key]['cstate'] = Regexp.last_match(1) next end @@ -725,6 +902,77 @@ def get_nim_clients_info(lpar_type) info_hash end + # ----------------------------------------------------------------- + # Run NIM unmirror command + # + # raise UnMirrorError in case of error + # ----------------------------------------------------------------- + def perform_unmirror(nim_vios, vios, vg_name) + ret = 0 + success_unmirror = 1 + cmd_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{nim_vios[vios]['vios_ip']} \"/usr/sbin/unmirrorvg #{vg_name} 2>&1 \"" + + log_info("perform_unmirror: '#{cmd_s}'") + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, _stderr, wait_thr| + stdout.each_line do |line| + STDOUT.puts line + log_info("[STDOUT] #{line.chomp}") + success_unmirror = 0 if line.include? 'successfully unmirrored' + end + unless wait_thr.value.success? + stdout.each_line { |line| log_info("[STDOUT] #{line.chomp}") } + msg = "Failed to unmirror '#{vg_name}' on #{vios}, command \"#{cmd_s}\" returns above error!" + raise UnMirrorError, msg + end + end + if success_unmirror == 1 + msg = "Failed to unmirror '#{vg_name}' on '#{vios}'" + put_error(msg) + raise UnMirrorError, msg + end + put_info("Unmirror '#{vg_name}' on '#{vios}' successful.") + ret + end + + # ----------------------------------------------------------------- + # Run NIM mirror command + # + # raise MirrorError in case of error + # ----------------------------------------------------------------- + def perform_mirror(nim_vios, vios, vg_name, vg_info) + ret = 0 + copies_h = vg_info[vios]['copy_dict'] + nb_copies = copies_h.keys.length + success_mirror = -1 + if nb_copies > 1 + success_mirror = 0 + cmd_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{nim_vios[vios]['vios_ip']} \"/usr/sbin/mirrorvg -m -c #{nb_copies} #{vg_name} #{copies_h[2]} \"" + cmd_s += copies_h[3] if nb_copies > 2 + cmd_s += ' 2>&1' + + log_info("perform_mirror: '#{cmd_s}'") + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, _stderr, wait_thr| + stdout.each_line do |line| + STDOUT.puts line + log_info("[STDERR] #{line.chomp}") + success_mirror = 1 if line.include? 'Failed to mirror the volume group' + end + unless wait_thr.value.success? + stdout.each_line { |line| log_info("[STDOUT] #{line.chomp}") } + msg = "Failed to mirror '#{vg_name}' on #{vios}, command \"#{cmd_s}\" returns above error!" + raise MirrorError, msg + end + end + if success_mirror == 1 + msg = "Failed to mirror '#{vg_name}' on '#{vios}'" + put_error(msg) + raise MirrorError, msg + end + end + put_info("Mirror '#{vg_name}' on '#{vios}' successful.") if success_mirror == 0 + ret + end + # ----------------------------------------------------------------- # Run the NIM alt_disk_install command to launch # the alternate copy operation on specified vios @@ -766,7 +1014,7 @@ def perform_altdisk_install(vios, source, disk, set_bootlist = 'no', boot_client # 1 if the alt_disk_install operation failed # -1 if the alt_disk_install operation timed out # - # raise NimLparInfoError if cannot get NIM state + # raise NimLparInfoError if cannot get NIM state+ # ----------------------------------------------------------------- def wait_alt_disk_install(vios, check_count = 180, sleep_time = 10) nim_info_prev = '___' # this info should not appears in nim info attribute @@ -783,7 +1031,7 @@ def wait_alt_disk_install(vios, check_count = 180, sleep_time = 10) nim_result = '' nim_info = '' - Open3.popen3({ 'LANG' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| stderr.each_line do |line| STDERR.puts line log_info("[STDERR] #{line.chomp}") @@ -845,6 +1093,156 @@ def wait_alt_disk_install(vios, check_count = 180, sleep_time = 10) put_error(msg) -1 end + + # ----------------------------------------------------------------- + # check if viosupgrade is finish + # + # return 0 if finish + # return 1 if not finish + # return -1 if error detected + # raise ViosUpgradeQueryError in case of error + # ----------------------------------------------------------------- + def viosupgrade_query_status(vios) + rc = 1 + nb_check = 0 + # cmd_s = "/usr/sbin/viosupgrade -q -n #{vios}" + cmd_s = "/usr/sbin/lsnim -l #{vios}" + err_info = false + log_info("viosupgrade_query_status: '#{cmd_s}'") + exit_status = Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stdout.each_line do |line| + log_info("[STDOUT] #{line.chomp}") + line.chomp! + line.strip! + nb_check += 1 if line =~ /^Cstate\s+=\s+ready for a NIM operation$/ + nb_check += 1 if line =~ /^Mstate\s+=\s+ready for use$/ + nb_check += 1 if line =~ /^Mstate\s+=\s+currently running$/ + nb_check += 1 if line =~ /^Cstate_result\s+=\s+success$/ + nb_check -= 1 if line =~ /^info\s+=/ + err_info = true if line =~ /^err_info\s+=/ + end + stderr.each_line do |line| + STDERR.puts line + log_info("[STDERR] #{line.chomp}") + end + wait_thr.value # Process::Status object returned. + end + raise ViosUpgradeQueryError, "viosupgrade_query_status: #{cmd_s}' returns above error." unless exit_status.success? + rc = 0 if nb_check == 3 + rc = -1 if err_info + rc + end + + # ----------------------------------------------------------------- + # get cluster status + # + # + # Raise ViosCmdError in case of command error + # ----------------------------------------------------------------- + def get_cluster_status(nim_vios, vios) + rc = 1 + cmd_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{nim_vios[vios]['vios_ip']} \"/usr/ios/cli/ioscli cluster -status -fmt :\"" + log_info("get_cluster_status: '#{cmd_s}'") + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stderr.each_line do |line| + STDERR.puts line + log_info("[STDERR] #{line.chomp}") + end + unless wait_thr.value.success? + stdout.each_line { |line| log_info("[STDOUT] #{line.chomp}") } + msg = "Failed to get cluster status on #{vios}, command \"#{cmd_s}\" returns above error!" + raise ViosCmdError, msg + end + # stdout is like: + # p7juf_cluster:OK:p7jufv1:8205-E6C020686AFR:1:OK:OK + # p7juf_cluster:OK:p7jufv2:8205-E6C020686AFR:7:OK:OK + stdout.each_line do |line| + log_debug("[STDOUT] #{line.chomp}") + line.chomp! + line.strip! + if line =~ /^(\S+):(\S+):(\S+):(\S+):(\d+):(\S+):(.*)/ + rc = if Regexp.last_match(6) == 'DOWN' || Regexp.last_match(7) == 'DOWN' + 1 + else + 0 + end + end + end + end + rc + end + + # ----------------------------------------------------------------- + # Wait for viosupgrade operation to finish + # + # when viosupgrade operation ends the NIM object state changes + # + # You migh want a timeout of 60 minutes (count=360, sleep=10s) + # + # Return + # 0 if the viosupgrade operation ends with success + # 1 if the viosupgrade operation failed + # -1 if the viosupgrade operation timed out + # + # raise ViosUpgradeQueryError if cannot get viosupgrade status + # rubocop:disable Style/GuardClause + # ----------------------------------------------------------------- + def wait_viosupgrade(nim_vios, vios, check_count = 360, sleep_time = 10) + count = 0 + wait_time = 0 + + log_info("wait_viosupgrade for vios: '#{vios}'") + upgrade_status_ok = false + st_upg = -2 + st_clust = -2 + + while count <= check_count + sleep(sleep_time) + wait_time += 10 + + unless upgrade_status_ok + begin + st_upg = viosupgrade_query_status(vios) + log_info("viosupgrade succeded for vios: '#{vios}'") if st_upg == 0 + rescue ViosUpgradeQueryError => e + msg = "viosuprade status error: #{e.message}" + put_error(msg) + end + end + case st_upg + when 0 + upgrade_status_ok = true + sleep 120 + # if cluster defined - check if the cluster is started if exists + log_debug("--- Check if cluster ssp_id: '#{nim_vios[vios]['ssp_id']}'") + if nim_vios[vios]['ssp_id'] != 'none' && nim_vios[vios]['ssp_id'] != '' + begin + st_clust = get_cluster_status(nim_vios, vios) + rescue ViosCmdError => e + msg = "Cluster status error: #{e.message}" + put_error(msg) + end + # success with cluster check + return 0 if st_clust == 0 + else + # success with no cluster check + return 0 + end + when -1 + # error detected + return 1 + end + if wait_time.modulo(60) == 0 + msg = "Waiting VIOSUPGRADE on #{vios}... duration: #{wait_time / 60} minute(s)" + print("\033[2K\r#{msg}") + log_info(msg) + end + end # while count + # timed out before the end of viosupgrade + msg = "VIOS UPGRADE OPERATION for #{vios} TIME OUT #{count * sleep_time / 60} minute(s)" + put_error(msg) + -1 + end end # Nim ################# @@ -866,7 +1264,7 @@ def get_pvs(nim_vios, vios) cmd_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{nim_vios[vios]['vios_ip']} \"/usr/ios/cli/ioscli lspv\"" log_debug("get_pvs: '#{cmd_s}'") - Open3.popen3({ 'LANG' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| stderr.each_line do |line| STDERR.puts line log_info("[STDERR] #{line.chomp}") @@ -914,7 +1312,7 @@ def get_free_pvs(nim_vios, vios) cmd_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{nim_vios[vios]['vios_ip']} \"/usr/ios/cli/ioscli lspv -free\"" log_debug("get_free_pvs: '#{cmd_s}'") - Open3.popen3({ 'LANG' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| stderr.each_line do |line| STDERR.puts line log_info("[STDERR] #{line.chomp}") @@ -959,7 +1357,7 @@ def get_vg_size(nim_vios, vios, vg_name) cmd_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{nim_vios[vios]['vios_ip']} \"/usr/ios/cli/ioscli lsvg #{vg_name}\"" log_info("get_vg_size: '#{cmd_s}'") - Open3.popen3({ 'LANG' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| stderr.each_line do |line| STDERR.puts line log_info("[STDERR] #{line.chomp}") @@ -998,6 +1396,240 @@ def get_vg_size(nim_vios, vios, vg_name) [vg_size, used_size] end + # ----------------------------------------------------------------- + # Return the vg pp size + # + # Raise ViosCmdError in case of error + # ----------------------------------------------------------------- + def get_vg_pp_size(nim_vios, vios, vg_name) + vg_pp_size = -1 + + cmd_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{nim_vios[vios]['vios_ip']} \"/usr/sbin/lsvg #{vg_name}\"" + + log_info("get_vg_size: '#{cmd_s}'") + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stderr.each_line do |line| + STDERR.puts line + log_info("[STDERR] #{line.chomp}") + end + unless wait_thr.value.success? + stdout.each_line { |line| log_info("[STDOUT] #{line.chomp}") } + msg = "Failed to get Volume Group '#{vg_name}' size on #{vios}, command \"#{cmd_s}\" returns above error!" + raise ViosCmdError, msg + end + + # stdout is like: + # parse lsvg outpout to get the size in megabytes: + # VG PERMISSION: read/write TOTAL PPs: 558 (285696 megabytes) + stdout.each_line do |line| + log_debug("[STDOUT] #{line.chomp}") + line.chomp! + + if line =~ /.*PP SIZE:\s+(\d+)\s+megabyte\(s\).*/ + vg_pp_size = Regexp.last_match(1).to_i + next + end + end + end + + if vg_pp_size == -1 + msg = "Failed to get '#{vg_name}' pp size on #{vios}" + raise ViosCmdError, msg + end + + log_info("VG '#{vg_name}' size =#{vg_pp_size} PPs") + vg_pp_size + end + + # ----------------------------------------------------------------- + # Return pv_size (PPs) from vg for hdisk (return -1 in case of + # parsing error + # + # Raise ViosCmdError in case of command error + # ----------------------------------------------------------------- + def get_pv_size_from_hdisk(nim_vios, vios, vg_name, hdisk) + pv_size = -1 + cmd_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{nim_vios[vios]['vios_ip']} \"/usr/sbin/lsvg -p #{vg_name}\"" + + log_info("get_pv_size_from_hdisk: '#{cmd_s}'") + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stderr.each_line do |line| + STDERR.puts line + log_info("[STDERR] #{line.chomp}") + end + unless wait_thr.value.success? + stdout.each_line { |line| log_info("[STDOUT] #{line.chomp}") } + msg = "Failed to get Volume Group '#{vg_name}' size on #{vios}, command \"#{cmd_s}\" returns above error!" + raise ViosCmdError, msg + end + + # stdout is like: + # parse lsvg outpout to get the size in megabytes: + # rootvg: + # PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION + # hdisk4 active 639 254 126..00..00..00..128 + stdout.each_line do |line| + log_debug("[STDOUT] #{line.chomp}") + line.chomp! + if line =~ /^(\S+)\s+\S+\s+(\d+)\s+\d+\s+\S+/ + pv_size = Regexp.last_match(2).to_i + break if Regexp.last_match(1) == hdisk + end + end + end + + if pv_size == -1 + msg = "Failed to get pv size on #{vios} from #{vg_name} for #{hdisk}" + put_error(msg) + end + pv_size + end + + # ---------------------------------------------------------------- + # Check the rootvg + # - check if the rootvg is mirrored + # - check stale partitions + # - calculate the total and used size of the rootvg + # return: + # hach table with following keys: value + # "status": + # 0 the rootvg can be saved in a alternate disk copy + # 1 otherwise (cannot unmirror then mirror again) + # "copy_dict": + # dictionary key, value + # key: copy number (int) + # value: hdiskx + # example: {1: 'hdisk4', : 2: 'hdisk8', 3: 'hdisk9'} + # "rootvg_size": size in Megabytes (int) + # "used_size": size in Megabytes (int) + # ---------------------------------------------------------------- + def check_rootvg(nim_vios, vios) + vg_info = {} + copy_dict = {} + vg_info['status'] = 1 + vg_info['rootvg_size'] = 0 + vg_info['used_size'] = 0 + + nb_lp = 0 + copy = 0 + hdisk_dict = {} + hdisk = '' + + # The lsvg -M command lists the physical disks that contain the various logical volumes. + cmd_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{nim_vios[vios]['vios_ip']} \"/usr/sbin/lsvg -M rootvg\"" + log_info("check_rootvg: '#{cmd_s}'") + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stderr.each_line do |line| + STDERR.puts line + log_info("[STDERR] #{line.chomp}") + end + unless wait_thr.value.success? + stdout.each_line { |line| log_info("[STDOUT] #{line.chomp}") } + msg = 'Failed to get the physical disks from rootvg' + log_warn("[#{vios}] #{msg}") + raise ViosCmdError, "Error: #{msg} on #{vios}, command \"#{cmd_s}\" returns above error!" + end + + # lsvg -M rootvg command OK, check mirroring + # lists all PV, LV, PP details of a vg (PVname:PPnum LVname: LPnum :Copynum) + # hdisk4:453 hd1:101 + # hdisk4:454 hd1:102 + # hdisk4:257 hd10opt:1:1 + # hdisk4:258 hd10opt:2:1 + # hdisk4:512-639 + # hdisk8:255 hd1:99:2 stale + # hdisk8:256 hd1:100:2 stale + # hdisk8:257 hd10opt:1:2 + # hdisk8:258 hd10opt:2:2 + # .. + # hdisk9:257 hd10opt:1:3 + stdout.each_line do |line| + line.chomp! + log_debug("[STDOUT] #{line}") + # case: hdisk8:257 hd10opt:1:2 + if line.strip =~ /^(\S+):\d+\s+\S+:\d+:(\d+)$/ + hdisk = Regexp.last_match(1) + copy = Regexp.last_match(2).to_i + elsif line.strip =~ /^(\S+):\d+\s+\S+:\d+$/ + # case: hdisk8:258 hd10opt:2 + hdisk = Regexp.last_match(1) + copy = 1 + else + if line.include? 'stale' + msg = "#{vios} rootvg contains stale partitions" + STDERR.puts msg + STDERR.puts stdout + log_warn("[#{vios}] #{msg}") + return vg_info + end + next + end + + nb_lp += 1 if copy == 1 + + if hdisk_dict.key?(hdisk) + if hdisk_dict[hdisk] != copy + msg = 'rootvg data structure is not compatible with an '\ + 'alt_disk_copy operation (2 copies on the same disk)' + put_error(msg) + return vg_info + end + else + hdisk_dict[hdisk] = copy + end + + next unless copy_dict.key?(copy) + if copy_dict.value?(hdisk) + msg = 'rootvg data structure is not compatible with an alt_disk_copy operation' + put_error(msg) + return vg_info + end + copy_dict[copy] = hdisk + end + end + + if copy_dict.keys.length > 1 + if copy_dict.keys.length != hdisk_dict.keys.length + msg = "The #{vios} rootvg is partially or completly mirrored but some "\ + 'LP copies are spread on several disks. This prevent the '\ + 'system from building an alternate rootvg disk copy' + put_error(msg) + return vg_info + end + end + + # the (rootvg) is mirrored then get the size of hdisk from copy1 + begin + pv_size = get_pv_size_from_hdisk(nim_vios, vios, 'rootvg', copy_dict[1]) + rescue ViosCmdError => e + msg = "Failed to get pv size of hdisk from copy1: #{vios}: #{e.message}" + put_warn(msg) + return vg_info + end + return vg_info if pv_size == -1 + + # get now the rootvg pp size + begin + pp_size = get_vg_pp_size(nim_vios, vios, 'rootvg') + rescue ViosCmdError => e + msg = "Failed to get rootvg pp size: #{vios}: #{e.message}" + put_warn(msg) + return vg_info + end + return vg_info if pp_size == -1 + + total_size = pp_size * pv_size + used_size = pp_size * (nb_lp + 1) + + vg_info['status'] = 0 + vg_info['copy_dict'] = copy_dict + vg_info['rootvg_size'] = total_size + vg_info['used_size'] = used_size + + log_debug("VG INFO : #{vg_info}\n") + vg_info + end + # ----------------------------------------------------------------- # Find a valid alternate disk that # - exists, @@ -1016,13 +1648,21 @@ def get_vg_size(nim_vios, vios, vg_name) # # Raise AltDiskFindError in case of error # ----------------------------------------------------------------- - def find_valid_altdisk(nim_vios, vios_list, vios_key, targets_status, altdisk_hash, disk_size_policy) + # rubocop:disable Metrics/ParameterLists + def find_valid_altdisk(nim_vios, vios_list, vios_key, rootvg_info, targets_status, altdisk_hash, disk_size_policy) rootvg_size = 0 used_size = 0 used_pv = [] vios_list.each do |vios| err_label = 'FAILURE-ALTDC1' + # check rootvg + if rootvg_info[vios]['status'] != 0 + targets_status[vios_key] = "#{err_label} wrong rootvg state on #{vios}" + put_error("wrong rootvg state on #{vios}") + return 1 + end + # get pv list begin get_pvs(nim_vios, vios) @@ -1039,12 +1679,8 @@ def find_valid_altdisk(nim_vios, vios_list, vios_key, targets_status, altdisk_ha return 1 end - begin - rootvg_size, used_size = get_vg_size(nim_vios, vios, 'rootvg') - rescue ViosCmdError => e - msg = "Failed to find a valid alternate disk on #{vios}: #{e.message}" - raise AltDiskFindError, msg - end + used_size = rootvg_info[vios]['used_size'] + rootvg_size = rootvg_info[vios]['rootvg_size'] begin get_free_pvs(nim_vios, vios) @@ -1053,7 +1689,7 @@ def find_valid_altdisk(nim_vios, vios_list, vios_key, targets_status, altdisk_ha raise AltDiskFindError, msg end - if nim_vios[vios]['free_pvs'] == {} + if nim_vios[vios]['free_pvs'].empty? targets_status[vios_key] = "#{err_label} no disk available on #{vios}" put_error("No disk available on #{vios}") return 1 @@ -1184,7 +1820,6 @@ def find_valid_altdisk(nim_vios, vios_list, vios_key, targets_status, altdisk_ha # ----------------------------------------------------------------- def get_altinst_rootvg_disk(nim_vios, vios, altdisk_hash) ret = 0 - begin get_pvs(nim_vios, vios) rescue ViosCmdError => e @@ -1379,7 +2014,7 @@ def list_fixes(machine) array_fixes = [] emgr_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{machine} \"/usr/sbin/emgr -l\"" log_debug("EMGR list: #{emgr_s}") - exit_status = Open3.popen3({ 'LANG' => 'C' }, emgr_s) do |_stdin, stdout, stderr, wait_thr| + exit_status = Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, emgr_s) do |_stdin, stdout, stderr, wait_thr| stdout.each_line do |line| line_array = line.split(' ') if line_array[0] =~ /[0-9]/ @@ -1398,6 +2033,87 @@ def list_fixes(machine) array_fixes end + # ----------------------------------------------------------------- + # name : get_locked_files + # param : input:target:string + # + # return array of locked files for a specific target + # description : get files impacted from specific fileset + # raise CmdError in case of error + # ----------------------------------------------------------------- + def get_locked_files(target) + log_info('Into get_locked_files (target=' + target + ')') + locked_files = [] + locked_labels = [] + # get efix label already installed + emgr_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{target} \"/usr/sbin/emgr -P\"" + log_info("EMGR efix already install: #{emgr_s}") + exit_status = Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, emgr_s) do |_stdin, stdout, stderr, wait_thr| + stdout.each_line do |line| + next if line =~ /^PACKAGE\s*INSTALLER\s*LABEL/ + next if line =~ /^=*\s\=*\s\=*/ + line_array = line.split(' ') + log_debug("emgr: adding locked file #{line_array[2]} to locked files list") + locked_labels.push(line_array[2]) + log_debug("[STDOUT] #{line.chomp}") + end + stderr.each_line do |line| + log_debug("[STDERR] #{line.chomp}") + end + wait_thr.value # Process::Status object returned. + end + raise CmdError, "Error: Command \"#{emgr_s}\" returns above error!" unless exit_status.success? + locked_labels.delete_if { |item| item.nil? || item.empty? } + locked_labels.uniq! + log_info("get_locked_files : get labels for: #{target} => #{locked_labels}") + locked_labels.each do |label| + files = get_efix_files(target, label) + files.each do |file| + locked_files << file + end + end + locked_files.uniq! + log_info("get_locked_files : for: #{target} => #{locked_files}") + locked_files + end + + # ----------------------------------------------------------------- + # name : get_efix_files + # param : input:target:string + # param : input:label:string + # + # return array of files + # description : get file locations impacted from specific efix + # raise CmdError in case of error + # ----------------------------------------------------------------- + def get_efix_files(target, + label) + log_info('Into get_efix_files (target=' + + target + + '), label=' + + label) + file_locations = [] + cmd_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{target} \"/usr/sbin/emgr -l -L #{label} -v3 | /bin/grep -w 'LOCATION:' | /bin/cut -c17-\"" + log_info("get_efix_files: #{cmd_s}") + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stderr.each_line do |line| + log_debug("[STDERR] #{line.chomp}") + end + unless wait_thr.value.success? + stdout.each_line { |line| log_info("[STDOUT] #{line.chomp}") } + raise CmdError, "Error: Command \"#{cmd_s}\" returns above error!" + end + stdout.each_line do |line| + log_debug("[STDOUT] #{line.chomp}") + next unless line.include?('/') + file_locations << line.strip + end + end + file_locations.uniq! + log_info("get_efix_files for: #{target} : #{label} => #{file_locations}") + file_locations + end + # ----------------------------------------------------------------- # Remove fix with emgr # @@ -1757,6 +2473,7 @@ def expand_vios_pair_targets(targets, vios_nim_list, altdisks, altdisk_hash) vios_list_tuples = targets.delete(' ').gsub('),(', ')(').split('(') vios_list_tuples.delete_at(0) # after the split, 1rst elt is nil + # vios_list_tuples.delete_at(0) if vios_list_tuples.length > 1 unless altdisks.nil? || altdisks == 'auto' # Remove the spaces added here after after the tuple lengh has beed tested @@ -1832,5 +2549,184 @@ def expand_vios_pair_targets(targets, vios_nim_list, altdisks, altdisk_hash) log_info("List of altdisk: #{altdisk_hash}") selected_vios end + + # ----------------------------------------------------------------- + # Buidl installdisk regarding the target vios pair list parameter + # + # targets are in the form (vios1,vios2) (vios3,vios4) (vios5) (vios6) + # + # for no installdisk checking installdisks should be nil otherwise + # it should be the keyword 'skip' or in the form + # (hdisk1,hdisk2) (hdisk1,) (hdisk5) () + # with the same number of hdisk than VIOSes even if empty + # + # raise InvalidTargetsProperty in case of error + # + # ----------------------------------------------------------------- + def build_installdisks(targets, vios_nim_list, installdisks) + installdisk_hash = {} + selected_vios = [] + vios_list = [] + + vios_list_tuples = targets.delete(' ').gsub('),(', ')(').split('(') + vios_list_tuples.delete_at(0) # after the split, 1rst elt is nil + + unless installdisks.nil? || installdisks == 'skip' + # Remove the spaces added here after after the tuple lengh has beed tested + hd_list_tuples = installdisks.delete(' ').gsub('),(', ')(').gsub('(,', '( ,').gsub(',)', ', )').split('(') + hd_list_tuples.delete_at(0) + if hd_list_tuples.length != vios_list_tuples.length + raise InvalidTargetsProperty, "Error: Install hdisks '#{installdisks}' and vios target '#{targets}' must have the same number of element" + end + end + + # Build targets list + hd_tuple_index = 0 + vios_list_tuples.each do |vios_tuple| + my_tuple = vios_tuple.delete(')') + tuple_elts = my_tuple.split(',') + tuple_len = tuple_elts.length + + # check targets has the form of (vios1,vios2) or (vios3) + if tuple_len != 1 && tuple_len != 2 + raise InvalidTargetsProperty, "Error: Malformed vios targets '#{targets}'" + end + + # check vios not already exists in the target list + if vios_list.include?(tuple_elts[0]) || + (tuple_len == 2 && (vios_list.include?(tuple_elts[1]) || + tuple_elts[0] == tuple_elts[1])) + raise InvalidTargetsProperty, "Error: Malformed vios targets, Duplicated values '#{targets}'" + end + + # check vios is knowed by the NIM master - if not ignore it + if !vios_nim_list.include?(tuple_elts[0]) || + tuple_len == 2 && !vios_nim_list.include?(tuple_elts[1]) + next + end + + if tuple_len == 2 + vios_list.push(tuple_elts[0], tuple_elts[1]) + else + vios_list.push(tuple_elts[0]) + end + selected_vios.push(my_tuple) + + # Handle hdisk list if installdisks not nil + next if installdisks.nil? + + # in skip mode, just add empty hdisk for the 2 vioses + if installdisks == 'skip' + installdisk_hash[tuple_elts[0]] = '' + installdisk_hash[tuple_elts[1]] = '' + next + end + + # parse the hdisk tuple + hd_tuple = hd_list_tuples[hd_tuple_index].delete(')') + hd_tuple_elts = hd_tuple.split(',') + hd_tuple_len = hd_tuple_elts.length + if hd_tuple_len != tuple_len + raise InvalidTargetsProperty, "Error: install hdsik tuple '#{hd_tuple}' and vios tuple '#{my_tuple}' must have the same number of element" + end + installdisk_hash[tuple_elts[0]] = hd_tuple_elts[0].delete(' ') + if tuple_len == 2 + installdisk_hash[tuple_elts[1]] = hd_tuple_elts[1].delete(' ') + end + hd_tuple_index += 1 + end + log_info("List of install disks: #{installdisk_hash}") + installdisk_hash + end + + # ----------------------------------------------------------------- + # Buidl resources regarding the target vios pair list parameter + # + # targets are in the form (vios1,vios2) (vios3,vios4) (vios5) (vios6) + # + # for no resources checking resources should be nil otherwise + # it should be the keyword 'skip' or in the form + # (res11:res12,res21) (res1,) (res) () + # with the same number of resource than VIOSes even if empty + # + # raise InvalidTargetsProperty in case of error + # + # ----------------------------------------------------------------- + def build_resources(targets, vios_nim_list, resources) + resource_hash = {} + selected_vios = [] + vios_list = [] + + vios_list_tuples = targets.delete(' ').gsub('),(', ')(').split('(') + vios_list_tuples.delete_at(0) # after the split, 1rst elt is nil + + unless resources.nil? || resources == 'skip' + # Remove the spaces added here after after the tuple lengh has beed tested + rs_list_tuples = resources.delete(' ').gsub('),(', ')(').gsub('(,', '( ,').gsub(',)', ', )').split('(') + rs_list_tuples.delete_at(0) + if rs_list_tuples.length != vios_list_tuples.length + raise InvalidTargetsProperty, "Error: Install hdisks '#{resources}' and vios target '#{targets}' must have the same number of element" + end + end + + # Build targets list + rs_tuple_index = 0 + vios_list_tuples.each do |vios_tuple| + my_tuple = vios_tuple.delete(')') + tuple_elts = my_tuple.split(',') + tuple_len = tuple_elts.length + + # check targets has the form of (vios1,vios2) or (vios3) + if tuple_len != 1 && tuple_len != 2 + raise InvalidTargetsProperty, "Error: Malformed vios targets '#{targets}'" + end + + # check vios not already exists in the target list + if vios_list.include?(tuple_elts[0]) || + (tuple_len == 2 && (vios_list.include?(tuple_elts[1]) || + tuple_elts[0] == tuple_elts[1])) + raise InvalidTargetsProperty, "Error: Malformed vios targets, Duplicated values '#{targets}'" + end + + # check vios is knowed by the NIM master - if not ignore it + if !vios_nim_list.include?(tuple_elts[0]) || + tuple_len == 2 && !vios_nim_list.include?(tuple_elts[1]) + next + end + + if tuple_len == 2 + vios_list.push(tuple_elts[0], tuple_elts[1]) + else + vios_list.push(tuple_elts[0]) + end + selected_vios.push(my_tuple) + + # Handle resources list if installdisks not nil + next if resources.nil? + + # in skip mode, just add empty hdisk for the 2 vioses + if resources == 'skip' + resource_hash[tuple_elts[0]] = '' + resource_hash[tuple_elts[1]] = '' + next + end + + # parse the resource tuple + rs_tuple = rs_list_tuples[rs_tuple_index].delete(')') + rs_tuple_elts = rs_tuple.split(',') + rs_tuple_len = rs_tuple_elts.length + if rs_tuple_len != tuple_len + raise InvalidTargetsProperty, "Error: install hdsik tuple '#{rs_tuple}' and vios tuple '#{my_tuple}' must have the same number of element" + end + resource_hash[tuple_elts[0]] = rs_tuple_elts[0].delete(' ') + if tuple_len == 2 + resource_hash[tuple_elts[1]] = rs_tuple_elts[1].delete(' ') + end + + rs_tuple_index += 1 + end + log_info("List of resources: #{resource_hash}") + resource_hash + end end # module PatchMgmt end # module AIX diff --git a/metadata.rb b/metadata.rb index 0686943..2796daa 100644 --- a/metadata.rb +++ b/metadata.rb @@ -4,7 +4,7 @@ license 'Apache-2.0' description 'Custom resources useful for AIX systems' long_description IO.read(File.join(File.dirname(__FILE__), 'README.md')) -version '2.3.0' +version '2.3.1' source_url 'https://github.com/chef-cookbooks/aix' issues_url 'https://github.com/chef-cookbooks/aix/issues' diff --git a/resources/flrtvc.rb b/resources/flrtvc.rb index 358880a..3b37673 100644 --- a/resources/flrtvc.rb +++ b/resources/flrtvc.rb @@ -34,6 +34,7 @@ property :path, String property :verbose, [true, false], default: false property :clean, [true, false], default: true +property :force, [true, false], default: false property :check_only, [true, false], default: false property :download_only, [true, false], default: false @@ -112,9 +113,9 @@ def run_flrtvc(m, apar, filesets, csv, path, verbose) shell_out!("/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{m} \"/usr/sbin/emgr -lv3\" > #{emgr_file}") end - # execute both compact and verbose flrtvc script - out_c = shell_out!("/usr/bin/flrtvc.ksh -l #{lslpp_file} -e #{emgr_file} #{apar_s} #{filesets_s} #{csv_s}", environment: { 'LANG' => 'C' }).stdout - out_v = shell_out!("/usr/bin/flrtvc.ksh -l #{lslpp_file} -e #{emgr_file} #{apar_s} #{filesets_s} #{csv_s} -v", environment: { 'LANG' => 'C' }).stdout + # execute flrtvc script (verbose if needed) + out_c = shell_out!("/usr/bin/flrtvc.ksh -l #{lslpp_file} -e #{emgr_file} #{apar_s} #{filesets_s} #{csv_s}", environment: { 'LANG' => 'C', 'LC_ALL' => 'C' }).stdout + out_v = shell_out!("/usr/bin/flrtvc.ksh -l #{lslpp_file} -e #{emgr_file} #{apar_s} #{filesets_s} #{csv_s} -v", environment: { 'LANG' => 'C', 'LC_ALL' => 'C' }).stdout if verbose # write report file unless path.nil? @@ -317,6 +318,7 @@ def download(src, dst) download(src, dst) rescue StandardError => e Chef::Log.warn("Propagating exception of type '#{e.class}' when downloading!") + ::File.delete(dst) raise e end @@ -327,10 +329,12 @@ def untar(src, dest) increase_filesystem(dest) untar(src, dest) else + ::File.delete(src) Chef::Log.warn("Propagating exception of type '#{e.class}' when untarring!") raise e end rescue StandardError => e + ::File.delete(src) Chef::Log.warn("Propagating exception of type '#{e.class}' when untarring!") raise e end @@ -341,7 +345,7 @@ def check_level_prereq?(machine, src) so.lines[3..-2].each do |line| Chef::Log.debug(line.to_s) next if line.start_with?('#') # skip comments - next unless line =~ /^(.*?)\s+(\d+\.\d+\.\d+\.\d+)\s+(\d+\.\d+\.\d+\.\d+)$/ + next unless line =~ /^(.*?)\s+(\d+\.\d+\.\d+\.\d+)\s+(\d+\.\d+\.\d+\.\d+)(.*?)$/ lslpp_file = ::File.join(Chef::Config[:file_cache_path], "lslpp_#{machine}.txt") ref = shell_out!("/bin/cat #{lslpp_file} | /bin/grep -w #{Regexp.last_match(1)} | /bin/cut -d: -f3", environment: { 'LANG' => 'C' }).stdout @@ -419,13 +423,15 @@ def check_level_prereq?(machine, src) Chef::Log.debug("filesets=#{new_resource.filesets}") Chef::Log.debug("csv=#{new_resource.csv}") Chef::Log.debug("path=#{new_resource.path}") + Chef::Log.debug("force=#{new_resource.force}") check_flrtvc puts '' # create directory based on date/time - base_dir = ::File.join(Chef::Config[:file_cache_path], Time.now.to_s.gsub(/[:\s-]/, '_')) + # base_dir = ::File.join(Chef::Config[:file_cache_path], Time.now.to_s.gsub(/[:\s-]/, '_')) + base_dir = path ::FileUtils.mkdir_p(base_dir) # build list of targets @@ -434,6 +440,17 @@ def check_level_prereq?(machine, src) target_list = expand_targets(new_resource.targets, so) Chef::Log.debug("target_list: #{target_list}") + # force interim fixes automatic removal + if property_is_set?(:force) && new_resource.force == true + target_list.each do |m| + fixes = list_fixes(m) + fixes.each do |fix| + remove_fix(m, fix) + Chef::Log.warn("Interim fix #{fix} has been automatically removed") + end + end + end + # loop on clients target_list.each do |m| # run flrtvc @@ -476,6 +493,8 @@ def check_level_prereq?(machine, src) # copy efix efixes_basenames = [] + sort_efixes_basenames = [] + efixes.each do |efix| # build the efix basenames array basename = efix['Filename'].split('/')[-1] @@ -497,7 +516,7 @@ def check_level_prereq?(machine, src) converge_by("geninstall: install all efixes from '#{lpp_source_base_dir}'") do puts "\nStart patching nim master or local machine." geninstall_s = "/usr/sbin/geninstall -d #{lpp_source_base_dir} #{efixes_basenames.join(' ')}" - exit_status = Open3.popen3({ 'LANG' => 'C' }, geninstall_s) do |_stdin, stdout, stderr, wait_thr| + exit_status = Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, geninstall_s) do |_stdin, stdout, stderr, wait_thr| stdout.each_line do |line| line.chomp! print "\033[2K\r#{line}" if line =~ /^Processing Efix Package [0-9]+ of [0-9]+.$/ @@ -536,7 +555,79 @@ def check_level_prereq?(machine, src) nim = Nim.new nim.define_lpp_source(lpp_source, lpp_source_base_dir) unless nim.exist?(lpp_source) begin - nim.perform_efix_customization(lpp_source, m, efixes_basenames.join(' ')) + sort_efixes_basenames = nim.efix_sort_by_packaging_date(lpp_source_dir, efixes_basenames) + locked_fil = [] + list_files_loc = {} + # get locked package from client + begin + locked_fil = nim.get_locked_files(m) + rescue CmdError => e + Chef::Log.warn("get_locked_files Error for client #{target}:#{e}") + end + Chef::Log.debug("Locked package list for client [#{m}]: #{locked_fil}") + # get package name from efix list + list_files_loc = nim.get_efix_files_loc(lpp_source_dir, sort_efixes_basenames) + list_files_loc_copy = list_files_loc.dup + Chef::Log.debug("Package list name: #{list_files_loc}") + # remove efix from list if package is locked + locked_fil.each do |item| + list_files_loc.delete_if { |_, v| v.include?(item) } + end + # remove efix with package name doublon + unlock_efixes_basenames = {} + list_files_loc.each do |k, v| + unlock_efixes_basenames[k] = v + del_key = [] + list_files_loc.delete(k) + v.each do |item| + list_files_loc.each do |ky, va| + del_key << ky if va.include?(item) + end + list_files_loc.delete_if { |kk, _| del_key.include?(kk) } + end + end + # next if efix list to apply is empty + if unlock_efixes_basenames.keys.empty? + msg1 = "\n[#{m}] Have #{urls.size} vulnerabilities but no installion will be done due to locked files" + msg2 = "[#{m}] Use force option to remove locked packages before update" + Chef::Log.warn(msg1) + Chef::Log.warn(msg2) + puts msg1 + puts msg2 + next + end + # check if conflict detected to log message + unlock_efixes_basenames.each do |key, _| + list_files_loc_copy.delete(key) + end + unless list_files_loc_copy.empty? + msg = "\n[#{m}] Some Efixes will not be installed due to a conflict on files:" + Chef::Log.warn(msg) + puts msg + unless locked_fil.empty? + puts "\t[#{m}]Conflicting files:" + puts "\t" + locked_fil.join("\n\t") + puts "\n\tUse force option to remove locked packages before update" + end + msg1 = "\t\tEFix\t=> Files " + msg2 = "\t\t------------------------" + Chef::Log.warn(msg1) + Chef::Log.warn(msg2) + puts msg1 + puts msg2 + list_files_loc_copy.each do |k, v| + msg1 = "\t\t#{format('%s', k)}" + Chef::Log.warn(msg1) + puts msg1 + v.each do |item| + msg2 = "\t\t\t\t=>#{format('%s', item)}" + Chef::Log.warn(msg2) + puts msg2 + end + end + end + # perform efix installation + nim.perform_efix_customization(lpp_source, m, unlock_efixes_basenames.keys.join(' ')) rescue NimCustError => e STDERR.puts e.message Chef::Log.warn("[#{m}] Failed installing some efixes. See /var/adm/ras/emgr.log on #{m} for details") @@ -548,4 +639,4 @@ def check_level_prereq?(machine, src) # clean temporary files ::FileUtils.remove_dir(base_dir) if new_resource.clean == true -end +end \ No newline at end of file diff --git a/resources/inittab.rb b/resources/inittab.rb index 4ba763c..e938bbb 100644 --- a/resources/inittab.rb +++ b/resources/inittab.rb @@ -26,10 +26,10 @@ fields = so.stdout.lines.first.chomp.split(':') # perfstat:2:once:/usr/lib/perf/libperfstat_updt_dictionary >/dev/console 2>&1 - identifier(fields[0]) - runlevel(fields[1]) - processaction(fields[2]) - command(fields[3]) + identifier fields[0] + runlevel fields[1] + processaction fields[2] + command fields[3] end action :install do diff --git a/resources/nimviosupdate.rb b/resources/nimviosupdate.rb index 8095b72..170960e 100644 --- a/resources/nimviosupdate.rb +++ b/resources/nimviosupdate.rb @@ -96,7 +96,7 @@ def check_lpp_source(lpp_source) # find location of lpp_source cmd_s = "/usr/sbin/lsnim -a location #{lpp_source}" log_info("check_lpp_source: '#{cmd_s}'") - exit_status = Open3.popen3({ 'LANG' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + exit_status = Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| stdout.each_line do |line| log_info("[STDOUT] #{line.chomp}") location = Regexp.last_match(1) if line =~ /.*location\s+=\s+(\S+)\s*/ @@ -136,7 +136,7 @@ def vios_health_init(nim_vios, hmc_id, hmc_ip) cmd_s = "/usr/sbin/vioshc.py -i #{hmc_ip} -l a" log_info("Health Check: init command '#{cmd_s}'") - Open3.popen3({ 'LANG' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| stderr.each_line do |line| # nothing is print on stderr so far but log anyway STDERR.puts line @@ -176,7 +176,7 @@ def vios_health_init(nim_vios, hmc_id, hmc_ip) put_warn("Health Check: unexpected script output: consecutive Managed System UUID: '#{line.strip}'") end cec_uuid = Regexp.last_match(1) - cec_serial = Regexp.last_match(2).tr('*', '_') + cec_serial = Regexp.last_match(2) log_info("Health Check: init found managed system: cec_uuid:'#{cec_uuid}', cec_serial:'#{cec_serial}'") next @@ -193,7 +193,7 @@ def vios_health_init(nim_vios, hmc_id, hmc_ip) end # new vios partition but skip if lparid is not found. - next if line =~ /^\s+(\S+)\s+Not found$/ + next if line =~ /^\s+(\S+)\s+none$/ # regular new vios partition if line =~ /^\s+(\S+)\s+(\S+)$/ @@ -247,7 +247,7 @@ def vios_health_check(nim_vios, hmc_ip, vios_list) end log_info("Health Check: init command '#{cmd_s}'") - Open3.popen3({ 'LANG' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| stderr.each_line do |line| STDERR.puts line log_info("[STDERR] #{line.chomp}") @@ -350,7 +350,7 @@ def nim_updateios(vios, cmd_s) wait_thr.value # Process::Status object returned. end put_info("Finish updating vios '#{vios}'.") - + put_info('With commit operation.') if cmd_s.include?('updateios_flags=-commit') raise ViosUpdateError, "Failed to perform NIM updateios operation on '#{vios}', see above error!" unless exit_status.success? end @@ -379,7 +379,7 @@ def get_vios_ssp_status(nim_vios, vios_list, vios_key, targets_status) cmd_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{nim_vios[vios]['vios_ip']} \"/usr/ios/cli/ioscli cluster -status -fmt :\"" log_debug("ssp_status: '#{cmd_s}'") - Open3.popen3({ 'LANG' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| stderr.each_line do |line| STDERR.puts line log_info("[STDERR] #{line.chomp}") @@ -566,11 +566,20 @@ def ssp_stop_start(vios_list, vios, nim_vios, action) log_info('Get NIM info for HMC') nim_hmc = nim.get_hmc_info() + # get CEC list + log_info('Get NIM info for Cecs') + nim_cec = nim.get_cecs_info() + # get the vios info log_info('Get NIM info for VIOSes') nim_vios = nim.get_nim_clients_info('vios') vio_server = VioServer.new + # Complete the Cec serial in nim_vios dict + nim_vios.keys.each do |key| + nim_vios[key]['mgmt_cec_serial'] = nim_cec[nim_vios[key]['mgmt_cec']]['serial'] if nim_cec.keys.include?(nim_vios[key]['mgmt_cec']) + end + # build array of vios log_info("List of VIOS known in NIM: #{nim_vios.keys}") @@ -679,8 +688,15 @@ def ssp_stop_start(vios_list, vios, nim_vios, action) # first find the right hdisk and check if we can perform the copy ret = 0 + rootvg_info = {} begin - ret = vio_server.find_valid_altdisk(nim_vios, vios_list, vios_key, targets_status, altdisk_hash, new_resource.disk_size_policy) + + # check if the rootvg is mirrored + vios_list.each do |vios| + rootvg_info[vios] = vio_server.check_rootvg(nim_vios, vios) + end + + ret = vio_server.find_valid_altdisk(nim_vios, vios_list, vios_key, rootvg_info, targets_status, altdisk_hash, new_resource.disk_size_policy) next if ret == 1 rescue AltDiskFindError => e put_error(e.message) @@ -690,18 +706,36 @@ def ssp_stop_start(vios_list, vios, nim_vios, action) # actually perform the alternate disk copy vios_list.each do |vios| + error_label = if vios == vios1 + 'FAILURE-ALTDCOPY1' + else + 'FAILURE-ALTDCOPY2' + end converge_by("nim: perform alt_disk_install for vios '#{vios}' on disk '#{altdisk_hash[vios]}'\n") do begin + # unmirror the vg if necessary + # check mirror + nb_copies = rootvg_info[vios]['copy_dict'].keys.length + put_info("rootvg_info = '#{rootvg_info}'\n") + if nb_copies > 1 + begin + nim.perform_unmirror(nim_vios, vios, 'rootvg') + rescue UnMirrorError => e + # ADD status + STDERR.puts e.message + log_warn("[#{vios}] #{e.message}") + targets_status[vios_key] = error_label + put_info("Finish NIM alt_disk_install operation using disk '#{altdisk_hash[vios]}' on vios '#{vios}': #{targets_status[vios_key]}.") + break + end + end + put_info("Start NIM alt_disk_install operation using disk '#{altdisk_hash[vios]}' on vios '#{vios}'.") nim.perform_altdisk_install(vios, 'rootvg', altdisk_hash[vios]) rescue NimAltDiskInstallError => e msg = "Failed to start the alternate disk copy on #{altdisk_hash[vios]} of #{vios}: #{e.message}" put_error(msg) - targets_status[vios_key] = if vios == vios1 - 'FAILURE-ALTDCOPY1' - else - 'FAILURE-ALTDCOPY2' - end + targets_status[vios_key] = error_label put_info("Finish NIM alt_disk_install operation using disk '#{altdisk_hash[vios]}' on vios '#{vios}': #{targets_status[vios_key]}.") break end @@ -730,11 +764,23 @@ def ssp_stop_start(vios_list, vios, nim_vios, action) end ret = 1 - targets_status[vios_key] = if vios == vios1 - 'FAILURE-ALTDCOPY1' - else - 'FAILURE-ALTDCOPY2' - end + targets_status[vios_key] = error_label + end + + # mirror the vg if necessary + nb_copies = rootvg_info[vios]['copy_dict'].keys.length + if nb_copies > 1 + log_debug('mirror') + begin + nim.perform_mirror(nim_vios, vios, 'rootvg', rootvg_info) + rescue MirrorError => e + # ADD status + STDERR.puts e.message + log_warn("[#{vios}] #{e.message}") + targets_status[vios_key] = error_label + put_info("Finish NIM alt_disk_install operation using disk '#{altdisk_hash[vios]}' on vios '#{vios}': #{targets_status[vios_key]}.") + break + end end put_info("Finish NIM alt_disk_install operation for disk '#{altdisk_hash[vios]}' on vios '#{vios}': #{targets_status[vios_key]}.") break unless ret == 0 @@ -743,7 +789,6 @@ def ssp_stop_start(vios_list, vios, nim_vios, action) else put_warn("Alternate disk copy for #{vios_key} skipped: time limit '#{new_resource.time_limit}' reached") end - log_info("Alternate disk copy status for #{vios_key}: #{targets_status[vios_key]}") end # altdisk_copy @@ -823,6 +868,16 @@ def ssp_stop_start(vios_list, vios, nim_vios, action) cmd_to_run = cmd + vios converge_by("nim: perform NIM updateios for vios '#{vios}'\n") do begin + # Commit applied lpps if necessay + if new_resource.preview.eql?('no') + put_info("Start Autocommit before NIM updateios for vios '#{vios}'.") + cmd_commit = "/usr/sbin/nim -o updateios -a updateios_flags=-commit -a filesets=all '#{vios}'" + begin + nim_updateios(vios, cmd_commit) + rescue ViosUpdateError => e + put_error(e.message) + end + end put_info("Start NIM updateios for vios '#{vios}'.") nim_updateios(vios, cmd_to_run) rescue ViosUpdateError => e diff --git a/resources/viosupgrade.rb b/resources/viosupgrade.rb new file mode 100644 index 0000000..3931316 --- /dev/null +++ b/resources/viosupgrade.rb @@ -0,0 +1,1148 @@ +# +# Copyright 2017, International Business Machines Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# TBC - uniformize use of log_xxx instead of Chef::Log.xxx in previous code +# TBC - uniformize use of put_xxx (pach_mgmt.rb) in previous code + +include AIX::PatchMgmt + +############################## +# PROPERTIES +############################## +property :desc, String, name_property: true +property :targets, String, required: true +property :ios_mksysb_name, String, required: true +property :viosupgrade_type, String, equal_to: %w(altdisk bosinst) +property :altdisks, String +property :installdisks, String +property :resources, String +property :common_resources, String +property :action_list, String, default: 'validate,upgrade' # no altdisk_cleanup by default +property :time_limit, String # mm/dd/YY HH:MM +property :disk_size_policy, String, default: 'nearest', equal_to: %w(minimize upper lower nearest) +property :preview, default: 'no', equal_to: %w(yes no) +property :viosupgrade_alt_disk_copy, default: 'no', equal_to: %w(yes no) +default_action :upgrade + +############################## +# load_current_value +############################## +load_current_value do +end + +############################## +# DEFINITIONS +############################## + +class ViosHealthCheckError < StandardError +end + +class ViosUpgradeConfFileError < StandardError +end + +class ViosUpgradeError < StandardError +end + +class ViosValUpgradeError < StandardError +end + +class ViosUpgradeBadProperty < StandardError +end + +class ViosResourceBadResource < StandardError +end + +class ViosResourceBadLocation < StandardError +end + +class ViosNoClusterFound < StandardError +end + +# ----------------------------------------------------------------- +# Check the vioshc script can be used +# +# return 0 if success +# +# raise ViosHealthCheckError in case of error +# ----------------------------------------------------------------- +def check_vioshc + vioshc_file = '/usr/sbin/vioshc.py' + + unless ::File.exist?(vioshc_file) + msg = "Error: Health check script file '#{vioshc_file}': not found" + raise ViosHealthCheckError, msg + end + + unless ::File.executable?(vioshc_file) + msg = "Error: Health check script file '#{vioshc_file}' not executable" + raise ViosHealthCheckError, msg + end + + 0 +end + +# ----------------------------------------------------------------- +# Check if viosupgrade file file can be used +# +# return 0 if success +# +# raise ViosUpgradeConfFileError in case of error +# ----------------------------------------------------------------- +def check_viosupgrade_file(path_file) + unless ::File.exist?(path_file) + msg = "Error: viosupgrade config file '#{path_file}': not found" + raise ViosUpgradeConfFileError, msg + end + 0 +end + +# ----------------------------------------------------------------- +# Check the specified resource location exists +# +# return true if success +# +# raise ViosUpgradeBadProperty in case of error +# raise ViosResourceBadLocation in case of error +# ----------------------------------------------------------------- +def check_resource_location(resource) + location = '' + ret = true + + # find location of resource + cmd_s = "/usr/sbin/lsnim -a location #{resource}" + log_info("check_source_location: '#{cmd_s}'") + exit_status = Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stdout.each_line do |line| + log_info("[STDOUT] #{line.chomp}") + location = Regexp.last_match(1) if line =~ /.*location\s+=\s+(\S+)\s*/ + end + stderr.each_line do |line| + STDERR.puts line + log_info("[STDERR] #{line.chomp}") + end + wait_thr.value # Process::Status object returned. + end + raise ViosUpgradeBadProperty, "Cannot find location of resource='#{resource}': Command '#{cmd_s}' returns above error." unless exit_status.success? + + # check to make sure path exists + raise ViosResourceBadLocation, "Cannot find location='#{location}' of resource='#{resource}'" unless ::File.exist?(location) + ret +end + +# ----------------------------------------------------------------- +# get the spot from resource +# +# return spot name +# +# raise ViosUpgradeBadProperty in case of error +# raise ViosResourceBadResource in case of error +# ----------------------------------------------------------------- +def get_spot_from_mksysb(resource) + spot = '' + + # find spot of resource + cmd_s = "/usr/sbin/lsnim -a extracted_spot #{resource}" + log_info("get_spot_from_mksysb: '#{cmd_s}'") + exit_status = Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stdout.each_line do |line| + log_info("[STDOUT] #{line.chomp}") + spot = Regexp.last_match(1) if line =~ /.*extracted_spot\s+=\s+(\S+)\s*/ + end + stderr.each_line do |line| + STDERR.puts line + log_info("[STDERR] #{line.chomp}") + end + wait_thr.value # Process::Status object returned. + end + raise ViosUpgradeBadProperty, "Cannot find extracted_spot of resource='#{resource}': Command '#{cmd_s}' returns above error." unless exit_status.success? + + # check to make sure spot exists + raise ViosResourceBadResource, "Cannot find extracted_spot='#{spot}' of resource='#{resource}'" if shell_out("lsnim -l #{spot}").error? + spot +end + +# ----------------------------------------------------------------- +# Collect VIOS and Managed System UUIDs. +# +# This first call to the vioshc.py script intend to collect +# UUIDs. The actual health assessment is performed in a second +# call. +# +# Return 0 if success +# +# raise ViosHealthCheckError in case of error +# ----------------------------------------------------------------- +def vios_health_init(nim_vios, hmc_id, hmc_ip) + log_debug("vios_health_init: hmc_id='#{hmc_id}', hmc_ip='#{hmc_ip}'") + ret = 0 + + # first call to collect UUIDs + cmd_s = "/usr/sbin/vioshc.py -i #{hmc_ip} -l a" + log_info("Health Check: init command '#{cmd_s}'") + + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stderr.each_line do |line| + # nothing is print on stderr so far but log anyway + STDERR.puts line + log_info("[STDERR] #{line.chomp}") + end + unless wait_thr.value.success? + stdout.each_line { |line| log_info("[STDOUT] #{line.chomp}") } + raise ViosHealthCheckError, "Heath check init command \"#{cmd_s}\" returns above error!" + end + + data_start = 0 + vios_section = 0 + cec_uuid = '' + cec_serial = '' + + # Parse the output and store the UUIDs + stdout.each_line do |line| + log_info("[STDOUT] #{line.chomp}") + if line.include?('ERROR') || line.include?('WARN') + # Needed this because vioshc.py script does not prints error to stderr + put_warn("Heath check (vioshc.py) script: '#{line.strip}'") + next + end + line.rstrip! + + if vios_section == 0 + # skip the header + if line =~ /^-+\s+-+$/ + data_start = 1 + next + end + next if data_start == 0 + + # New managed system section + if line =~ /^(\S+)\s+(\S+)\s*$/ + unless cec_uuid == '' && cec_serial == '' + put_warn("Health Check: unexpected script output: consecutive Managed System UUID: '#{line.strip}'") + end + cec_uuid = Regexp.last_match(1) + cec_serial = Regexp.last_match(2) + + log_info("Health Check: init found managed system: cec_uuid:'#{cec_uuid}', cec_serial:'#{cec_serial}'") + next + end + + # New vios section + if line =~ /^\s+-+\s+-+$/ + vios_section = 1 + next + end + + # skip all header and empty lines until the vios section + next + end + + # new vios partition but skip if lparid is not found. + next if line =~ /^\s+(\S+)\s+none$/ + + # regular new vios partition + if line =~ /^\s+(\S+)\s+(\S+)$/ + vios_uuid = Regexp.last_match(1) + vios_part_id = Regexp.last_match(2) + + # retrieve the vios with the vios_part_id and the cec_serial value + # and store the UUIDs in the dictionaries + nim_vios.keys.each do |vios_key| + next unless nim_vios[vios_key]['mgmt_vios_id'] == vios_part_id && nim_vios[vios_key]['mgmt_cec_serial'] == cec_serial + + nim_vios[vios_key]['vios_uuid'] = vios_uuid + nim_vios[vios_key]['cec_uuid'] = cec_uuid + log_info("Health Check: init found matching vios #{vios_key}: vios_part_id='#{vios_part_id}' vios_uuid='#{vios_uuid}'") + break + end + next + end + + # skip empty line after vios section. stop the vios section + if line =~ /^\s*$/ + vios_section = 0 + cec_uuid = '' + cec_serial = '' + next + end + + raise ViosHealthCheckError, "Health Check: init failed, bad script output for the #{hmc_id} hmc: '#{line}'" + end + end + ret +end + +# ----------------------------------------------------------------- +# Health assessment of the VIOSes targets to ensure they can support +# a rolling update operation. +# +# This operation uses the vioshc.py script to evaluate the capacity +# of the pair of the VIOSes to support the rolling update operation: +# +# return: 0 if ok, 1 otherwise +# ----------------------------------------------------------------- +def vios_health_check(nim_vios, hmc_ip, vios_list) + log_debug("vios_health_check: hmc_ip: #{hmc_ip} vios_list: #{vios_list}") + ret = 0 + rate = 0 + + cmd_s = "/usr/sbin/vioshc.py -i #{hmc_ip} -m #{nim_vios[vios_list[0]]['cec_uuid']} " + vios_list.each do |vios| + cmd_s << "-U #{nim_vios[vios]['vios_uuid']} " + end + log_info("Health Check: init command '#{cmd_s}'") + + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stderr.each_line do |line| + STDERR.puts line + log_info("[STDERR] #{line.chomp}") + end + ret = 1 unless wait_thr.value.success? + + # Parse the output to get the "Pass rate" + stdout.each_line do |line| + log_info("[STDOUT] #{line.chomp}") + + if line.include?('ERROR') || line.include?('WARN') + # Need because vioshc.py script does not prints error to stderr + put_warn("Heath check (vioshc.py) script: '#{line.strip}'") + end + next unless line =~ /Pass rate of/ + + rate = Regexp.last_match(1).to_i if line =~ /Pass rate of (\d+)%/ + + if ret == 0 && rate == 100 + put_info("VIOSes #{vios_list.join('-')} can be updated") + else + put_warn("VIOSes #{vios_list.join('-')} can NOT be updated: only #{rate}% of checks pass") + ret = 1 + end + break + end + end + ret +end + +# ----------------------------------------------------------------- +# Build the viosupgrade command to run +# +# return the command string to pass to run_viosupgrade() +# +# rubocop:disable Metrics/ParameterLists +# rubocop:disable Style/IfInsideElse +# ----------------------------------------------------------------- +def get_viosupgrade_cmd(nim_vios, vios, upgrade_type, ios_mksysb, installdisk, altdisk, resources, common_resources, preview, upg_altdisk) + cmd = '/usr/sbin/viosupgrade ' + + # type + if !upgrade_type.nil? && !upgrade_type.empty? + cmd << " -t #{upgrade_type}" + log_info("[CMD] #{cmd}") + end + + # mksysb and spot if necessary + if !ios_mksysb.nil? && !ios_mksysb.empty? && check_resource_location(ios_mksysb) + cmd << " -m #{ios_mksysb}" + # get spot from mksysb + if upgrade_type == 'bosinst' + spot = get_spot_from_mksysb(ios_mksysb) + cmd << " -p #{spot}" + log_info("[CMD] #{cmd}") + end + end + + # altdisk for bosinst + if !altdisk[vios].nil? && !altdisk[vios].empty? && upg_altdisk == 'yes' && upgrade_type == 'bosinst' + cmd << " -r #{altdisk[vios]}" + log_info("[CMD] #{cmd}") + end + + # altdisk + if !installdisk[vios].nil? && !installdisk[vios].empty? + cmd << " -a #{installdisk[vios]}" if upgrade_type == 'altdisk' + log_info("[CMD] #{cmd}") + end + + # resources + if !resources[vios].nil? && !resources[vios].empty? + cmd << " -e #{resources[vios]}" + cmd << ":#{common_resources}" if !common_resources.nil? && !common_resources.empty? + else + cmd << " -e #{common_resources}" if !common_resources.nil? && !common_resources.empty? + end + log_info("[CMD] #{cmd}") + + # cluster + cmd << ' -c' if nim_vios[vios]['ssp_vios_status'] == 'OK' + + # validation preview mode + cmd << ' -v' if !preview.nil? && !preview.empty? && preview == 'yes' + + # skip clone from viosupgrade command + cmd << ' -s' if upgrade_type == 'bosinst' && upg_altdisk == 'no' + + # add vios target + cmd << " -n #{vios}" + log_debug("get_viosupgrade_cmd - return cmd: '#{cmd}'") + cmd +end + +# ----------------------------------------------------------------- +# Run vuisupgrade operation on specified vios +# The command to run is built by get_viosupgrade_cmd() +# +# raise ViosUpgradeError in case of error +# ----------------------------------------------------------------- +def run_viosupgrade(vios, cmd_s) + put_info("Start upgrading vios '#{vios}' with viosupgrade.") + log_info("run_viosupgrade: '#{cmd_s}'") + if cmd_s.include?(' -v') + put_info('validate operation.') + else + put_info("Starting viosupgrade operation for vios '#{vios}'.") + end + exit_status = Open3.popen3({ 'LANG' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stdout.each_line { |line| log_info("[STDOUT] #{line.chomp}") } + stderr.each_line do |line| + STDERR.puts line + log_info("[STDERR] #{line.chomp}") + end + wait_thr.value # Process::Status object returned. + end + raise ViosUpgradeError, "Failed to perform viosupgrade operation on '#{vios}', see log file!" unless exit_status.success? +end + +# ----------------------------------------------------------------- +# get the spot from resource +# +# return ssp id +# +# raise ViosNoClusterFound if no cluster found +# ----------------------------------------------------------------- +def get_ssp_name_id(nim_vios, vios) + ssp_id = '' + cmd_c = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{nim_vios[vios]['vios_ip']} \"/etc/lsattr -El vioscluster0 \"" + log_info("get_ssp_name_id: '#{cmd_c}'") + exit_status = Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_c) do |_stdin, stdout, stderr, wait_thr| + stdout.each_line do |line| + log_info("[STDOUT] #{line.chomp}") + line.strip! + ssp_id = Regexp.last_match(1) if line =~ /^cluster_id\s+(\S+).*/ + end + stderr.each_line do |line| + STDERR.puts line + log_info("[STDERR] #{line.chomp}") + end + wait_thr.value # Process::Status object returned. + end + raise ViosCmdError, "#{msg}, command: \"#{cmd_s}\" returns above error!" unless exit_status.success? + ssp_id +end + +# ----------------------------------------------------------------- +# Check the SSP status of the VIOS tuple +# viosupgrade copy can be done when both VIOSes in the tuple +# refer to the same cluster and have the same SSP status = UP +# or if the vios is down +# +# return 0 if OK +# 1 else +# rubocop:disable Style/GuardClause +# ----------------------------------------------------------------- +def get_vios_ssp_status_for_upgrade(nim_vios, vios_list, vios_key, targets_status) + ssp_name = '' + vios_ssp_status = '' + vios_name = '' + err_label = 'FAILURE-SSP' + + vios_list.each do |vios| + nim_vios[vios]['ssp_vios_status'] = 'none' + nim_vios[vios]['ssp_name'] = 'none' + nim_vios[vios]['ssp_id'] = 'none' + end + + # get the SSP status + vios_list.each do |vios| + # vios = vios_list[0] + + # check if cluster defined + nim_vios[vios]['ssp_id'] = get_ssp_name_id(nim_vios, vios) + # cluster found + if nim_vios[vios]['ssp_id'] != '' + log_info("[VIOS CLUSTER ID] #{nim_vios[vios]['ssp_id']}") + else + msg = "No cluster found => continue to upgrade" + log_info(msg) + return 0 # no cluster found => continue to upgrade + end + + # command for cluster status + cmd_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{nim_vios[vios]['vios_ip']} \"/usr/ios/cli/ioscli cluster -status -fmt :\"" + log_debug("ssp_status: '#{cmd_s}'") + Open3.popen3({ 'LANG' => 'C', 'LC_ALL' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stderr.each_line do |line| + STDERR.puts line + log_info("[STDERR] #{line.chomp}") + end + unless wait_thr.value.success? + stdout.each_line { |line| log_info("[STDOUT] #{line.chomp}") } + msg = "Failed to get SSP status of #{vios_key}" + log_warn("[#{vios}] #{msg}") + raise ViosCmdError, "Error: #{msg} on #{vios}, command \"#{cmd_s}\" returns above error!" + end + + # check that the VIOSes belong to the same cluster and have the same satus + # or there is no SSP + # stdout is like: + # gdr_ssp3:OK:castor_gdr_vios3:8284-22A0221FD4BV:17:OK:OK + # gdr_ssp3:OK:castor_gdr_vios2:8284-22A0221FD4BV:16:OK:OK + # or + # Cluster does not exist. + # + stdout.each_line do |line| + log_debug("[STDOUT] #{line.chomp}") + line.chomp! + if line =~ /^Cluster does not exist.$/ + log_debug("There is no cluster or the node #{vios} is DOWN") + nim_vios[vios]['ssp_vios_status'] = 'DOWN' + return 1 if vios_list.length == 1 + break + end + + next unless line =~ /^(\S+):\S+:(\S+):\S+:\S+:(\S+):.*/ + cur_ssp_name = Regexp.last_match(1) + cur_vios_name = Regexp.last_match(2) + cur_vios_ssp_status = Regexp.last_match(3) + + next unless vios_list.include?(cur_vios_name) + nim_vios[cur_vios_name]['ssp_vios_status'] = cur_vios_ssp_status + nim_vios[cur_vios_name]['ssp_name'] = cur_ssp_name + # single VIOS case + if vios_list.length == 1 + # When single vios into the list => upgrade + put_info('Single VIOS in the list is included into a cluster - stop upgrade') + return 1 + end + # first VIOS in the pair + if ssp_name == '' + ssp_name = cur_ssp_name + vios_name = cur_vios_name + vios_ssp_status = cur_vios_ssp_status + next + end + + # both VIOSes found + if cur_vios_ssp_status == 'OK' && vios_ssp_status == cur_vios_ssp_status + return 0 + elsif ssp_name != cur_ssp_name && cur_vios_ssp_status == 'OK' + err_msg = "Both VIOSes: #{vios_key} does not belong to the same SSP. VIOSes cannot be updated" + put_error(err_msg) + targets_status[vios_key] = err_label + return 1 + else + return 1 + end + end + end + end + + err_msg = 'SSP status undefined' + put_error(err_msg) + targets_status[vios_key] = err_label + 1 +end + +# ----------------------------------------------------------------- +# Stop/start the SSP for a VIOS +# +# ret = 0 if OK +# 1 else +# ----------------------------------------------------------------- +def ssp_stop_start(vios_list, vios, nim_vios, action) + # if action is start SSP, find the first node running SSP + node = vios + if action == 'start' + vios_list.each do |n| + if nim_vios[n]['ssp_vios_status'] == 'OK' + node = n + break + end + end + end + cmd_s = "/usr/lpp/bos.sysmgt/nim/methods/c_rsh #{nim_vios[node]['vios_ip']} \"/usr/sbin/clctrl -#{action} -n #{nim_vios[vios]['ssp_name']} -m #{vios}\"" + + log_debug("ssp_stop_start: '#{cmd_s}'") + Open3.popen3({ 'LANG' => 'C' }, cmd_s) do |_stdin, stdout, stderr, wait_thr| + stderr.each_line do |line| + STDERR.puts line + log_info("[STDERR] #{line.chomp}") + end + unless wait_thr.value.success? + stdout.each_line { |line| log_info("[STDOUT] #{line.chomp}") } + msg = "Failed to #{action} cluster #{nim_vios[vios]['ssp_name']} on vios #{vios}" + log_warn(msg) + raise ViosCmdError, "#{msg}, command: \"#{cmd_s}\" returns above error!" + end + end + + nim_vios[vios]['ssp_vios_status'] = if action == 'stop' + 'DOWN' + else + 'OK' + end + log_info("#{action} cluster #{nim_vios[vios]['ssp_name']} on vios #{vios} succeed") + + 0 +end + +############################## +# ACTION: upgrade +############################## +action :upgrade do + # inputs + log_info("VIOS UPGRADE - desc=\"#{new_resource.desc}\"") + log_info("VIOS UPGRADE - action_list=\"#{new_resource.action_list}\"") + log_info("VIOS UPGRADE - targets=#{new_resource.targets}") + log_info("VIOS UPGRADE - targets=#{new_resource.viosupgrade_type}") + STDOUT.puts '' + STDERR.puts '' # TBC - needed for message presentation + + # check the action_list property + allowed_action = %w(check altdisk_copy validate upgrade altdisk_cleanup) + new_resource.action_list.delete(' ').split(',').each do |my_action| + unless allowed_action.include?(my_action) + raise ViosUpgradeBadProperty, "Invalid action '#{my_action}' in action_list '#{new_resource.action_list}', must be in: #{allowed_action.join(',')}" + end + end + + # check mandatory properties for the action_list + if new_resource.action_list.include?('altdisk_copy') && (new_resource.altdisks.nil? || new_resource.altdisks.empty?) + raise ViosUpgradeBadProperty, "Please specify an 'altdisks' property for altdisk_copy operation" + end + + # check mandatory properties for the action_list + if new_resource.viosupgrade_type == 'bosinst' + if (new_resource.viosupgrade_alt_disk_copy == 'yes') && (new_resource.altdisks.nil? || new_resource.altdisks.empty?) + raise ViosUpgradeBadProperty, "Please specify an 'altdisks' property for altdisk_copy operation " + end + end + + if new_resource.action_list.include?('upgrade') + raise ViosUpgradeBadProperty, 'ios_mksysb_name is required for the upgrade operation' if new_resource.ios_mksysb_name.nil? || new_resource.ios_mksysb_name.empty? + end + + # build time object from time_limit attribute, + end_time = nil + unless new_resource.time_limit.nil? + if new_resource.time_limit =~ %r/^(\d{2})\/(\d{2})\/(\d{2,4}) (\d{1,2}):(\d{1,2})$/ + end_time = Time.local(Regexp.last_match(3).to_i, + Regexp.last_match(2).to_i, + Regexp.last_match(1).to_i, + Regexp.last_match(4).to_i, + Regexp.last_match(5).to_i) + log_info("End time for operation: '#{end_time}'") + else + raise ViosUpgradeBadProperty, "Error: 'time_limit' property must be in the format: 'mm/dd/yy HH:MM', got:'#{new_resource.time_limit}'" + end + end + + log_info('Check NIM info is well configured') + nim = Nim.new + check_nim_info(node) + + # get hmc info + log_info('Get NIM info for HMC') + nim_hmc = nim.get_hmc_info() + + # get CEC list + log_info('Get NIM info for Cecs') + nim_cec = nim.get_cecs_info() + + # get the vios info + log_info('Get NIM info for VIOSes') + nim_vios = nim.get_nim_clients_info('vios') + vio_server = VioServer.new + + # Complete the Cec serial in nim_vios dict + nim_vios.keys.each do |key| + nim_vios[key]['mgmt_cec_serial'] = nim_cec[nim_vios[key]['mgmt_cec']]['serial'] if nim_cec.keys.include?(nim_vios[key]['mgmt_cec']) + end + + # build array of vios + log_info("List of VIOS known in NIM: #{nim_vios.keys}") + + # build list of targets + altdisk_hash = {} + target_list = expand_vios_pair_targets(new_resource.targets, nim_vios.keys, new_resource.altdisks, altdisk_hash) + + # build installdisks + installdisk_hash = build_installdisks(new_resource.targets, nim_vios.keys, new_resource.installdisks) + + # build resources + resource_hash = build_resources(new_resource.targets, nim_vios.keys, new_resource.resources) + + # check vioshc script is executable + check_vioshc if new_resource.action_list.include?('check') + + # main loop on target: can be 1-tuple or 2-tuple of VIOS + targets_status = {} + vios_key = '' + target_list.each do |target_tuple| + log_info("Working on target tuple: #{target_tuple}") + + vios_list = target_tuple.split(',') + tup_len = vios_list.length + vios1 = vios_list[0] + if tup_len == 2 + vios2 = vios_list[1] + vios_key = "#{vios1}-#{vios2}" + else + vios_key = vios1 + vios2 = nil + end + + ############### + # health_check + if new_resource.action_list.include?('check') + Chef::Log.info('VIOS UPGRADE - action=check') + put_info("Health Check for VIOS tuple: #{target_tuple}") + + # Credentials + log_info("Credentials (for VIOS: #{vios1})") + hmc_id = nim_vios[vios1]['mgmt_hmc_id'] + + unless nim_hmc.key?(hmc_id) + # this should not happen + put_error("Health Check, VIOS '#{vios1}' NIM management HMC ID '#{hmc_id}' not found") + targets_status[vios_key] = 'FAILURE-HC' + next # continue with next target tuple + end + hmc_ip = nim_hmc[hmc_id]['ip'] + + # if needed call vios_health_init to get the UUIDs value + if !nim_vios[vios1].key?('vios_uuid') || tup_len == 2 && !nim_vios[vios2].key?('vios_uuid') + begin + vios_health_init(nim_vios, hmc_id, hmc_ip) + rescue ViosHealthCheckError => e + targets_status[vios_key] = 'FAILURE-HC' + put_error(e.message) + end + # Error case is handle by the next if statement + end + + if tup_len == 1 && nim_vios[vios1].key?('vios_uuid') || + tup_len == 2 && nim_vios[vios1].key?('vios_uuid') && nim_vios[vios2].key?('vios_uuid') + + # run the vios_health check for the vios tuple + ret = vios_health_check(nim_vios, hmc_ip, vios_list) + + targets_status[vios_key] = if ret == 0 + 'SUCCESS-HC' + else + 'FAILURE-HC' + end + else + # vios uuid's not found + if !nim_vios[vios1].key?('vios_uuid') && !nim_vios[vios2].key?('vios_uuid') + vios_err = "#{vios1} and #{vios2}" + elsif !nim_vios[vios1].key?('vios_uuid') + vios_err = vios1 unless nim_vios[vios1].key?('vios_uuid') + else + vios_err = vios2 unless nim_vios[vios2].key?('vios_uuid') + end + targets_status[vios_key] = 'FAILURE-HC' + msg = "Health Check did not get the UUID of VIOS: #{vios_err}" + put_error(msg) + end + + log_info("Health Check status for #{vios_key}: #{targets_status[vios_key]}") + + next if targets_status[vios_key] == 'FAILURE-HC' # continue with next target tuple + + end # check + + ############### + # Alternate disk copy operation + + # check previous status and skip if failure + if new_resource.action_list.include?('altdisk_copy') && new_resource.viosupgrade_alt_disk_copy == 'no' + log_info('VIOS UPGRADE - action=altdisk_copy') + log_info("VIOS UPGRADE - altdisks=#{new_resource.altdisks}") + log_info("VIOS UPGRADE - disk_size_policy=#{new_resource.disk_size_policy}") + log_info("Alternate disk copy for VIOS tuple: #{target_tuple}") + + # if health check status is known, check the vios tuple has passed + if new_resource.action_list.include?('check') && targets_status[vios_key] != 'SUCCESS-HC' + put_warn("Alternate disk copy for #{vios_key} VIOSes skipped (previous status: #{targets_status[vios_key]})") + next + end + + # check if there is time to handle this tuple + if end_time.nil? || Time.now <= end_time + # first find the right hdisk and check if we can perform the copy + ret = 0 + + rootvg_info = {} + begin + + # check if the rootvg is mirrored + vios_list.each do |vios| + rootvg_info[vios] = vio_server.check_rootvg(nim_vios, vios) + end + + ret = vio_server.find_valid_altdisk(nim_vios, vios_list, vios_key, rootvg_info, targets_status, altdisk_hash, new_resource.disk_size_policy) + next if ret == 1 + rescue AltDiskFindError => e + put_error(e.message) + put_info("Finish NIM alt_disk_install operation for disk '#{altdisk_hash[vios_key]}' on vios '#{vios_key}': #{targets_status[vios_key]}.") + next + end + + # actually perform the alternate disk copy + vios_list.each do |vios| + error_label = if vios == vios1 + 'FAILURE-ALTDCOPY1' + else + 'FAILURE-ALTDCOPY2' + end + converge_by("\n nim: perform alt_disk_install for vios '#{vios}' on disk '#{altdisk_hash[vios]}'\n") do + begin + # unmirror the vg if necessary + # check mirror + nb_copies = rootvg_info[vios]['copy_dict'].keys.length + put_info("rootvg_info = '#{rootvg_info}'\n") + if nb_copies > 1 + begin + nim.perform_unmirror(nim_vios, vios, 'rootvg') + rescue UnMirrorError => e + # ADD status + STDERR.puts e.message + log_warn("[#{vios}] #{e.message}") + targets_status[vios_key] = error_label + put_info("Finish NIM alt_disk_install operation using disk '#{altdisk_hash[vios]}' on vios '#{vios}': #{targets_status[vios_key]}.") + break + end + end + + put_info("Start NIM alt_disk_install operation using disk '#{altdisk_hash[vios]}' on vios '#{vios}'.") + nim.perform_altdisk_install(vios, 'rootvg', altdisk_hash[vios]) + rescue NimAltDiskInstallError => e + msg = "Failed to start the alternate disk copy on #{altdisk_hash[vios]} of #{vios}: #{e.message}" + put_error(msg) + targets_status[vios_key] = error_label + put_info("Finish NIM alt_disk_install operation using disk '#{altdisk_hash[vios]}' on vios '#{vios}': #{targets_status[vios_key]}.") + break + end + + # wait the end of the alternate disk copy operation + begin + ret = nim.wait_alt_disk_install(vios) + rescue NimLparInfoError => e + STDERR.puts e.message + log_warn("[#{vios}] #{e.message}") + ret = 1 + end + if ret == 0 + targets_status[vios_key] = 'SUCCESS-ALTDC' + log_info("[#{vios}] VIOS altdisk copy succeeded on #{altdisk_hash[vios]}") + else + if ret == 1 + STDERR.puts e.message + msg = "Alternate disk copy failed on #{altdisk_hash[vios]} of vios #{vios}" + put_error(msg) + ret = 1 + else + msg = "Alternate disk copy failed on #{altdisk_hash[vios]}: timed out" + put_warn(msg) + STDERR.puts "#{msg} on vios #{vios}" + end + ret = 1 + + targets_status[vios_key] = error_label + end + + # mirror the vg if necessary + nb_copies = rootvg_info[vios]['copy_dict'].keys.length + if nb_copies > 1 + log_debug('mirror') + begin + nim.perform_mirror(nim_vios, vios, 'rootvg', rootvg_info) + rescue MirrorError => e + # ADD status + STDERR.puts e.message + log_warn("[#{vios}] #{e.message}") + targets_status[vios_key] = error_label + put_info("Finish NIM alt_disk_install operation using disk '#{altdisk_hash[vios]}' on vios '#{vios}': #{targets_status[vios_key]}.") + break + end + end + put_info("Finish NIM alt_disk_install operation for disk '#{altdisk_hash[vios]}' on vios '#{vios}': #{targets_status[vios_key]}.") + break unless ret == 0 + end + end + else + put_warn("Alternate disk copy for #{vios_key} skipped: time limit '#{new_resource.time_limit}' reached") + end + log_info("Alternate disk copy status for #{vios_key}: #{targets_status[vios_key]}") + end # altdisk_copy + + ############### + # validate + if new_resource.action_list.include?('validate') || new_resource.preview == 'yes' + Chef::Log.info('VIOS UPGRADE - action=validate') + put_info("viosupgrade for VIOS tuple: #{target_tuple}") + log_info("VIOS UPGRADE - type=#{new_resource.viosupgrade_type}") + log_info("VIOS UPGRADE - mksysb resource=#{new_resource.ios_mksysb_name}") + log_info("VIOS UPGRADE - preview=#{new_resource.preview}") + + # check SSP status of the tuple + # Upgrade can only be done when Current Vios is UP and Vios Dual is UP + # or if current vios is DOWN + ret = 0 + begin + ret = get_vios_ssp_status_for_upgrade(nim_vios, vios_list, vios_key, targets_status) + rescue ViosCmdError => e + put_error(e.message) + targets_status[vios_key] = 'FAILURE-VALIDATE' + log_info("Upgrade status for #{vios_key}: #{targets_status[vios_key]}") + next # cannot continue - switch to next tuple + end + if ret == 1 + put_warn("Upgrade operation for #{vios_key} vioses skipped due to bad SSP status") + put_info('Upgrade operation can be done if both of the VIOSes have the SSP status = UP') + targets_status[vios_key] = 'FAILURE-VALIDATE' + next # switch to next tuple + end + + vios_list.each do |vios| + targets_status[vios_key] = 'SUCCESS-VALIDATE' + begin + cmd_to_run = get_viosupgrade_cmd(nim_vios, vios, new_resource.viosupgrade_type, + new_resource.ios_mksysb_name, installdisk_hash, altdisk_hash, + resource_hash, new_resource.common_resources, 'yes', new_resource.viosupgrade_alt_disk_copy) + rescue ViosUpgradeBadProperty, ViosResourceBadLocation => e + put_error("Upgrade #{vios_key}: #{e.message}") + targets_status[vios_key] = 'FAILURE-VALIDATE' + log_info("Update status for #{vios_key}: #{targets_status[vios_key]}") + break # + end + + begin + put_info("Start viosupgrade - validate operation - for vios '#{vios}'.") + put_info("CMD= '#{cmd_to_run}'.") + run_viosupgrade(vios, cmd_to_run) + rescue ViosUpgradeError => e + put_error(e.message) + targets_status[vios_key] = 'FAILURE-VALIDATE' + put_info("Finish viosupgrade validation for vios '#{vios}': #{targets_status[vios_key]}.") + break # + end + end + put_info("Validate status for #{vios_key}: #{targets_status[vios_key]}") + next if targets_status[vios_key] == 'FAILURE-VALIDATE' # continue with next target tuple + end # validate + + ######## + # upgrade + if new_resource.action_list.include?('upgrade') && new_resource.preview == 'no' + log_info('VIOS UPGRADE - action=upgrade') + put_info("viosupgrade for VIOS tuple: #{target_tuple}") + put_info("VIOS UPGRADE - type=#{new_resource.viosupgrade_type}") + put_info("VIOS UPGRADE - mksysb resource=#{new_resource.ios_mksysb_name}") + + # check SSP status of the tuple + # Upgrade can only be done when current VIOS is UP and VIOS Dual is UP + # or if current vios is DOWN + ret = 0 + begin + ret = get_vios_ssp_status_for_upgrade(nim_vios, vios_list, vios_key, targets_status) + rescue ViosCmdError => e + put_error(e.message) + targets_status[vios_key] = 'FAILURE-VALIDATE' + log_info("Upgrade status for #{vios_key}: #{targets_status[vios_key]}") + next # cannot continue - switch to next tuple + end + if ret == 1 + put_warn("Upgrade operation for #{vios_key} vioses skipped due to bad SSP status") + put_info('Upgrade operation can be done if both of the VIOSes have the SSP status = UP') + targets_status[vios_key] = 'FAILURE-VALIDATE' + next # switch to next tuple + end + + if new_resource.action_list.include?('validate') && targets_status[vios_key] != 'SUCCESS-VALIDATE' + put_warn("Upgrade of #{vios_key} vioses skipped (previous status: #{targets_status[vios_key]})") + next + end + + # check if there is time to handle this tuple + if end_time.nil? || Time.now <= end_time + targets_status[vios_key] = 'SUCCESS-UPGRADE' + vios_list.each do |vios| + # check if altinst_rootvg exists else next tuple only for bosint + if new_resource.viosupgrade_type == 'bosinst' && new_resource.viosupgrade_alt_disk_copy == 'no' + ret = 0 + begin + thash = {} + ret = vio_server.get_altinst_rootvg_disk(nim_vios, vios, thash) + rescue AltDiskFindError => e + put_error(msg) + ret = 1 + end + if ret != 0 + targets_status[vios_key] = if vios == vios1 + 'FAILURE-CHECK_ALT_DISK_VIOS1' + else + 'FAILURE-CHECK_ALT_DISK_VIOS2' + end + put_warn("No No alternate disk found on '#{vios}'.") + break # switch to next tuple + end + end + + # get upgrade command + begin + cmd_to_run = get_viosupgrade_cmd(nim_vios, vios, + new_resource.viosupgrade_type, new_resource.ios_mksysb_name, + installdisk_hash, altdisk_hash, resource_hash, + new_resource.common_resources, 'no', new_resource.viosupgrade_alt_disk_copy) + rescue ViosUpgradeBadProperty, VioslppSourceBadLocation => e + put_error("Upgrade #{vios_key}: #{e.message}") + targets_status[vios_key] = 'FAILURE-UPGRAD1' + log_info("Upgrade status for #{vios_key}: #{targets_status[vios_key]}") + break # switch to next tuple + end + + break_required = false + # set the error label + err_label = if vios == vios1 + 'FAILURE-UPGRAD1' + else + 'FAILURE-UPGRAD2' + end + + converge_by("\n nim: perform NIM viosupgrade for vios '#{vios}'\n") do + begin + put_info("Start viosupgrade for vios '#{vios}'.") + put_info("CMD= '#{cmd_to_run}'.") + run_viosupgrade(vios, cmd_to_run) + rescue ViosUpgradeError => e + put_error(e.message) + targets_status[vios_key] = err_label + put_info("Finish viosupgrade for vios '#{vios}': #{targets_status[vios_key]}.") + # in case of failure + break_required = true + end + # wait the end of viosupgrade operation + begin + ret = nim.wait_viosupgrade(nim_vios, vios) + rescue ViosUpgradeQueryError => e + STDERR.puts e.message + log_warn("[#{vios}] #{e.message}") + ret = 1 + end + case ret + when 0 + targets_status[vios_key] = 'SUCCESS-UPGRADE' + put_info("[#{vios}] VIOS Upgrade succeeded") + when -1 + msg = "VIOSUPGRADE failed on #{vios}: timed out" + put_warn(msg) + STDERR.puts msg + targets_status[vios_key] = error_label + else + msg = "VIOSUPGRADE failed on #{vios}" + put_warn(msg) + STDERR.puts msg + targets_status[vios_key] = error_label + end + end # end converge_by + break if break_required + end + else + put_warn("Upgrade #{vios_key} skipped: time limit '#{new_resource.time_limit}' reached") + end + put_info("Upgrade status for vios '#{vios_key}': #{targets_status[vios_key]}.") + end # upgrade + + ############### + # Alternate disk cleanup operation + next unless new_resource.action_list.include?('altdisk_cleanup') + log_info('VIOS UPGRADE - action=altdisk_cleanup') + log_info("VIOS UPGRADE - altdisks=#{new_resource.altdisks}") + log_info("Alternate disk cleanup for VIOS tuple: #{target_tuple}") + + # check previous status and skip if failure + if new_resource.action_list.include?('upgrade') && targets_status[vios_key] != 'SUCCESS-UPGRADE' || + !new_resource.action_list.include?('upgrade') && new_resource.action_list.include?('altdisk_copy') && targets_status[vios_key] != 'SUCCESS-ALTDC' || + !new_resource.action_list.include?('upgrade') && !new_resource.action_list.include?('altdisk_copy') && new_resource.action_list.include?('check') && targets_status[vios_key] != 'SUCCESS-HC' + put_warn("Alternate disk cleanup for #{vios_key} VIOSes skipped (previous status: #{targets_status[vios_key]}") + next + end + + # find the altinst_rootvg disk + ret = 0 + vios_list.each do |vios| + log_info("Alternate disk cleanup, get the alternate rootvg disk for vios #{vios}") + begin + ret = vio_server.get_altinst_rootvg_disk(nim_vios, vios, altdisk_hash) + rescue AltDiskFindError => e + msg = "Cleanup failed: #{e.message}" + put_error(msg) + ret = 1 + targets_status[vios_key] = if vios == vios1 + 'FAILURE-ALTDCLEAN1' + else + 'FAILURE-ALTDCLEAN2' + end + end + put_warn("Failed to get the alternate disk on #{vios}") unless ret == 0 + end + + # perform the alternate disk cleanup + vios_list.select { |k| altdisk_hash[k] != '' }.each do |vios| + converge_by("vios: cleanup altinst_rootvg disk on vios '#{vios}'\n") do + targets_status[vios_key] = if vios == vios1 + 'FAILURE-ALTDCOPY1' + else + 'FAILURE-ALTDCOPY2' + end + begin + ret = vio_server.altdisk_copy_cleanup(nim_vios, vios, altdisk_hash) + rescue AltDiskCleanError => e + msg = "Cleanup failed: #{e.message}" + put_error(msg) + end + if ret == 0 + targets_status[vios_key] = if vios == vios1 + 'SUCCESS-ALTDCLEAN1' + else + 'SUCCESS-ALTDCLEAN2' + end + log_info("Alternate disk cleanup succeeded on #{altdisk_hash[vios]} of #{vios}") + else + put_warn("Failed to clean the alternate disk on #{altdisk_hash[vios]} of #{vios}") unless ret == 0 + end + end + end + + log_info("Alternate disk cleanup status for #{vios_key}: #{targets_status[vios_key]}") + # altdisk_cleanup + end # target_list.each + # Print target status + put_info("Status synthesis for viosupgrade operation:\n") + targets_status.each do |targ, v| + put_info("Status for :#{targ} => #{v}\n") + end +end