Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inject postgres admin into appliance console #153

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
9fb04ed
PostgresAdmin isn't rails, move to util.
jrafanie Aug 17, 2015
48c8d27
Drop Vmdb::Logging now that we're out of the Rails app.
jrafanie Aug 17, 2015
513c2e0
Add common appliance pg environment methods.
jrafanie Aug 17, 2015
cd1c7a1
Make the bulk of the start and stop commands a testable query method.
jrafanie Aug 19, 2015
71e9f3c
Fix minor bugs exposed by adding tests.
jrafanie Aug 19, 2015
7532ed1
Use Rails.root, not a hardcoded appliance path for the certs location.
jrafanie Aug 19, 2015
fd148fd
Outside of Rails, we need to build RAILS_ROOT manually.
jrafanie Aug 20, 2015
12506b8
Convert some missing constants to methods.
jrafanie Aug 24, 2015
47da2bc
Rubocop cleanup.
jrafanie Aug 24, 2015
6db1c21
ManageIQ code shouldn't know where the old system/TEMPLATE is
jrafanie Aug 28, 2015
2ecc233
Mass reformat
Oct 2, 2015
1c4f7d4
Use environment instead of pgpass file
kbrock Jan 25, 2016
e7fc458
added PostgresAdmin.database_in_recovery?(connection) method
yrudman Aug 11, 2016
f28cec8
Clear up the appliance console summary screen
carbonin Aug 10, 2016
cca905d
Add primary/standby to appliance console database status
carbonin Aug 24, 2016
9363c62
Move prep_data_directory method to PostgresAdmin
carbonin Sep 30, 2016
36953c9
Get tests running
bdunne Oct 10, 2016
e9e2eba
Move test directories to the root
bdunne Oct 18, 2016
8e4d6e6
Clean up references to GEMS_PENDING_ROOT and bundler_setup
bdunne Oct 27, 2016
45516ec
Run `bundle gem` against the repo to default most files.
Fryguy Nov 17, 2016
ffe3d03
Switch from coveralls to codeclimate-test-reporter
Fryguy Nov 18, 2016
1178621
Drop pglogical subscriptions before the restore
carbonin Nov 18, 2016
3d78935
Prevent Trollop from exiting the rspec process
carbonin Feb 14, 2017
fe3c521
Move the trollop stubs and errors to their own spec/support file
carbonin Feb 14, 2017
09412af
Original implementation of patch from #71
lmiccini Feb 22, 2017
8073be1
Add specs for PostgresAdmin.mount_point
carbonin Mar 15, 2017
012aaef
Alter PostgresAdmin.prep_data_directory to remove all contents
carbonin Apr 18, 2017
af67742
Remove the pglogical extension before attempting a database restore
carbonin Apr 25, 2017
98701fb
Fix typo dropping pglogical extension
gtanzillo May 3, 2017
ef6e1a3
Move logic for dropping pglogical extension to a separate method
gtanzillo May 3, 2017
1d7e924
Wait for "pglogical manager" db connection to go away before allowing…
gtanzillo May 3, 2017
ec38b72
Make tests pass
chessbyte May 19, 2017
bb5d906
Removed code moved to manageiq-smartstate repo
chessbyte May 31, 2017
b313143
Remove methods with no content
carbonin Jul 11, 2017
fed7cb8
Remove the start and stop methods from PostgresAdmin
carbonin Jul 11, 2017
4612949
Remove PostgresAdmin.pg_ctl
carbonin Jul 11, 2017
add26e5
Remove the scl methods from PostgresAdmin
carbonin Jul 11, 2017
c4637b4
Move database_in_recovery? from PostgresAdmin to PostgresHaAdmin
bdunne Oct 18, 2017
858a463
PostgresAdmin no longer uses 'pg' gem
bdunne Oct 19, 2017
8706549
load the file in lib/
bdunne Oct 19, 2017
3485cb2
reintroduce "require 'active_support/all'"
bdunne Oct 19, 2017
3708de5
Use pg_basebackup rather than pg_dump for backing up the database
carbonin Oct 23, 2017
427cb4b
Remove the before_restore method
carbonin Oct 24, 2017
67002af
Remove annoying comment
carbonin Oct 24, 2017
eaec9a2
Choose restore type based on the target file type
carbonin Oct 24, 2017
b114ceb
Removing test because because `before_restore` method was also removed.
gtanzillo Oct 25, 2017
c041909
Create an empty directory for pg_basebackup
carbonin Jan 25, 2018
c6e2884
Fix ENOENT error from backup_pg_compress
agrare Jan 26, 2018
6bedfeb
Dont mess with pglogical from PostgresAdmin
carbonin Feb 1, 2018
1d16b19
Remove references to RAILS_ROOT
carbonin Apr 30, 2018
aeb48b1
Add back pg_dump support to PostgresAdmin
NickLaMuro May 9, 2018
a43144f
Add PostgresAdmin#combine_command_args (private)
NickLaMuro May 9, 2018
e636e24
Use #combine_command_args in #runcmd
NickLaMuro May 9, 2018
0e51620
Handle multi-value args in PostgresAdmin.backup_pg_dump
NickLaMuro May 10, 2018
979da90
Update PG_DUMP_MULTI_VALUE_ARGS for underscore args
NickLaMuro Aug 14, 2018
21de264
Use --xlog-method=stream for pg_basebackup
carbonin Aug 15, 2018
381c81a
Revert to using --xlog-method fetch for postgres backup
carbonin Aug 24, 2018
74d9d4a
Use STDOUT for pg_basebackup
NickLaMuro Aug 22, 2018
aceff70
Use correct variable name for PostgresAdmin
NickLaMuro Aug 30, 2018
2ba4ce2
Don't set empty ENV values for database dumps
kbrock Oct 4, 2018
b5cf1dd
Add PostgresAdmin.group
NickLaMuro Oct 18, 2018
e7ae10f
Add --port option config to PostgresAdmin
NickLaMuro Oct 18, 2018
acd69cf
Add PgEnvironmentUpdater
NickLaMuro Oct 18, 2018
5d7d01b
Add PostgresRunner as a helper
NickLaMuro Oct 18, 2018
fd73435
Add sample pg backup and dump for testing
NickLaMuro Oct 18, 2018
3144de3
Add connection helper context for db restores
NickLaMuro Oct 18, 2018
ba3ebba
Add postgres backup specs
NickLaMuro Oct 18, 2018
b424c39
Add postgresql-9.5 to travis
NickLaMuro Oct 11, 2018
d33ab6d
Adds PostgresAdmin file testing specs
NickLaMuro Oct 25, 2018
9a9e896
Remove need for `file` shell call via .file_type
NickLaMuro Sep 21, 2018
80a7efd
Handle pipes for pg_restore
NickLaMuro Oct 1, 2018
b371d2c
[PostgresAdmin] Add backup_type option to .restore
NickLaMuro Oct 31, 2018
dd32246
[PostgresAdmin] Fix backup_type being ignored
NickLaMuro Nov 7, 2018
4a036f1
Use Gem::Package for tar pg restore unpacking
NickLaMuro Nov 7, 2018
fd4ba2a
Change pg_basebackup xlog-method option to wal-method
carbonin Feb 19, 2019
2dea6ea
Update CI PG restore tests for PG 10
carbonin Feb 20, 2019
816c545
Update backup and dump files for PG 10
carbonin Feb 20, 2019
7d0014a
ramfs-ing speed! - restore directly to ramfs data directory
carbonin Feb 20, 2019
121a5ae
Remove MiqUtil.runcmd in favor of AwesomeSpawn
Fryguy Jan 27, 2021
aa13289
Use Rakefile from manageiq-gems-pending
NickLaMuro Mar 9, 2021
fdea187
[Gemfile] Remove manageiq-gems-pending
NickLaMuro Mar 11, 2021
2811360
[PostgresAdmin] Namespace to ManageIQ::ApplianceConsole
NickLaMuro Mar 9, 2021
6400c62
[PostgresAdmin] Fix $log (to self.logger)
NickLaMuro Mar 9, 2021
4efb575
[RestoreHelper] Use proper data dir for backups
NickLaMuro Mar 9, 2021
8bfc157
[PostgresRunner] Use Dir::Tmpname.create
NickLaMuro Mar 9, 2021
6590665
[.travis.yml] postgresql addon
NickLaMuro Mar 10, 2021
21c6c85
[.rspec_ci] Load ci_helper
NickLaMuro Mar 10, 2021
ca94973
[.rspec_ci] --format documentation
NickLaMuro Mar 12, 2021
a83610e
[ci_helper.rb][CI] Set ENV vars earlier
NickLaMuro Mar 11, 2021
84938a0
[ci_helper.rb] LinuxAdmin::Service.new override
NickLaMuro Mar 11, 2021
9583441
[spec_helper.rb] Configure :with_postgres_specs
NickLaMuro Mar 9, 2021
4edbc2f
[spec/] Avoid ENV overwrite in specs
NickLaMuro Mar 11, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .rspec_ci
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
--require ci_helper
--require spec_helper
--color
--order random
--profile
--format documentation
2 changes: 2 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ rvm:
- 2.5.8
- 2.6.6
cache: bundler
addons:
postgresql: '10'
env:
global:
- RUBY_GC_HEAP_GROWTH_MAX_SLOTS=300000
Expand Down
2 changes: 0 additions & 2 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,3 @@ source 'https://rubygems.org'

# Specify your gem's dependencies in manageiq-appliance_console.gemspec
gemspec

gem "manageiq-gems-pending", :git => "https://github.com/ManageIQ/manageiq-gems-pending", :branch => "master"
21 changes: 20 additions & 1 deletion Rakefile
Original file line number Diff line number Diff line change
@@ -1,6 +1,25 @@
require "bundler/gem_tasks"
require "rspec/core/rake_task"

desc "Run RSpec code examples (skip postgres required ones)"
RSpec::Core::RakeTask.new(:spec)

task :default => :spec
# In CI, as part of the .rspec_ci, load a helper that sets the configuration
# setting to allow the postgres specs to run.
desc "Run RSpec code examples (assumes ci dependencies)"
RSpec::Core::RakeTask.new("spec:ci") do |t|
t.rspec_opts = "--options #{File.expand_path(".rspec_ci", __dir__)}"
end

desc "Run RSpec code examples (with local postgres dependencies)"
RSpec::Core::RakeTask.new("spec:dev") do |t|
# Load the PostgresRunner helper to facilitate a clean postgres environment
# for testing locally (not necessary for CI), and enables the postgres test
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though it's not necessary for CI, this seems to over complicate the code (and making local runs "different"). Can we just run it in CI anyway presuming it's relatively inexpensive?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So there is a couple of things here, but I did explain a lot of the architecture of this in the original PR here:

ManageIQ/manageiq-gems-pending#385

So I won't fully repeat that here (and I honestly forget a lot of my decisions without using that to cheat), but the main take aways were:

  • A travis box is ephemeral, so it is fine it we just nuke the entire DB, but on a local machine, a developers DB(s) for vmdb_development would be trashed every time this is run locally
  • When trying to do TDD with this code, having a way to be able to validate you haven't borked anything locally (instead of waiting for Travis) is really nice.

That all said, I also agree that basically adding an "init script written in Ruby" (see spec/postgres_runner_helper.rb ) is basically a lot of code to accomplish this, for like 2 tests...

They are arguably important tests, but depending on how much we do or do not move away from this backup strategy might determine if it is still of value.

P.S. This is also still WIP, so definitely things are still subject to change a little more yet.

# via the helper.
pg_runner = File.join("spec", "postgres_runner_helper.rb")
t.rspec_opts = "-r #{File.expand_path(pg_runner, __dir__)}"
end

task :default do
Rake::Task["spec#{':ci' if ENV['CI']}"].invoke
end
2 changes: 1 addition & 1 deletion bin/appliance_console
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ require 'highline/system_extensions'
require 'rubygems'
require 'bcrypt'
require 'linux_admin'
require 'util/postgres_admin'
require 'manageiq/appliance_console/postgres_admin'
require 'awesome_spawn'
include HighLine::SystemExtensions

Expand Down
1 change: 0 additions & 1 deletion lib/manageiq-appliance_console.rb
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ def self.logger
require 'manageiq/appliance_console/logging'
require 'manageiq/appliance_console/prompts'

require 'manageiq-gems-pending'
require 'highline'

require 'manageiq/appliance_console/auth_utilities'
Expand Down
2 changes: 1 addition & 1 deletion lib/manageiq/appliance_console/certificate_authority.rb
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
require 'fileutils'
require 'tempfile'
require 'util/postgres_admin'
require 'manageiq/appliance_console/postgres_admin'

module ManageIQ
module ApplianceConsole
Expand Down
2 changes: 1 addition & 1 deletion lib/manageiq/appliance_console/database_replication.rb
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
require 'pg'
require 'English'
require 'util/postgres_admin'
require 'manageiq/appliance_console/postgres_admin'

module ManageIQ
module ApplianceConsole
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
require 'util/postgres_admin'
require 'manageiq/appliance_console/postgres_admin'
require 'fileutils'
require 'linux_admin'

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
require "pathname"
require "util/postgres_admin"
require "manageiq/appliance_console/postgres_admin"
require "pg"
require "linux_admin"

Expand Down
Loading