Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fulcrum demo #642

Draft
wants to merge 129 commits into
base: production
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
129 commits
Select commit Hold shift + click to select a range
fe13c69
try updating jdk and tomcat
antmoth Aug 25, 2023
d8a8238
fix demo-fedora name
antmoth Aug 25, 2023
733e4c7
fix demofedora name again?
antmoth Aug 25, 2023
435bc00
try disabling ssl for mysql
antmoth Aug 25, 2023
79b4dcc
reference demomysql in demofedora
antmoth Aug 25, 2023
f09f601
set password for demomysql?
antmoth Aug 25, 2023
4f5c1fa
use override_options for demomysql
antmoth Aug 25, 2023
7649583
fix override options hash syntax
antmoth Aug 25, 2023
ed35b6d
remove ssl-mode from demomysql
antmoth Aug 25, 2023
04cee8b
ensure that the parent directory of the webroot also exists
antmoth Aug 31, 2023
c7e2b7a
ensure solr bin dir exists
antmoth Aug 31, 2023
d3c8218
try setting JAVA_HOME?
antmoth Aug 31, 2023
f7dcf28
Add JAVA_HOME to solr service env vars
antmoth Aug 31, 2023
f923566
Update openjdk to adoptium/temurin, the maintained replacement
antmoth Sep 1, 2023
56062a7
undo solr java 11 env var
antmoth Sep 1, 2023
bda550b
change skip-ssl mysqld option to ssl-disable
antmoth Sep 5, 2023
4e0bdcb
change mysql override_options to just options
antmoth Sep 5, 2023
b3961cf
try to secure mysql?
antmoth Sep 11, 2023
480a29b
simplify fulcrum profile
antmoth Sep 11, 2023
45e4ba1
there's already something aliasing mysql-client to mariadb-client
antmoth Sep 11, 2023
37e4f5f
trying to get puppet to complete
antmoth Sep 13, 2023
92b25e8
just remove all of the fulcrum reqs for now
antmoth Sep 13, 2023
05214dc
adding back in fulcrum base
antmoth Sep 13, 2023
9ed36be
add back in fulcrum hosts + app
antmoth Sep 13, 2023
422b407
add fulcrum::logrotate back in
antmoth Sep 13, 2023
0727c09
add fulcrum::nginx back in
antmoth Sep 13, 2023
339829f
add back everything in fulcrum::standalone except mysql and fedora
antmoth Sep 13, 2023
b3f30bc
solr needs to use temurin jre
antmoth Sep 13, 2023
e105f52
nginx is failing
antmoth Sep 13, 2023
2c23a66
also comment out shib
antmoth Sep 14, 2023
cea3da0
also comment out solr
antmoth Sep 14, 2023
81a748b
add nginx back into the mix
antmoth Sep 14, 2023
653a2f4
try pulling in some apache stuff from fulcrum_www_and_app
antmoth Sep 14, 2023
2353578
do i need to remove the nginx profile file as well?
antmoth Sep 14, 2023
ab19b08
add .tool-versions to gitignore
antmoth Sep 18, 2023
15179d6
comment out most of the apache stuff
antmoth Sep 18, 2023
8ac8706
update www_lib to use temurin jdk
antmoth Sep 18, 2023
f3575c1
add apache fulcrum profile
antmoth Sep 18, 2023
5bde075
shibboleth?
antmoth Sep 18, 2023
1159506
odbc-mariadb => mariadb-unixodbc
antmoth Sep 18, 2023
f69ddcf
take out apache shib config; add back solr
antmoth Sep 18, 2023
2cc4152
try adding back in mysql
antmoth Sep 18, 2023
681d59f
remove mysql securing command thingy
antmoth Sep 18, 2023
af59626
simplify mysql setup
antmoth Sep 20, 2023
278c929
finish setting up mysql service
antmoth Sep 20, 2023
6b64bfd
does it need mysql-named packages instead of mariadb for some reason?
antmoth Sep 20, 2023
22507fe
still bashing at mysql
antmoth Sep 21, 2023
1b4e6dd
strip fulcrum standalone role back down to minimum for re-creating
antmoth Sep 21, 2023
b98fc88
update .ruby-version to 2.7
antmoth Sep 21, 2023
ceca98c
add things back in to standalone role
antmoth Sep 28, 2023
29a444c
add www lib perl
antmoth Sep 28, 2023
b2e9562
add in some apache stuff
antmoth Sep 28, 2023
f18f1f8
add rest of apache stuff
antmoth Sep 28, 2023
e8e57a0
take out shib stuff
antmoth Sep 28, 2023
5d07ee1
add back in solr
antmoth Sep 28, 2023
9ef0dc6
mysql...
antmoth Sep 28, 2023
3e865ae
require mariadb-server instead of mysql-server
antmoth Sep 28, 2023
edabeb0
fix my.cnf?
antmoth Sep 28, 2023
09c9b88
etc my.cnf source is a filepath
antmoth Sep 28, 2023
bddca11
don't need to do anything to my.cnf probably
antmoth Sep 29, 2023
faa2074
we do need some kind of mysql.cnf?
antmoth Sep 29, 2023
a5a0baa
only need one my.cnf maybe
antmoth Sep 29, 2023
c15d3e2
temporarily comment out mysql again
antmoth Sep 29, 2023
af05206
put mysql back
antmoth Sep 29, 2023
88fe30f
ensure /etc/mysql/conf.d
antmoth Sep 29, 2023
ee69306
note about needing to install_db
antmoth Sep 29, 2023
4afc869
experimental mysqldb function
antmoth Sep 29, 2023
c60db9a
try function syntax
antmoth Sep 29, 2023
74076e8
try a lambda
antmoth Sep 29, 2023
f8b1f26
exec name problems
antmoth Sep 29, 2023
ffdf41b
fix my iterable data
antmoth Sep 29, 2023
fc4578a
tyop
antmoth Sep 29, 2023
369b7ea
add the other three mysql dbs
antmoth Sep 29, 2023
ae7a890
add shib back?
antmoth Sep 29, 2023
98c3a1c
fix shib pkg names
antmoth Sep 29, 2023
abebba8
missed one
antmoth Sep 29, 2023
2323de6
add shib thing from apache profile?
antmoth Sep 29, 2023
e44f1db
remove duplicate decaration
antmoth Sep 29, 2023
bbd8830
shib?
antmoth Sep 29, 2023
a5e4eff
add fedora back in
antmoth Sep 29, 2023
ade401e
fedora profile is depending on mysql::db resources... deal w/ it later
antmoth Sep 29, 2023
3aac82d
Add fedora back in
antmoth Oct 25, 2023
0cdaf4e
require Service mysqld instead of nonexistent Mysql resource
antmoth Oct 25, 2023
5e233e7
s/tomcat8/tomcat9
antmoth Oct 25, 2023
359cd61
debugging tomcat
antmoth Oct 27, 2023
56630b4
Revert "debugging tomcat"
antmoth Oct 27, 2023
38bb751
Revert "Revert "debugging tomcat""
antmoth Oct 27, 2023
8152ee8
qualify echo path
antmoth Oct 27, 2023
949825d
test for paradox
antmoth Oct 27, 2023
d4e1a87
put tomcat create command back
antmoth Oct 27, 2023
91b96a9
fulcrum user may not have permissions to run tomcat create
antmoth Nov 10, 2023
44ef7bb
remove redundant ensure
antmoth Nov 10, 2023
b0ceb42
force symlinking of /opt/fedora/logs
antmoth Nov 10, 2023
38cf1f6
update Fedora version
antmoth Nov 30, 2023
6bccdc6
add capacity to handle symlinks to fulcrum::mounts
antmoth Dec 6, 2023
1f7dd45
set up mounts and symlinks for fulcrum standalone
antmoth Dec 8, 2023
0041304
change name of symlinks config to `config`
antmoth Dec 8, 2023
cfae0f1
try adding print debugging to symlinks
antmoth Dec 8, 2023
f86b7f5
remove print debugging from symlinks
antmoth Dec 8, 2023
a1653cd
fulcrum::nginx profile mysteriously disappeared in the merge???
antmoth Dec 11, 2023
c6b0243
recursively chown /opt/fedora to fulcrum:fulcrum
antmoth Dec 15, 2023
c03e51a
qualify chown
antmoth Dec 15, 2023
29392fa
change /etc/environment to use solr's JAVA_HOME
antmoth Dec 18, 2023
6939ce8
?? java alternative??
antmoth Dec 19, 2023
8e60545
add JAVA_HOME to fedora.env
antmoth Dec 19, 2023
2888393
tidy up fulcrum-demo mysql dbs and hosts
antmoth Aug 22, 2024
bfaa58f
specify user for fulcrum mysql dbs
antmoth Sep 20, 2024
6643285
actual code change for prev commit
antmoth Sep 20, 2024
30c596e
remove merge conflict marker from .gitignore
antmoth Oct 2, 2024
757f15f
pull standalone fulcrum perl into its own file
antmoth Oct 2, 2024
43efddc
fix solr tests
antmoth Oct 9, 2024
eebf009
add fulcrum mysql root_password to fixture data
antmoth Oct 9, 2024
4efe047
don't need to change openjdk -> temurin outside fulcrum
antmoth Nov 12, 2024
cbc7450
Tidy up fulcrum solr profile, & add spec
antmoth Nov 21, 2024
1d471c6
add fulcrum fedora spec
antmoth Nov 21, 2024
221b343
more tests
antmoth Nov 25, 2024
6807e66
fix tests
antmoth Nov 26, 2024
9bdeaf4
fix rubocop offenses
antmoth Dec 3, 2024
11b0e52
use odbc-mariadb for shibboleth
antmoth Dec 3, 2024
d3e8808
restore java_home line
antmoth Dec 3, 2024
c8684cf
add -headless back to default jre
antmoth Dec 3, 2024
bbba2a7
pander to puppet-lint
antmoth Dec 3, 2024
5e5c088
rubocop
antmoth Dec 3, 2024
c4b8249
switch UseParNewGC is no longer supported on jdk 11
antmoth Dec 5, 2024
4e12b7f
if we are using java 11, we can use default solr profile
antmoth Dec 5, 2024
1e06705
update jre bin location?
antmoth Dec 5, 2024
ef3ddf8
misunderstood java_home format
antmoth Dec 5, 2024
5ade3bd
make fulcrum solr profile
antmoth Jan 29, 2025
9517bec
tell fulcrum role to use fulcrum solr profile
antmoth Jan 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,4 @@
.DS_Store
/modules/
Puppetfile.lock
.tool-versions
45 changes: 29 additions & 16 deletions manifests/profile/fulcrum/fedora.pp
Original file line number Diff line number Diff line change
Expand Up @@ -8,46 +8,59 @@
String $fedora_username = 'fedora',
String $fedora_password = lookup('nebula::profile::fulcrum::mysql::fedora_password'),
) {
$jdk_version = lookup('nebula::jdk_version')
# used in erb file
$java_home = "/usr/lib/jvm/temurin-${jdk_version}-jre-amd64"

nebula::usergroup { 'fulcrum': }

ensure_packages([
'tomcat8-user',
'tomcat9-user',
"temurin-${jdk_version}-jre",
])

file { '/etc/sudoers.d/fedora':
content => template('nebula/profile/fulcrum/sudoers-fedora.erb'),
}

file {
['/var/lib/fedora', '/var/log/fedora', '/opt/fedora', '/tmp/fedora']:
ensure => directory,
owner => 'fulcrum',
group => 'fulcrum',
;
}

exec { 'create fedora tomcat':
command => '/usr/bin/tomcat8-instance-create fedora',
command => '/usr/bin/tomcat9-instance-create fedora',
cwd => '/opt',
user => 'fulcrum',
creates => '/opt/fedora',
user => 'fulcrum',
require => [
User['fulcrum'],
Package['tomcat8-user'],
Package['tomcat9-user'],
],
}

file {
['/var/lib/fedora', '/var/log/fedora', '/opt/fedora', '/tmp/fedora']:
ensure => directory,
owner => 'fulcrum',
group => 'fulcrum',
require => Exec['create fedora tomcat'],
;
}

exec { 'chown -r /opt/fedora':
command => '/usr/bin/chown -R fulcrum:fulcrum /opt/fedora',
require => Exec['create fedora tomcat'],
}

file { '/opt/fedora/logs':
ensure => 'symlink',
owner => 'fulcrum',
group => 'fulcrum',
force => true,
target => '/var/log/fedora',
require => Exec['create fedora tomcat'],
}

archive { '/opt/fedora/webapps/fedora.war':
ensure => present,
extract => false,
source => 'https://github.com/fcrepo/fcrepo/releases/download/fcrepo-4.7.4/fcrepo-webapp-4.7.4.war',
checksum => '11e06c843f40cf2b9f26bda94ddfe6d85d69a591',
source => 'https://github.com/fcrepo/fcrepo/releases/download/fcrepo-4.7.6/fcrepo-webapp-4.7.6.war',
checksum => '5882d8a4dc8b3817374503dff2043be79d9bbd72',
checksum_type => 'sha1',
cleanup => false,
user => 'fulcrum',
Expand Down Expand Up @@ -81,7 +94,7 @@
File['/etc/systemd/system/fedora.service'],
File['/var/lib/fedora'],
Archive['/opt/fedora/webapps/fedora.war'],
Mysql::Db['fedora'],
Service['mysqld'],
],
}
}
5 changes: 5 additions & 0 deletions manifests/profile/fulcrum/hosts.pp
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
class nebula::profile::fulcrum::hosts (
$fedora = '127.0.0.1',
$mysql = '127.0.0.1',
$keycard = '127.0.0.1',
$redis = '127.0.0.1',
$solr = '127.0.0.1',
) {
Expand All @@ -17,6 +18,10 @@
ip => $mysql,
}

host { 'keycard':
ip => $keycard,
}

host { 'redis':
ip => $redis,
}
Expand Down
56 changes: 38 additions & 18 deletions manifests/profile/fulcrum/mysql.pp
Original file line number Diff line number Diff line change
Expand Up @@ -7,32 +7,52 @@
class nebula::profile::fulcrum::mysql (
String $fedora_password,
String $fulcrum_password,
String $checkpoint_password,
String $shibd_password,
String $root_password,
) {
include nebula::profile::mysql

mysql::db { 'fedora':
user => 'fedora',
password => $fedora_password,
host => 'localhost',
# Install and configure mysql server
ensure_packages(['mariadb-common','mariadb-server', 'mariadb-client'])

# at some point need to do equivalent to `mysql_install_db --user=mysql --ldata=/var/lib/mysql`

service { 'mysqld':
ensure => running,
enable => true,
require => Package['mariadb-server'],
}

file { '/etc/mysql/conf.d':
ensure => 'directory'
}

mysql::db { 'fulcrum':
user => 'fulcrum',
password => $fulcrum_password,
host => 'localhost',
file { '/etc/mysql/my.cnf':
owner => 'mysql',
group => 'mysql',
content => template('nebula/mysql/my.cnf.erb'),
notify => Service['mysqld'],
require => Package['mariadb-server'],
}

mysql::db { 'checkpoint':
user => 'checkpoint',
password => $checkpoint_password,
host => 'localhost',
exec { 'set-mysql-password':
unless => "mysqladmin -uroot -p${root_password} status",
path => ['/bin', '/usr/bin'],
command => "mysqladmin -uroot password ${root_password}",
require => Service['mysqld'],
}

mysql::db { 'shibd':
user => 'shibd',
password => $shibd_password,
host => 'localhost',
$dbs = [['fedora', 'fedora', $fedora_password], ['fulcrum', 'fulcrum', $fulcrum_password],
['checkpoint', 'fulcrum', $fulcrum_password], ['shibd', 'shibd', $shibd_password]]

$dbs.each |$db| {
$name = $db[0]
$user = $db[1]
$password = $db[2]
exec { "create-${name}-db":
unless => "/usr/bin/mysql -u${user} -p${password} ${name}",
command => "/usr/bin/mysql -uroot -p${root_password} -e \"create database ${name}; grant all on ${name}.* to ${user}@localhost identified by '${password}';\"",
require => Service['mysqld'],
}
}

}
2 changes: 1 addition & 1 deletion manifests/profile/fulcrum/nginx.pp
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,6 @@
proto => 'tcp',
dport => 443,
state => 'NEW',
jump => 'accept',
action => 'accept',
}
}
37 changes: 37 additions & 0 deletions manifests/profile/fulcrum/perl.pp
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# The perl profile is needed for monitor_pl to work, but it pulls in a
# ton of stuff. We should probably allow for different haproxy http checks
# for a service, and eliminate the perl/monitor_pl dependency here.

class nebula::profile::fulcrum::perl (
Hash $hosts = {}
) {

include nebula::profile::www_lib::perl

create_resources('host',$hosts)

include nebula::profile::www_lib::apache::base
include nebula::profile::www_lib::apache::fulcrum

cron {
default:
user => 'root',
;

'purge apache access logs 1/2':
hour => 1,
minute => 7,
command => '/usr/bin/find /var/log/apache2 -type f -mtime +14 -name "*log*" -exec /bin/rm {} \; > /dev/null 2>&1',
;

'purge apache access logs 2/2':
hour => 1,
minute => 17,
command => '/usr/bin/find /var/log/apache2 -type f -mtime +2 -name "*log*" ! -name "*log*gz" -exec /usr/bin/pigz {} \; > /dev/null 2>&1',
require => Package['pigz'],
;
}

ensure_packages(['pigz'])

}
8 changes: 4 additions & 4 deletions manifests/profile/fulcrum/shibboleth.pp
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@
class nebula::profile::fulcrum::shibboleth {
ensure_packages([
'unixodbc',
'shibboleth-sp2-common',
'shibboleth-sp2-utils',
'mariadb-unixodbc',
'shibboleth-sp-common',
'shibboleth-sp-utils',
'odbc-mariadb',
])

file { '/etc/odbcinst.ini':
Expand Down Expand Up @@ -52,7 +52,7 @@
ensure => 'running',
enable => true,
hasrestart => true,
require => [Package['shibboleth-sp2-utils'], Package['mariadb-unixodbc']]
require => [Package['shibboleth-sp-utils'], Package['odbc-mariadb']]
}

service { 'shibauthorizer.socket':
Expand Down
74 changes: 69 additions & 5 deletions manifests/profile/fulcrum/solr.pp
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,75 @@

# nebula::profile::fulcrum::solr

class nebula::profile::fulcrum::solr {
class { 'nebula::profile::solr':
base => '/var/lib/solr',
home => '/var/lib/solr/data',
logs => '/var/log/solr',
class nebula::profile::fulcrum::solr (
String $base = '/var/lib/solr',
String $home = "${base}/home",
String $logs = "${base}/logs",
String $log4j_properties = "${base}/log4j.properties",
String $solr_in_sh = "${base}/solr.in.sh",
String $solr_xml = "${home}/solr.xml",
String $jdk_version = '8',
String $solr_home = '/var/lib/solr',
String $java_home = "/usr/lib/jvm/temurin-${jdk_version}-jre-${::os['architecture']}",
String $heap = '16G',
String $timezone = 'America/Detroit',
String $solr_bin = '/opt/solr/bin/solr',
){

ensure_packages([
"temurin-${jdk_version}-jre",
'solr',
'lsof',
])

nebula::usergroup { 'solr': }

file {
default:
owner => 'solr',
group => 'solr',
;
[$base, $home, $logs]:
ensure => 'directory',
mode => '0750',
;
$log4j_properties:
ensure => 'file',
mode => '0644',
content => template('nebula/profile/fulcrum/solr/log4j.properties.erb'),
;
$solr_in_sh:
ensure => 'file',
mode => '0644',
content => template('nebula/profile/fulcrum/solr/solr.in.sh.erb'),
;
$solr_xml:
ensure => 'file',
mode => '0644',
content => template('nebula/profile/fulcrum/solr/solr.xml.erb'),
;
}
file { '/etc/systemd/system/solr.service':
owner => 'root',
group => 'root',
content => template('nebula/profile/fulcrum/solr/solr.service.erb'),
}
service { 'solr':
ensure => 'running',
enable => true,
require => [Package['solr'], File['/etc/systemd/system/solr.service']],
}

class { 'nebula::profile::openjdk_java':
jdk_packages => ["temurin-${jdk_version}-jre"],
default_jdk => "temurin-${jdk_version}-jre",
base_alternative => $java_home,
java_alternative => "temurin-${jdk_version}-jre-amd64",
}

file { '/etc/environment':
content => inline_template("JAVA_HOME=${java_home}")
;
}

file {
Expand Down
6 changes: 3 additions & 3 deletions manifests/profile/kubernetes/haproxy.pp
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,9 @@

firewall {
default:
proto => 'tcp',
state => 'NEW',
jump => 'accept',
proto => 'tcp',
state => 'NEW',
jump => 'accept',
;

'200 private api':
Expand Down
8 changes: 4 additions & 4 deletions manifests/profile/letsencrypt.pp
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@
}

firewall { '200 HTTP':
proto => 'tcp',
dport => 80,
state => 'NEW',
jump => 'accept',
proto => 'tcp',
dport => 80,
state => 'NEW',
jump => 'accept',
}
}
6 changes: 3 additions & 3 deletions manifests/profile/networking/firewall.pp
Original file line number Diff line number Diff line change
Expand Up @@ -152,9 +152,9 @@

# Default IPv4 items, sorted by title
firewall { '001 accept related established rules':
proto => 'all',
state => ['RELATED', 'ESTABLISHED'],
jump => 'accept',
proto => 'all',
state => ['RELATED', 'ESTABLISHED'],
jump => 'accept',
}

firewall { '001 accept all to lo interface':
Expand Down
8 changes: 4 additions & 4 deletions manifests/profile/prometheus.pp
Original file line number Diff line number Diff line change
Expand Up @@ -173,10 +173,10 @@
},
}
firewall { '200 HTTPS: Client Cert':
proto => 'tcp',
dport => [443],
state => 'NEW',
jump => 'accept',
proto => 'tcp',
dport => [443],
state => 'NEW',
jump => 'accept',
}
}

Expand Down
2 changes: 1 addition & 1 deletion manifests/profile/solr.pp
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
ensure_packages(["openjdk-${jdk_version}-jre-headless",'solr','lsof'])

# Note: Along with variables above these are used in erb files also.
$java_home = "/usr/lib/jvm/java-${jdk_version}-openjdk-amd64/jre"
$java_home = "/usr/lib/jvm/java-${jdk_version}-openjdk-amd64"
$solr_bin = '/opt/solr/bin/solr'

nebula::usergroup { 'solr': }
Expand Down
Loading