Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Needbackup #364

Open
wants to merge 99 commits into
base: 2.1
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
ea61250
Add Visual Studio Code directory with settings.json and launch.json s…
terwey Jan 13, 2021
b0e3d0a
Started initial work on refactoring cluster/prx*
terwey Jan 21, 2021
9628642
spelling correction
terwey Jan 21, 2021
f1906fe
Tiny code revert to get a smaller diff
terwey Jan 21, 2021
a9d040a
Merge branch '2.1' into prx-refactor
svaroqui Jan 23, 2021
e7fb982
Merge branch '2.1' of github.com:signal18/replication-manager into pr…
terwey Feb 9, 2021
221a394
Merge branch 'prx-refactor' of github.com:terwey/replication-manager …
terwey Feb 9, 2021
731ce57
Fix a few missed conflicts, remove cluster.newProxy() and replace by …
terwey Feb 9, 2021
4cf9a6f
Rename MdbsProxy to MariadbShardProxy and move Flags for Proxies from…
terwey Feb 13, 2021
56f3c21
Rename the file
terwey Feb 13, 2021
ed5ee94
Simplify build-flags
terwey Feb 13, 2021
a51562c
Remove no longer used build tags
terwey Feb 13, 2021
34a5ed9
Merge pull request #2 from terwey/build-flags
terwey Feb 13, 2021
0190a9b
Cleanup more build-flags and simplify the settings.json for vscode
terwey Feb 13, 2021
29a2fa6
Stop multi source on leader during failover
svaroqui Feb 14, 2021
950d2c4
More fixing on head cluster switchover
svaroqui Feb 18, 2021
584352b
Avoid crahing if Default section not present
svaroqui Feb 18, 2021
5e71d8c
Move prx.Id repetition to Proxy.SetID() and move crcTable to be clust…
terwey Feb 22, 2021
f0f69a1
Correct ordering of assignment
terwey Feb 22, 2021
ac171be
Move Get/Set methods to appropriate file
terwey Feb 22, 2021
8a542ee
Correct scoping for crcTable
terwey Feb 22, 2021
98fcab4
Reduce complexity
terwey Feb 22, 2021
8fd9a3b
Reduce complexity even more, also deduplicate
terwey Feb 22, 2021
d017dce
Remove futile error check
terwey Feb 22, 2021
e2041e9
Move the initialisation of the Proxy to a New*Proxy method in the app…
terwey Feb 22, 2021
d90b9cb
Small reductions, also checking for an error and doing continue is ea…
terwey Feb 23, 2021
00db46c
Simplify set/has/del of cookies for prx and prov
terwey Feb 23, 2021
b6745c1
More deduplication
terwey Feb 23, 2021
c95bcd1
Single commit to reduce the complexity of the cluster.checkFailed as …
terwey Feb 23, 2021
4a36415
Merge pull request #329 from terwey/clustercheckfailed
svaroqui Feb 24, 2021
cf59f33
Inter clusters replication source failover
svaroqui Feb 26, 2021
332dd85
Merge pull request #323 from terwey/prx-refactor
svaroqui Mar 2, 2021
f7620fe
Merge branch 'develop' into failover_multisource
svaroqui Mar 2, 2021
57c151f
Merge pull request #327 from signal18/failover_multisource
svaroqui Mar 2, 2021
1abc4b8
Fixing server id
svaroqui Mar 2, 2021
279f6b1
Add consul proxy and use interface for each proxy
svaroqui Mar 2, 2021
4c31364
Merge pull request #331 from signal18/ConsulProxy
svaroqui Mar 2, 2021
d126965
Fixing proxy flags inside virtual interface
svaroqui Mar 3, 2021
8f0c6cb
Don't load PFS query sample on monitoring push it to explain, this cr…
svaroqui Mar 17, 2021
38e62d6
Fix broken receiver
terwey Mar 17, 2021
cdf4c02
Add missing servie plan in share in case not fetchable from internet
svaroqui Mar 19, 2021
0576a87
Disable default backup-binlogs
svaroqui Apr 26, 2021
e481381
MariaDB Backup Stage support for mysqldump
svaroqui May 4, 2021
d93b98d
Implement provision call external script
svaroqui May 7, 2021
3b482f7
Fixing service plan download
svaroqui May 7, 2021
8293c03
ServicePlan copy csv from share if can not download
svaroqui May 7, 2021
939d845
Fixing multiple crash in service plan add cluster
svaroqui May 12, 2021
ad2cdc0
More fixing to cluster add via service plan
svaroqui May 12, 2021
823a36d
Preserve state for low frenquency of server CheckPrivileges
svaroqui May 14, 2021
697e1eb
Disable cross origin protocol for graph
svaroqui May 17, 2021
e4d287c
Add splitredolog tag in configurator
svaroqui May 20, 2021
64934da
Rig the election for switchover in case of multiple prefered master
svaroqui Jun 2, 2021
a35dce0
Fixing switchover with multiple prefered master #345
svaroqui Jun 3, 2021
5273b44
Stop database service under infinit loop
svaroqui Jun 4, 2021
5cbe43a
Add cluster name parameter to all prov scripts
svaroqui Jun 4, 2021
265d08b
Implement alerting for specific monitor triggers
terwey Jun 5, 2021
163ec0c
Resolve PR requests
terwey Jun 10, 2021
e8c68ea
Merge pull request #348 from terwey/issue_347
svaroqui Jun 10, 2021
a419004
Don't purge local binlog when not backup-binlogs enabled
svaroqui Jun 11, 2021
8105433
Add some extra config sample
svaroqui Jun 11, 2021
370e89f
Rig tag bing vs nobind
svaroqui Jun 11, 2021
700f060
Add logicalbackup physicalbackup cookie when backup is finished
svaroqui Jun 14, 2021
327ca04
Fix logging an error when none occured
terwey Jun 15, 2021
d61eddd
Only email on Opened states
terwey Jun 15, 2021
92ef95e
Exit checking the alert earlier if no email or script is set
terwey Jun 15, 2021
80296ee
Merge pull request #352 from terwey/issue_347
svaroqui Jun 15, 2021
9d594c9
Changing default mac config
svaroqui Jun 23, 2021
153772d
Wait stop database server should wait for state failed and ignore aut…
svaroqui Jun 23, 2021
447de0d
API disable/enable scheduler crash and is inverted
svaroqui Jun 23, 2021
13b9f25
API Scheduler can't start when never initialized from config
svaroqui Jun 23, 2021
b62f497
Fix logging an error when none occured
terwey Jun 15, 2021
66c6321
Only email on Opened states
terwey Jun 15, 2021
e69b600
Exit checking the alert earlier if no email or script is set
terwey Jun 15, 2021
91f5522
API disable/enable scheduler crash and is inverted
svaroqui Jun 23, 2021
110bc4c
API Scheduler can't start when never initialized from config
svaroqui Jun 23, 2021
ec8e054
Exit checking the alert earlier if no email or script is set
terwey Jun 15, 2021
b0bcdda
Failover election can failed when no master discovery
svaroqui Jun 28, 2021
b4d185f
Move DelWaitStartCookie, before refresh in case of refresh error
svaroqui Jun 28, 2021
f064496
Fix broken Cookies delete
svaroqui Jun 28, 2021
393a2e7
Add cookie_waitbackup on ressed or flashback if cookie_physicalbackup…
svaroqui Jun 30, 2021
f7e9dba
API refine error messages for get config
svaroqui Jul 3, 2021
9e82e56
Merge branch 'develop' into needbackup
svaroqui Jul 3, 2021
163a4c9
Config fixing proxysql can have empty monitor_writer_is_also_reader
svaroqui Jul 7, 2021
3904f87
Reseed on closing WARN0101
svaroqui Jul 8, 2021
c6e9f9c
Schema Monitor can crash when master discoverd from slave and master …
svaroqui Jul 8, 2021
003c3b9
Wrong warning missing GtidStrictMode in case not MariaDB #360
svaroqui Jul 8, 2021
142fa61
Print server infos in bad user grants
svaroqui Jul 8, 2021
dd80a83
Don't print RELAY_LOG insode variable diff
svaroqui Jul 9, 2021
1f164df
Track variable diff in cluster structure
svaroqui Jul 9, 2021
c7c2008
Fixing variable diff
svaroqui Jul 9, 2021
1ea7927
Fixing proxy delcookie
svaroqui Jul 9, 2021
3dc5eb0
GetBackupServer to send master if no server defined in config
svaroqui Jul 12, 2021
14440f9
ssh job script to set and use session env variable not to expose pass…
svaroqui Jul 15, 2021
a32090b
Split configurator in his on package for later usage ( still need a c…
svaroqui Jul 17, 2021
e61989c
Addding configurator
svaroqui Jul 17, 2021
b08702f
Merge branch '2.1' into needbackup
svaroqui Jul 17, 2021
ef2d34c
Merge branch 'develop' into needbackup
svaroqui Jul 17, 2021
7465c74
Flashback failed as always looking for all slave status and wrongly l…
svaroqui Jul 19, 2021
f28385c
Merge branch 'develop' into needbackup
svaroqui Jul 19, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Move prx.Id repetition to Proxy.SetID() and move crcTable to be clust…
…er property
  • Loading branch information
terwey committed Feb 22, 2021
commit 5e71d8c13dbe152c8fe3b0b164f69800bf4bfd89
3 changes: 3 additions & 0 deletions cluster/cluster.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"encoding/json"
"errors"
"fmt"
"hash/crc64"
"io/ioutil"
"os"
"os/exec"
Expand Down Expand Up @@ -154,6 +155,7 @@ type Cluster struct {
WaitingSwitchover int `json:"waitingSwitchover"`
WaitingFailover int `json:"waitingFailover"`
sync.Mutex
crcTable *crc64.Table
}

type ClusterSorter []*Cluster
Expand Down Expand Up @@ -226,6 +228,7 @@ const (

// Init initial cluster definition
func (cluster *Cluster) Init(conf config.Config, cfgGroup string, tlog *s18log.TermLog, log *s18log.HttpLog, termlength int, runUUID string, repmgrVersion string, repmgrHostname string, key []byte) error {
cluster.crcTable = crc64.MakeTable(crc64.ECMA) // http://golang.org/pkg/hash/crc64/#pkg-constants
cluster.switchoverChan = make(chan bool)
// should use buffered channels or it will block
cluster.statecloseChan = make(chan state.State, 100)
Expand Down
15 changes: 6 additions & 9 deletions cluster/prx.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ package cluster

import (
"fmt"
"hash/crc64"
"os"
"strconv"
"strings"
Expand Down Expand Up @@ -220,7 +219,6 @@ type proxyList []DatabaseProxy
func (cluster *Cluster) newProxyList() error {
nbproxies := 0

crcTable := crc64.MakeTable(crc64.ECMA) // http://golang.org/pkg/hash/crc64/#pkg-constants
if cluster.Conf.MxsHost != "" && cluster.Conf.MxsOn {
nbproxies += len(strings.Split(cluster.Conf.MxsHost, ","))
}
Expand Down Expand Up @@ -277,9 +275,8 @@ func (cluster *Cluster) newProxyList() error {
if cluster.Conf.ProvNetCNI {
prx.Host = prx.Host + "." + cluster.Name + ".svc." + cluster.Conf.ProvOrchestratorCluster
}
prx.Id = "px" + strconv.FormatUint(crc64.Checksum([]byte(cluster.Name+prx.Name+":"+strconv.Itoa(prx.WritePort)), crcTable), 10)
prx.ClusterGroup = cluster

prx.SetID()
prx.SetDataDir()
prx.SetServiceName(cluster.Name, prx.Name)
cluster.LogPrintf(LvlInfo, "New proxy monitored %s: %s:%s", prx.Type, prx.Host, prx.GetPort())
Expand All @@ -306,8 +303,8 @@ func (cluster *Cluster) newProxyList() error {
if cluster.Conf.ProvNetCNI {
prx.Host = prx.Host + "." + cluster.Name + ".svc." + cluster.Conf.ProvOrchestratorCluster
}
prx.Id = "px" + strconv.FormatUint(crc64.Checksum([]byte(cluster.Name+prx.Name+":"+strconv.Itoa(prx.WritePort)), crcTable), 10)
prx.ClusterGroup = cluster
prx.SetID()
prx.SetDataDir()
prx.SetServiceName(cluster.Name, prx.Name)
cluster.LogPrintf(LvlInfo, "New proxy monitored %s: %s:%s", prx.Type, prx.Host, prx.GetPort())
Expand All @@ -330,8 +327,8 @@ func (cluster *Cluster) newProxyList() error {
if prx.Name == "" {
prx.Name = prx.Host
}
prx.Id = "px" + strconv.FormatUint(crc64.Checksum([]byte(cluster.Name+prx.Name+":"+strconv.Itoa(prx.WritePort)), crcTable), 10)
prx.ClusterGroup = cluster
prx.SetID()
prx.SetDataDir()
prx.SetServiceName(cluster.Name, prx.Name)
cluster.LogPrintf(LvlInfo, "New proxy monitored %s: %s:%s", prx.Type, prx.Host, prx.GetPort())
Expand Down Expand Up @@ -383,8 +380,8 @@ func (cluster *Cluster) newProxyList() error {
prx.Port = "3306"
}
prx.WritePort, _ = strconv.Atoi(prx.GetPort())
prx.Id = "px" + strconv.FormatUint(crc64.Checksum([]byte(cluster.Name+prx.Name+":"+strconv.Itoa(prx.WritePort)), crcTable), 10)
prx.ClusterGroup = cluster
prx.SetID()
prx.SetDataDir()
prx.SetServiceName(cluster.Name, prx.Name)
cluster.LogPrintf(LvlInfo, "New proxy monitored %s: %s:%s", prx.Type, prx.Host, prx.GetPort())
Expand Down Expand Up @@ -414,8 +411,8 @@ func (cluster *Cluster) newProxyList() error {
if cluster.Conf.ProvNetCNI {
prx.Host = prx.Host + "." + cluster.Name + ".svc." + cluster.Conf.ProvOrchestratorCluster
}
prx.Id = "px" + strconv.FormatUint(crc64.Checksum([]byte(cluster.Name+prx.Name+":"+strconv.Itoa(prx.WritePort)), crcTable), 10)
prx.ClusterGroup = cluster
prx.SetID()
prx.SetDataDir()
prx.SetServiceName(cluster.Name, prx.Name)
cluster.LogPrintf(LvlInfo, "New proxy monitored %s: %s:%s", prx.Type, prx.Host, prx.GetPort())
Expand All @@ -441,11 +438,11 @@ func (cluster *Cluster) newProxyList() error {
if prx.Name == "" {
prx.Name = prx.Host
}
prx.Id = "px" + strconv.FormatUint(crc64.Checksum([]byte(cluster.Name+prx.Name+":"+strconv.Itoa(prx.WritePort)), crcTable), 10)
if prx.Host == "" {
prx.Host = "repman." + cluster.Name + ".svc." + cluster.Conf.ProvOrchestratorCluster
}
prx.ClusterGroup = cluster
prx.SetID()
prx.SetDataDir()
prx.SetServiceName(cluster.Name, prx.Name)
cluster.LogPrintf(LvlInfo, "New proxy monitored %s: %s:%s", prx.Type, prx.Host, prx.GetPort())
Expand Down
12 changes: 5 additions & 7 deletions cluster/prx_mariadbshardproxy.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,6 @@ import (
"github.com/spf13/pflag"
)

var crcTable = crc64.MakeTable(crc64.ECMA)

type MariadbShardProxy struct {
Proxy
}
Expand Down Expand Up @@ -77,7 +75,7 @@ func (proxy *MariadbShardProxy) Failover() {
if s == "replication_manager_schema" {
foundReplicationManagerSchema = true
}
checksum64 := crc64.Checksum([]byte(s+"_"+cluster.GetName()), crcTable)
checksum64 := crc64.Checksum([]byte(s+"_"+cluster.GetName()), cluster.crcTable)

query := "CREATE OR REPLACE SERVER RW" + strconv.FormatUint(checksum64, 10) + " FOREIGN DATA WRAPPER mysql OPTIONS (HOST '" + misc.Unbracket(cluster.master.Host) + "', DATABASE '" + s + "', USER '" + cluster.master.User + "', PASSWORD '" + cluster.master.Pass + "', PORT " + cluster.master.Port + ")"
_, err = proxy.ShardProxy.Conn.Exec(query)
Expand Down Expand Up @@ -123,7 +121,7 @@ func (cluster *Cluster) CheckMdbShardServersSchema(proxy *MariadbShardProxy) {
if s == "replication_manager_schema" {
foundReplicationManagerSchema = true
}
checksum64 := crc64.Checksum([]byte(s+"_"+cluster.GetName()), crcTable)
checksum64 := crc64.Checksum([]byte(s+"_"+cluster.GetName()), cluster.crcTable)

query := "CREATE SERVER IF NOT EXISTS RW" + strconv.FormatUint(checksum64, 10) + " FOREIGN DATA WRAPPER mysql OPTIONS (HOST '" + misc.Unbracket(cluster.master.Host) + "', DATABASE '" + s + "', USER '" + cluster.master.User + "', PASSWORD '" + cluster.master.Pass + "', PORT " + cluster.master.Port + ")"
_, err = proxy.ShardProxy.Conn.Exec(query)
Expand Down Expand Up @@ -217,7 +215,7 @@ func (cluster *Cluster) ShardProxyGetHeadCluster() *Cluster {
}

func (cluster *Cluster) ShardProxyCreateVTable(proxy *MariadbShardProxy, schema string, table string, duplicates []*ServerMonitor, withreshard bool) error {
checksum64 := crc64.Checksum([]byte(schema+"_"+cluster.GetName()), crcTable)
checksum64 := crc64.Checksum([]byte(schema+"_"+cluster.GetName()), cluster.crcTable)
var err error
var ddl string
if len(duplicates) == 1 {
Expand All @@ -239,7 +237,7 @@ func (cluster *Cluster) ShardProxyCreateVTable(proxy *MariadbShardProxy, schema
link_status_def := " link_status \""
for _, cl := range cluster.ShardProxyGetShardClusters() {
cl.CheckMdbShardServersSchema(proxy)
checksum64 := crc64.Checksum([]byte(schema+"_"+cl.GetName()), crcTable)
checksum64 := crc64.Checksum([]byte(schema+"_"+cl.GetName()), cluster.crcTable)
srv_def = srv_def + "RW" + strconv.FormatUint(checksum64, 10) + " "
link_status_def = link_status_def + "0 "
}
Expand Down Expand Up @@ -281,7 +279,7 @@ func (cluster *Cluster) ShardProxyCreateVTable(proxy *MariadbShardProxy, schema
clusterList := cluster.ShardProxyGetShardClusters()
for _, cl := range clusterList {
cl.CheckMdbShardServersSchema(proxy)
checksum64 := crc64.Checksum([]byte(schema+"_"+cl.GetName()), crcTable)
checksum64 := crc64.Checksum([]byte(schema+"_"+cl.GetName()), cluster.crcTable)
query = query + " PARTITION pt" + strconv.Itoa(i) + " COMMENT ='srv \"RW" + strconv.FormatUint(checksum64, 10) + "\", tbl \"" + table + "\", database \"" + schema + "\"'"
if i != len(clusterList) {
query = query + ",\n"
Expand Down
3 changes: 1 addition & 2 deletions cluster/prx_proxysql.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ package cluster
import (
"errors"
"fmt"
"hash/crc64"
"strconv"

"github.com/signal18/replication-manager/config"
Expand Down Expand Up @@ -40,7 +39,7 @@ func NewProxySQLProxy(clusterName string, proxyHost string, conf config.Config)
}
}

prx.Id = "px" + strconv.FormatUint(crc64.Checksum([]byte(clusterName+prx.Name+":"+strconv.Itoa(prx.WritePort)), crcTable), 10)
prx.SetID()

return prx
}
Expand Down
9 changes: 9 additions & 0 deletions cluster/prx_set.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,19 @@ package cluster

import (
"fmt"
"hash/crc64"
"os"
"strconv"
"strings"
)

func (p *Proxy) SetID() {
cluster := p.ClusterGroup
p.Id = "px" + strconv.FormatUint(
crc64.Checksum([]byte(cluster.Name+p.Name+":"+strconv.Itoa(p.WritePort)), cluster.crcTable),
10)
}

func (proxy *Proxy) SetServiceName(namespace string, name string) {
proxy.ServiceName = namespace + "/svc/" + name
}
Expand Down
7 changes: 3 additions & 4 deletions cluster/srv.go
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,6 @@ func (cluster *Cluster) newServerMonitor(url string, user string, pass string, c
server.IsCompute = compute
server.Domain = domain
server.TLSConfigUsed = ConstTLSCurrentConfig
server.CrcTable = crc64.MakeTable(crc64.ECMA)
server.ClusterGroup = cluster
server.DBVersion = dbhelper.NewMySQLVersion("Unknowed-0.0.0", "")
server.Name, server.Port, server.PostgressDB = misc.SplitHostPortDB(url)
Expand All @@ -221,9 +220,9 @@ func (cluster *Cluster) newServerMonitor(url string, user string, pass string, c
}*/
url = server.Name + server.Domain + ":3306"
}
server.Id = "db" + strconv.FormatUint(crc64.Checksum([]byte(cluster.Name+server.Name+server.Port), crcTable), 10)
server.Id = "db" + strconv.FormatUint(crc64.Checksum([]byte(cluster.Name+server.Name+server.Port), cluster.crcTable), 10)
var sid uint64
sid, err = strconv.ParseUint(strconv.FormatUint(crc64.Checksum([]byte(server.Name+server.Port), server.CrcTable), 10), 10, 64)
sid, err = strconv.ParseUint(strconv.FormatUint(crc64.Checksum([]byte(server.Name+server.Port), cluster.crcTable), 10), 10, 64)
server.ServerID = sid
if cluster.Conf.TunnelHost != "" {
go server.Tunnel()
Expand Down Expand Up @@ -732,7 +731,7 @@ func (server *ServerMonitor) Refresh() error {
if server.DBVersion.IsPPostgreSQL() {
//PostgresQL as no server_id concept mimic via internal server id for topology detection
var sid uint64
sid, err = strconv.ParseUint(strconv.FormatUint(crc64.Checksum([]byte(server.SlaveStatus.MasterHost.String+server.SlaveStatus.MasterPort.String), server.CrcTable), 10), 10, 64)
sid, err = strconv.ParseUint(strconv.FormatUint(crc64.Checksum([]byte(server.SlaveStatus.MasterHost.String+server.SlaveStatus.MasterPort.String), cluster.crcTable), 10), 10, 64)
if err != nil {
server.ClusterGroup.LogPrintf(LvlWarn, "PG Could not assign server_id s", err)
}
Expand Down