Netbox fails to connect to read-only postgresql

[fumped]

Deployment Type

Self-hosted

NetBox Version

v4.3.1

Python Version

3.10

Steps to Reproduce

Netbox is supposed to be able to run towards read only replicas of Postgres.
#12490
#11233

It fails to start.

We're running the official helm-chart but after input from Benjamin Reed it seems like the issue is in core netbox and not specific to the containerized version.

We have two completely separate kubernetes klusters in different countries. Lets call the main country C1 and the other site C2.

Netbox and the workers + Redis lives in kubernetes as containers and the Postgres databases are installed on RHEL machines.
C1 has 2 DBs, one Primary RW and one Replica in RO. C2 Just has a Replica in RO.

The idea is to have C1 running all the time and have C2 read-only towards the replica. In a DR scenario we can promote the RO replicas to RW.

C1 is up and running and is initialized. I want C2 to launch towards the same, populated databases.
So Yes, the Database is set to the Replica in C2 and to the VIP in C1.

Values:

Chart netbox C1

netbox:
  externalDatabase:
    host: IP-FOR-C1-VIP
    port: 5432
    database: netbox

Chart netbox C2

netbox:
  externalDatabase:
    host: IP-FOR-C2-IP
    port: 5432
    database: netbox    

Expected Behavior

Netbox should successfully launch.

Observed Behavior

"django.db.utils.OperationalError: connection failed: connection to server at "ip-address-here", port 5432 failed: session is read-only
[ Use DB_WAIT_DEBUG=1 in netbox.env to print full traceback for errors here ]
:hourglass_flowing_sand: Waiting on DB... (3s / 30s)"

[arthanson]

@fumped Thank you for opening a bug report. Unfortunately, the information you have provided is not sufficient for someone else to attempt to reproduce the reported behavior. Remember, each bug report must include detailed steps that someone else can follow on a clean, empty NetBox installation to reproduce the exact problem you're experiencing. These instructions should include the creation of any involved objects, any configuration changes, and complete accounting of the actions being taken. Also be sure that your report does not reference data on the public NetBox demo, as that is subject to change at any time by an outside party and cannot be relied upon for bug reports.

Can you please add repro steps, you don't put anything there about how you are configuring anything - are you setting DATABASES to something or?

[fumped]

Hi,

We have two completely separate kubernetes klusters in different countries. Lets call the main country C1 and the other site C2.

Netbox and the workers + Redis lives in kubernetes as containers and the Postgres databases are installed on RHEL machines.
C1 has 2 DBs, one Primary RW and one Replica in RO. C2 Just has a Replica in RO.

The idea is to have C1 running all the time and have C2 read-only towards the replica. In a DR scenario we can promote the RO replicas to RW.

C1 is up and running and is initialized. I want C2 to launch towards the same, populated databases.
So Yes, the Database is set to the Replica in C2 and to the VIP in C1.

Values:

# Chart netbox C1
netbox:
  externalDatabase:
    host: IP-FOR-C1-VIP
    port: 5432
    database: netbox
    
# Chart netbox C2
netbox:
  externalDatabase:
    host: IP-FOR-C2-IP
    port: 5432
    database: netbox    

If this is not supported, please let me know :).

[pycolas]

Hi, do you set MAINTENANCE_MODE on C2 ?

[fumped]

Hi, do you set MAINTENANCE_MODE on C2 ?

Yes, it's set to maintenance mode, using this as value to the chart.

netbox:
  maintenanceMode: true

I tried setting this in C1 too and it got applied correctly with the banner.

[github-actions[bot]]

This is a reminder that additional information is needed in order to further triage this issue. If the requested details are not provided, the issue will soon be closed automatically.

[fumped]

Got notice it will be auto-closed, I updates the main post. Hopefully that was it :)

[RangerRick]

Looking at the original links about RO mode, the work was done in NetBox 3.x. I wonder if the move to Django 5 in nb4.x broke something?

The pod startup is running ./manage.py showmigrations to check if the DB is available, and that's where the above error comes from. Just the act of showing migrations shouldn't, in theory, require read-write access to the database.

[cruse1977]

@fumped - is this relevant ? - #17026

[github-actions[bot]]

This is a reminder that additional information is needed in order to further triage this issue. If the requested details are not provided, the issue will soon be closed automatically.

[fumped]

@fumped - is this relevant ? - #17026

Hi, thanks for the link.
It does not seem to be the same issue, i don't see this user trying to connect to a read-only replica and my pods don't get to the traceback stage. It just keeps sending the same messages.
Seems like the solution for this ticket is not in the chart anymore but I tried setting it anyway, it made no difference.

externalDatabase:
targetSessionAttrs: any

[jeremystretch]

I've converted this bug report to a discussion as it's not clear what (if any) changes are needed in NetBox. If a specific, reproducible failure can be identified, please submit a new bug report detailing the specific fix(es) necessary.

[fumped]

Thanks!
Works for me! Bug reports are not for when I've found what I believe is a bug but rather when I've identified what the issue is? I do believe there's is a "a specific reproducible failure." I am not a developer so I cannot narrow it down further than showing my config and what I believe is supposed to work :)

Lets take a few steps back. Is netbox supposed to be able to run on RO psql or should I drop this? I can have config in standby and spin up the pods whenever the DB has to be promoted in a DR scenario.

What are the steps needed in case i've missed something?
It seems like it has worked before, maybe it's conisidered unsupported for now or low prio (?).