Nave should not require elasticsearch to be running at startup

[hanswesterbeek]

I found this in the nave initscript:

  # Since nave craches silently after a few seconds when elasticsearch is not responding,
    # the init script fails to notice.
    # One reason for such craches is when elasticsearch is not awailable. during upstart.

This indicates a design error in Nave's startup process.

The desired behaviour from Nave is that it:

• starts up unless blatantly incorrectly configured
• reports 'unhealthy' status if it can't connect to a downstream service such as Elasticsearch
• Recovers once elasticsearch becomes available, and then reports 'healthy' status

[kiivihal]

Nave creates the required elasticsearch databases on startup and popuplates it with the right dynamic templates, so all the RDF can be indexed.

I think a similar check is made for the Triple Store. Without the correct configuration nave can't function properly.

I think when we take about the init scripts we just need to make sure all dependent services are up and running. Django will for example refuse to start when the configured database is not available.

So unless this really can't be fixed by better specifying the priority in the init script, I will mark this as won't fix.

[jaclu]

Sjoerd this is not a init script problem.

Starting elastic search just means firing it up.
Then elastic search needs some time to become ready to accept traffic - typically 20-30 seconds but could be longer some times.
We are using the default elastic search startup script, used by probably the vast majority of all deploys.

That a content backend needs some internal preparations before accepting requests is the norm rather than the exception.

I already use priorities during startup, but that is not sufficient for the above mentioned reasons.

The problem is that nave reports successful startup, then decides it cant actually run and dies.

Nave needs to check for critical services and refuse to go into daemon mode successfully if they are not met.

[jaclu]

as long as nave reports a required service is missing, i can then detect that and loop the startup to sleep 30 seconds and retry up to perhaps 10 times

[jaclu]

I didn't think of it first, but your example is describing exactly what you need to do:
"Django will for example refuse to start when the configured database is not available."

This is the exact behaviour that nave needs to follow, only difference is that in the case of nave the check isn't if the dataset is available, it is if the backend can be initiated. If an unconditional dependency isn't met at least fail, if possible fail with a msg and/or err_code indicating what dependency isn't full filled