Invincible #Postgres Saga, episode 3...
(At least this didn't happen on production
)
Earlier this week I tried to do some tests on my "wonderful" misconfigured testing environment. I had some errors here and there and they had to reveal themselves during my experiment...
To make everthing up-to-date I had to refresh data in my app, which uses #Patroni #Postgresql database. Of course it crashed when I tried to do it. It didn't turn down whole cluster, only that one server, and in theory it still had normal API access. I expected it to come back quickly after I fixed things breaking my servers. But for some reason it couldn't. It didn't even try, at least in any visible way.
Seriuosly it was the most disturbing thing there
It should try to do something, right?!?
I tried to restart services in panic, without any result. It was late and I finished work for that day. Tired and worried I had to leave it stuck in "cannot reach consistent state" stage.
Next day I thought about doing force reinit or something to move it. But when I log into that server... Everything just worked 
I checked logs... Almost nothing special, just like database was stuck and after ~14 hours suddenly started working. Only one line telling backup was restored.
Damn, what backup?!? Why I don't understand what it did?!? And why it needed so much time?!?
At least I had working environment and didn't had to wonder what to do. Maybe my first chaotic attempts to force it to work did more harm than good and I should just leave it...
Someone should write in docs, which API status means "just leave me alone, trying to fix things"
It would be easier.