This ticket is a follow up of #2805 (there are similar comments at the bottom after closing it)

What did you do?

Run prometheus in a kubernetes cluster. On a GCE PD disk.

What did you see instead? Under which circumstances?

It crashed upon start, logfile:

level=error ts=2018-04-07T04:28:53.784390578Z caller=main.go:582
err="Opening storage failed unexpected end of JSON input"
level=info ts=2018-04-07T04:28:53.784418708Z caller=main.go:584
msg="See you next time!"

The point here is, that the meta.json file has a size of zero:

> ls -l /data/*/meta.json
[...]
-rw-rw-r--    1 1000     2000           283 Apr  7 03:05 01CAF10VV5FNDJ6PG84E6RSEV3/meta.json
-rw-rw-r--    1 1000     2000             0 Apr  7 03:15 01CAF1K5SQZT4HBQE9P6W7J56E/meta.json

Manual resolution

I've deleted that directory 01CAF1K5SQZT4HBQE9P6W7J56E with the problematic meta.json file in it and now it start up fine again.

Environment

• System information:

  Linux 4.10.0-40-generic x86_64

• Prometheus version:

prometheus, version 2.2.1 (branch: HEAD, revision: bc6058c81272a8d938c05e75607371284236aadc)
  build user:       root@149e5b3f0829
  build date:       20180314-14:15:45
  go version:       go1.10

("official" docker build)

• Logs:

level=error ts=2018-04-07T08:51:28.789897822Z caller=main.go:582 err="Opening storage failed unexpected end of JSON input"

Expected behavior

What I would wish is that prometheus starts up and doesn't CrashLoop. It should either

• ignore that directory, saying in the log that meta.json is faulty
• maybe move it to [directoryname].broken/ ?
• reconstruct the meta.json file from the data
• delete the problematic directory (bit harsh, ignoring might be better)