We've set up an alert to fire if disk space usage is greater than 97%. However, yesterday, this alert fired despite the disk space usage being around 31% as can be seen from the chart in the alert. Only a single node was affected.
Here is my alert query: (ts(ops.infrastructure.df.*.df_complex.used, env=prod-*) / (ts(ops.infrastructure.df.*.df_complex.used, env=prod-*) + ts(ops.infrastructure.df.*.df_complex.free, env=prod-*))) > 0.97 . I have attached the screenshot of the alert firing. Please let me know if you need anything else.