I raised it in the percona community ( -discussions/percona-monitoring-and-management/53527-enabling-alerts-with-templated-dashboard) where i provided all the alert metrics& other details for investigation. Raising it in grafana community on their advice.
PMM Alerting With Grafana: Working With Templated Dashboards
Percona Monitoring and Management (PMM) 1.0.7 includes Grafana 4.0, which comes with the Alerting feature. Barrett Chambers shared how to enable alerting in general. This blog post looks at the specifics of setting up alerting based on the templated dashboards. Grafana 4.0 does not support basic alerting out-of-the-box.
In this article, I wanted to introduce you to Grafana, an Open Source dashboarding tool, allowing for aggregating different sources like Azure Monitor, Kubernetes, AWS, Docker and several other, and building up powerful dashboards. If you are only using Azure resources, I would personally recommend drilling down on all the capabilities of Azure Monitor and Log Analytics for sure. However, if you have a more hybrid or multi-cloud scenario, Grafana will definitely augment the power of Azure Monitor out of its integration with a huge amount of other data connectors.
Kubecost cross-correlates the usage statistics with cost data gathered from the billing records of public cloud providers (while also supporting on-premise implementations of Kubernetes) and allocates the costs across all Kubernetes resources such as namespaces, DaemonSets, pods, and even labels. The results are available in the form of dashboards and reports. The Kubecost alerts notify administrators of a sudden drop in efficiency or capacity headroom or detect a budget overrun.
vmalert executes a list of the given alerting or recording rules against configured -datasource.url compatible with Prometheus HTTP API. For sending alerting notifications vmalert relies on Alertmanager configured via -notifier.url flag. Recording rules results are persisted via remote write protocol and require -remoteWrite.url to be configured. Vmalert is heavily inspired by Prometheus implementation and aims to be compatible with its syntax.
Note there's a separate -remoteWrite.url command-line flag to allow writing results of alerting/recording rules into a different storage than the initial data that's queried. This allows using vmalert to aggregate data from a short-term, high-frequency, high-cardinality storage into a long-term storage with decreased cardinality and a bigger interval between samples. See also stream aggregation.
vmalert web UI can be accessed from single-node version of VictoriaMetrics and from cluster version of VictoriaMetrics. This may be used for better integraion with Grafana unified alerting system. See the following docs for details:
vmalert sends requests to /render?format=json during evaluation of alerting and recording rules if the corresponding group or rule contains type: "graphite" config option. It is expected that the /render implements Graphite Render API for format=json. When using vmalert with both graphite and prometheus rules configured against cluster version of VM do not forget to set -datasource.appendTypePrefix flag to true, so vmalert can adjust URL prefix automatically based on the query type. 2ff7e9595c
コメント