We recently broke our license limits in Splunk. After going back to find out why, we found that we were being attacked and the firewalls were going crazy spitting out logs. We also found that we were able to find misconfigured devices based on log spikes. We typically log at let’s say 500kbps with a maximum of 700 kbps throughout the day. Taking this I can create a search to find the current rate of indexing. I then take that search and create an alert that will let me know if we spike over 800 kbps so I can go back and find out why we are logging so much.
index=_internal source=*metrics.log* group=per_index_thruput series!=_summary series!=_audit | timechart span=1m per_second(kb) AS CKB | where CKB > 800
Guarding your IT Experience!