A network event stream processing system, in Clojure. Riemann - A network monitoring system

How can I use clj-http in riemann.config

I use riemann and now I write my riemann.config.

I want to use clj-http post all events from riemann stream to my web server. But I don't know how to import clj-http from riemann.jar.

I code (:use clj-http.client) or (:require [clj-http.client :as client]) in riemann.config but got error:

java.lang.ClassNotFoundException: clj-http.client

Could anyone help me ?

Source: (StackOverflow)

Riemann - trigger resolve based on metric threshold

I am trying to set up an alert in riemann (through pagerduty) based on a threshold for a metric. If the threshold is breached the alert should be triggered, if the metric goes back within the threshold the alert should be resolved.

My steps are: 1) Create an event with state "warning" if threshold is breached 2) Create an event with state "ok" if threshold is not breached

My code for this looks like -

(let [index (default :ttl 120 (index))]
      (where (service #"test")
         (where (>= metric 100)
            (smap (fn [e]
                    (event {:service (:service e) :metric (:metric e) 
                            :state "warning" }) 

(I have only shown the relevant bits of code)

I see that this code does not create a new event if threshold is breached.

I am not sure if I am making a mistake. Any help would be appreciated.



Source: (StackOverflow)

Clojure Dashboard query

I am trying to show a graph on riemann-dashboard using query "pingDelay > 0" .

I already have indexed my data using following code

(let [index (index)]
  (defn write-dht-metric [e]
    (let [dhtstate (re-find #"dht_status: health\.(\S+), msg count (\d+) \((\d+) bytes\).*peak \{ping = (\d+)" (:pgmsg e))]
      (if (not= dhtstate nil)
          (prn "RESULT>" dhtstate)
          (index {:host "dht-info"
                  :service (:service e)
                  :time (unix-time)
                  :dhtStatus (get dhtstate 1)
                  :msgCount (get dhtstate 2)
                  :pingDelay (get dhtstate 3)}

However, I am not getting anything on graph. Earlier, I thought that perhaps its because my "pingDelay" is in string "12345", so, i also tried ":pingDelay #(Long. (get dhtstate 3))" without any success.

Can anyone please help me about what I must do to make it work?


Source: (StackOverflow)

Riemann Dashboard not outputting internal Riemann metrics

I'm new to Riemann and also new to ruby and Clojure as well. I'm trying to output the internal riemann events via (streams prn) in my riemann.config file. I currently see messages being printed out in the terminal from where I launched riemann.


#riemann.codec.Event{:host "localhost.localdomain", :service "riemann server ws in latency 0.999", :state "ok", :description nil, :metric nil, :tags nil, :time 283857867047/200, :ttl 20}

In my dashboard however I'm unable to get these to print to any sort of log or gauge.

I tried using the following as a service =~ "riemann %" from here

I get an orange message displaying 1 socket errors; check the server field above or a large question market above the title.

Not sure what else to try or do from here to identify what went wrong.

Source: (StackOverflow)

ALerting in Riemann?

I am using ELK (logstash, ES, Kibana) stack for log analysis and Riemann for alerting. I have logs in which users is one of the fields parsed by logstash and I send the events to riemann from riemann output plugin.

Logstash parses logs and user is one of the field. Eg: logs parsed

Timestamp              user     command-name
 2014-06-07...         root      sh ./scripts/
 2014-06-08...         sid       sh ./scripts/
 2014-06-08...         abc       sh ./scripts/
 2014-06-09...         root      sh ./scripts/


riemann {
    riemann_event => {
        "service"     => "logins"
        "unique_user" => "%{user}"

So users values will be like: root, sid, abc, root, sid, def, etc....

So I split stream by user i.e one stream for each unique user. Now, I want to alert when number of unique users count go more than 3. I wrote the following but it's not achieving my purpose.



 (where (service "logins")
  (by :unique_user
    (moving-time-window 3600 
     (smap (fn [events]
        [users (count events)]
         (if (> users 3)
          (email "")       

I am new to Riemann and clojure. Any help is appreciated.

Source: (StackOverflow)

clojure.lang.LazySeq cannot be cast to clojure.lang.IFn

I'm new to Riemann and Clojure. All I want to do is to send email notifications to three email groups when some service's TTL is expired. I created some sort of config file where I store a list of emails:

  :email_group_1 (
  :email_group_2 (

My riemann config looks like this:

(logging/init {:console true})
(import org.apache.log4j.Level)
(logging/set-level Level/DEBUG)

(require '[ :as io])
(import '[ PushbackReader])

(let [host ""]
  (tcp-server {:host host :port 60001})
  (udp-server {:host host})
  (ws-server  {:host host :port 60003}))
(repl-server  {:host ""})

(def cwd (System/getProperty "user.dir"))

(def emails
  (with-open [r (io/reader (str cwd "/etc/emails.clj"))]
             (read (PushbackReader. r))))

(periodically-expire 5)

(def email (mailer))

(defn notify [& egroups]
  (for [egroup egroups]
    (rollup 1 60 (apply email (emails egroup)))))

(let [index (index)]
    (default :ttl 60

          (where (service "service_connect_active")
                    #(info "expired" %)
                    (notify :email_group_1 :email_group_2))))))

Code looks good (for me), but when this service is expired I get the following error:

09:45:39 riemann.1      | INFO [2015-05-08 10:45:39,313] Thread-5 - riemann.config - expired {:ttl 60, :time 357766884827/250, :state expired, :service service_connect_active, :host ava.local}
09:45:39 riemann.1      | WARN [2015-05-08 10:45:39,319] Thread-5 - riemann.config - clojure.lang.LazySeq@841649b8 threw
09:45:39 riemann.1      | java.lang.ClassCastException: clojure.lang.LazySeq cannot be cast to clojure.lang.IFn
09:45:39 riemann.1      |   at riemann.config$eval66$stream__70$fn__75.invoke(riemann.development.config:34)
09:45:39 riemann.1      |   at riemann.config$eval66$stream__70.invoke(riemann.development.config:45)
09:45:39 riemann.1      |   at riemann.streams$match$stream__3514$fn__3525.invoke(streams.clj:1209)
09:45:39 riemann.1      |   at riemann.streams$match$stream__3514.invoke(streams.clj:1209)
09:45:39 riemann.1      |   at riemann.streams$default$stream__3731$fn__3742.invoke(streams.clj:1328)
09:45:39 riemann.1      |   at riemann.streams$default$stream__3731.invoke(streams.clj:1328)
09:45:39 riemann.1      |   at riemann.core$stream_BANG_$fn__4415.invoke(core.clj:19)
09:45:39 riemann.1      |   at riemann.core$stream_BANG_.invoke(core.clj:18)
09:45:39 riemann.1      |   at riemann.core$reaper$worker__4529$fn__4539.invoke(core.clj:303)
09:45:39 riemann.1      |   at riemann.core$reaper$worker__4529.invoke(core.clj:297)
09:45:39 riemann.1      |   at riemann.service.ThreadService$thread_service_runner__1973$fn__1974.invoke(service.clj:71)
09:45:39 riemann.1      |   at riemann.service.ThreadService$thread_service_runner__1973.invoke(service.clj:70)
09:45:39 riemann.1      |   at
09:45:39 riemann.1      |   at

Could someone please help me? Thanks.

Source: (StackOverflow)

Riemann - Build a stream dynamically from a map

I have the following function which gets a map with service name and threshold. It checks if the service crossed a defined threshold and then calls multiple downstream children on the event.

(defn tc
  [s & children]
     (and (service (:service_name s)) (not (expired? event)))
       (by [:host :service]
         (where (> metric (:threshold s)
           (with :state "critical" 
             (apply sdo children)))))))

I would like to build a stream dynamically using a vector of maps:

(def services [{:service "cpu/usage" :threshold 90}
               {:service "memory/usage" :threshold 90}])

When trying to run it in a stream i'm getting the following warning:

  (doseq [s services] (tc s prn)))

WARN [2015-01-05 14:27:07,187] Thread-15 - riemann.core - instrumentation service caught
  at riemann.core$stream_BANG_$fn__11140.invoke(core.clj:19)
  at riemann.core$stream_BANG_.invoke(core.clj:18)
  at riemann.core$instrumentation_service$measure__11149.invoke(core.clj:57)
  at riemann.service.ThreadService$thread_service_runner__8782$fn__8783.invoke(service.clj:66)
  at riemann.service.ThreadService$thread_service_runner__8782.invoke(service.clj:65)

It works, if i run the streams function inside the doseq. This one works and gives the following output:

(doseq [s services]
  (streams (tc s prn)))

#riemann.codec.Event{:host "testhost", :service "memory/usage", :state "critical", :description nil, :metric 91.0, :tags nil, :time 1420460856, :ttl 60.0}

Source: (StackOverflow)

Riemann: triggering alerts with changed-state

I'm new to Riemann and clojure. I'm trying to configure alerts based on changed states. But the states never seem to be updated/indexed. So when I get to the changed-state block, state is nil. I can add the alerts within the splitp block, but that seems redundant. Maybe we will want different types of notifications between critical and warnings, but for now, I'd like to see this work (if possible).

(let [index (default :ttl 20 (index))]
    (where (not (state "expired"))
      (where (service "load/load/shortterm")
        (splitp < metric
          0.05 (with :state "critical" index)
          0.02 (with :state "warning" index)
               (with :state "ok" index))
      #(info %)
      (changed-state {:init "ok"}
        (stable 60 :state
          #(info "CHANGED STATE" %)
          (email "")))

Thanks for any help!

Riemann v0.2.9, collectd v5.5.0, OS CentOS 6.5

Source: (StackOverflow)

description event getting nil in riemann

I am trying to send email alert from logstash using Riemann. My email should get triggered on meeting certain criteria. I wrote Riemann config to send email alert for that I am sending certain events from logstash, I have hardcoded :description field but in my Riemann server I am seeing description as nil always. I don’t know where I am going wrong.

Riemann Config

(let [host ""]
  (tcp-server {:host host})
  (udp-server {:host host})
  (ws-server  {:host host}))

  ;Create index and print the values indexed
  (let [eindex (default :ttl 300 (update-index (index)))])

  ;Index event for reserve webservice failure
  (let [email (mailer{…….})]

    (where (service "e_log")
          (fn [events]
           (let [count-of-failures (count (filter #(re-find #"system space*" (:description %)) events))]        ;Calculate the count for matched value
                {:status "Failure"
                 :metric  count-of-failures 
                 :total-fail (>= count-of-failures 1)})))

          (where (and (= (:status event) "Failure")
                      (:total-fail event))

            (email ""))prn)))))

Logstash Config

             riemann_event => { "service" => "e_log"
"description" => "system space communication"
"metric" => "%{metric}"
"ttl" => "%{ttl}"                                                                                 

In my Riemann server I am seeing :description field as nil always so that :total-fail is false always.

Riemann Server

riemann.codec.Event{:host nil, :service nil, :state nil, :description nil, :metric 0, :tags nil, :time 1447406529, :ttl nil, :status "Failure", :total-fail false}
riemann.codec.Event{:host nil, :service nil, :state nil, :description nil, :metric 0, :tags nil, :time 1447406529, :ttl nil, :status "Failure", :total-fail false}
riemann.codec.Event{:host nil, :service nil, :state nil, :description nil, :metric 0, :tags nil, :time 1447406529, :ttl nil, :status "Failure", :total-fail false}
riemann.codec.Event{:host nil, :service nil, :state nil, :description nil, :metric 0, :tags nil, :time 1447406529, :ttl nil, :status "Failure", :total-fail false}
riemann.codec.Event{:host nil, :service nil, :state nil, :description nil, :metric 0, :tags nil, :time 1447406529, :ttl nil, :status "Failure", :total-fail false}

Thanks in advance

Source: (StackOverflow)

Need help in optimising clojure statement

I'm very new to clojure and need to set up a riemann config so that it would be easy to edit/add new conditions. What we have now:

(defn tell-ops 
    (by [:service]
        (throttle 3 360
            (rollup 2 360
                (email to)))))
([to channel]
    (by [:service]
        (throttle 3 360
            (rollup 2 360
                (slacker channel)
                (email to))))))


(where (state "FATAL")
    (where (service #"^Serv1")
            (tell-ops "" "#dev-ops1"))

    (where (service #"^Serv2")
            (tell-ops ""))


moreover, it lacks the default statement, something like if nothing matches, tell-ops ""

I think I need something like a top level struct

(def services 
 [{:regex #"^serv1" :mail "" :channel "#serv1"} 
  {:regex #"serv2$" :mail ""} ])

so that it would be easy to add new ones. But I have no idea how to loop throuth this array considering absence of :channel in the second case and making a "default call" if none of the regexes matches

Thanks in advance

Source: (StackOverflow)

Report CPU as always ok with Riemann

We're using Riemann and Riemann-health to monitor our servers. However now I get quite a lot of CPU critical warnings, because the CPU peaked for a very short time - This is nothing I even need to know about I think. From my understanding, a constant high CPU usage will increase the load avg, which will be reported as well and sounds way more useful.

I don't want to disable reporting the CPU, just every level should be considered to be ok. If possible, I'd like to change the events on the Riemann server, so I don't have to change all the servers.

Here our Riemann config:

Source: (StackOverflow)

Custom body message in riemann email

I am trying to create a custom message in the body section of email using riemann. I couldn't append the field dynamically.

Riemann config:

(let [email (mailer 
              {:host "XXXXX" :port XX :user "XXX" :pass "XXX" :auth "true"
               :subject (fn [events] "Team")
               :body (fn [events] 
                       (apply str "Hello Team, now the time is" (:timestamp event) "Thank You!"))
               :from ""})]

My output:

Hello Team, now the time is Thank You!

My expected output:

Hello Team, now the time is 12:13:45 Thank You!.

My timestamp not getting appended in the :body.

Source: (StackOverflow)

riemann email exception with SMTP

I was able to set up riemann with SMTP. Riemann appears to send the email out when the condition is met but I observe an exception (copied below) - any insight on how to troubleshoot/fix this will be appreciated.


    (def email (mailer {:host ""
                        :port "xxxx"
                        :user ""
                        :pass "user12345"
                        :from ""}))

 (where (and (service #"^riemann netty execution-handler"))
        (email "")))


#riemann.codec.Event{:host "ubuntu-3", :service "riemann netty execution-handler threads active", :state "ok", :description nil, :metric 0, :tags nil, :time 348380111059/250, :ttl 20000}
WARN [2014-02-27 12:00:44,278] Thread-10 - riemann.config -$mailer$make_stream__16773$stream__16774@4e9c33e9 threw
java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Number
        at postal.smtp$smtp_send_STAR_.invoke(smtp.clj:33)
        at postal.smtp$smtp_send.doInvoke(smtp.clj:58)
        at clojure.lang.RestFn.invoke(
        at postal.core$send_message.invoke(core.clj:35)
        at riemann.config$eval40$stream__41$fn__46.invoke(riemann.config:44)
        at riemann.config$eval40$stream__41.invoke(riemann.config:44)
        at riemann.core$stream_BANG_$fn__10513.invoke(core.clj:19)
        at riemann.core$stream_BANG_.invoke(core.clj:18)
        at riemann.core$instrumentation_service$measure__10522.invoke(core.clj:56)
        at riemann.service.ThreadService$thread_service_runner__8329$fn__8330.invoke(service.clj:64)
        at riemann.service.ThreadService$thread_service_runner__8329.invoke(service.clj:63)

Source: (StackOverflow)

Unable to install logtash contrib plugins?

I want to use logstash contrib plugin riemann in my config file. On running logstash error comes:

 An unexpected error occurred. This is probably a bug.   |
| You can find help with this problem in a few places:    |
|                                                         |
| * chat: #logstash IRC channel on freenode irc.          |
|     IRC via the web:                |
| * email:                |
| * bug system:                |
|                                                         |
The error reported is: 
  Couldn't find any output plugin named 'riemann'. Are you sure this is correct? Trying to load the riemann output plugin resulted in this error: no such file to load -- logstash/outputs/riemann

I have a folder inside which both the logstash and its contrib tar are present and extracted. I am using logstash 1.4.1 and logstash-contrib-1.4.1.

I tried the manual installation for contrib too by :

./bin/plugin install contrib

but nothing appears on the console on running the command.

Any help?


On ls the following is my directory structure:


    elasticsearch-1.1.1         kibana-3.1.0.tar.gz  logstash-1.4.1.tar.gz   logstash-contrib-1.4.1.tar.gz
    elasticsearch-1.1.1.tar.gz  logstash-1.4.1       logstash-contrib-1.4.1  riemann-0.2.5.tar.bz2

Thus I have untarred contrib in the same directory as logstash. Any IDEA??

Source: (StackOverflow)

Comparing event and adding new field in Riemann

I am facing one scenario. Let assume I have 3 events A, B and C which I am passing to Riemann.

Event A have following fields :ExamNo 3890 :ExamResult Pass :Rank 8
Event B have following fields :ExamNo 3890 :ExamResult Pass :Rank 5
Event C have following fields :ExamNo 3890 :ExamResult Fail :Rank 0

I need to compare the events based on ExamNo and whenver there is a change in ExamResult I need to add a new fields to event {:Eligible, :Grade}. I wrote the code to compare the events but the new field was not getting added to the event.

(let [examindex (default :ttl 300 (update-index (index)))]

  (where (service "Result")
          (fn [events]
        (changed :ExamResult))
            {    :Eligible :Eligible-Status
                 :Grade     :GradeValue

Since I am newbie to Riemann . I couldn't figure out the issue.

Source: (StackOverflow)