A network event stream processing system, in Clojure. Riemann - A network monitoring system

ERROR: Failed to buld gem native extension?

I am installing riemann monitoring tool on my PC. I run the following command from terminal:


sudo gem install riemann-tools

I get the following error:

ERROR:  Error installing riemann-tools:
        ERROR: Failed to build gem native extension.

            /usr/bin/ruby1.9.1 extconf.rb
    /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- mkmf (LoadError)
        from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from extconf.rb:1:in `<main>'

A similar error comes when installing riemann-dash but riemann-client get installed successfully.

I am using Ububtu 12.10. I read online about such errors and it had something with ruby version, but I am not able to figure out the reason for this.

Any help is appreciated.

Source: (StackOverflow)

How to integrate riemann into the dropwizard to capture metrics?

I have a dropwizard application which emits yammer metrics and can be monitored via a URL like http://localhost:8081/admin/metrics which gives the result in form of jsons.

I want to send these monitor these metrics in riemann and I have no idea on how to start. I went through the riemann-java-client which has a RiemannReporter class for yammer metrics but I do not how to use this in my application.

How to integrate this client into my application or how to capture jsons from the url and send these as events to riemann server?

Source: (StackOverflow)

Riemann Context for Hadoop to send metrics to Riemann using metrics2 interface

Is there a library which can be integrated with the different Hadoop components (Namenode, datanode, jobtracker, tasktracker) as well as the Hadoop 2 components (Resource Manager) to send metrics to Riemann?

Source: (StackOverflow) add jar to classpath

I have written custom clojure functions that I want to use in my riemann configuration. I am using leiningen to build jar file (with dependencies) containing my functions. What is the right way to include this jar file in the classpath when starting riemann ?

Source: (StackOverflow)

Count riemann events in given time window

In riemann config for specific service I'm trying to assign to all its events metric=1, sum them within 5sec and send the result to influxdb.

I gave up with following:

  (where (service "offers")
    (fixed-time-window 5
      (smap folds/sum (with :metric 1 index))))

it doesn't really work, events stored in influx do not match this rule. any hint?

Source: (StackOverflow)

How can I use clj-http in riemann.config

I use riemann and now I write my riemann.config.

I want to use clj-http post all events from riemann stream to my web server. But I don't know how to import clj-http from riemann.jar.

I code (:use clj-http.client) or (:require [clj-http.client :as client]) in riemann.config but got error:

java.lang.ClassNotFoundException: clj-http.client

Could anyone help me ?

Source: (StackOverflow)

Clojure Dashboard query

I am trying to show a graph on riemann-dashboard using query "pingDelay > 0" .

I already have indexed my data using following code

(let [index (index)]
  (defn write-dht-metric [e]
    (let [dhtstate (re-find #"dht_status: health\.(\S+), msg count (\d+) \((\d+) bytes\).*peak \{ping = (\d+)" (:pgmsg e))]
      (if (not= dhtstate nil)
          (prn "RESULT>" dhtstate)
          (index {:host "dht-info"
                  :service (:service e)
                  :time (unix-time)
                  :dhtStatus (get dhtstate 1)
                  :msgCount (get dhtstate 2)
                  :pingDelay (get dhtstate 3)}

However, I am not getting anything on graph. Earlier, I thought that perhaps its because my "pingDelay" is in string "12345", so, i also tried ":pingDelay #(Long. (get dhtstate 3))" without any success.

Can anyone please help me about what I must do to make it work?


Source: (StackOverflow)

Riemann Dashboard not outputting internal Riemann metrics

I'm new to Riemann and also new to ruby and Clojure as well. I'm trying to output the internal riemann events via (streams prn) in my riemann.config file. I currently see messages being printed out in the terminal from where I launched riemann.


#riemann.codec.Event{:host "localhost.localdomain", :service "riemann server ws in latency 0.999", :state "ok", :description nil, :metric nil, :tags nil, :time 283857867047/200, :ttl 20}

In my dashboard however I'm unable to get these to print to any sort of log or gauge.

I tried using the following as a service =~ "riemann %" from here

I get an orange message displaying 1 socket errors; check the server field above or a large question market above the title.

Not sure what else to try or do from here to identify what went wrong.

Source: (StackOverflow)

ALerting in Riemann?

I am using ELK (logstash, ES, Kibana) stack for log analysis and Riemann for alerting. I have logs in which users is one of the fields parsed by logstash and I send the events to riemann from riemann output plugin.

Logstash parses logs and user is one of the field. Eg: logs parsed

Timestamp              user     command-name
 2014-06-07...         root      sh ./scripts/
 2014-06-08...         sid       sh ./scripts/
 2014-06-08...         abc       sh ./scripts/
 2014-06-09...         root      sh ./scripts/


riemann {
    riemann_event => {
        "service"     => "logins"
        "unique_user" => "%{user}"

So users values will be like: root, sid, abc, root, sid, def, etc....

So I split stream by user i.e one stream for each unique user. Now, I want to alert when number of unique users count go more than 3. I wrote the following but it's not achieving my purpose.



 (where (service "logins")
  (by :unique_user
    (moving-time-window 3600 
     (smap (fn [events]
        [users (count events)]
         (if (> users 3)
          (email "")       

I am new to Riemann and clojure. Any help is appreciated.

Source: (StackOverflow)

clojure.lang.LazySeq cannot be cast to clojure.lang.IFn

I'm new to Riemann and Clojure. All I want to do is to send email notifications to three email groups when some service's TTL is expired. I created some sort of config file where I store a list of emails:

  :email_group_1 (
  :email_group_2 (

My riemann config looks like this:

(logging/init {:console true})
(import org.apache.log4j.Level)
(logging/set-level Level/DEBUG)

(require '[ :as io])
(import '[ PushbackReader])

(let [host ""]
  (tcp-server {:host host :port 60001})
  (udp-server {:host host})
  (ws-server  {:host host :port 60003}))
(repl-server  {:host ""})

(def cwd (System/getProperty "user.dir"))

(def emails
  (with-open [r (io/reader (str cwd "/etc/emails.clj"))]
             (read (PushbackReader. r))))

(periodically-expire 5)

(def email (mailer))

(defn notify [& egroups]
  (for [egroup egroups]
    (rollup 1 60 (apply email (emails egroup)))))

(let [index (index)]
    (default :ttl 60

          (where (service "service_connect_active")
                    #(info "expired" %)
                    (notify :email_group_1 :email_group_2))))))

Code looks good (for me), but when this service is expired I get the following error:

09:45:39 riemann.1      | INFO [2015-05-08 10:45:39,313] Thread-5 - riemann.config - expired {:ttl 60, :time 357766884827/250, :state expired, :service service_connect_active, :host ava.local}
09:45:39 riemann.1      | WARN [2015-05-08 10:45:39,319] Thread-5 - riemann.config - clojure.lang.LazySeq@841649b8 threw
09:45:39 riemann.1      | java.lang.ClassCastException: clojure.lang.LazySeq cannot be cast to clojure.lang.IFn
09:45:39 riemann.1      |   at riemann.config$eval66$stream__70$fn__75.invoke(riemann.development.config:34)
09:45:39 riemann.1      |   at riemann.config$eval66$stream__70.invoke(riemann.development.config:45)
09:45:39 riemann.1      |   at riemann.streams$match$stream__3514$fn__3525.invoke(streams.clj:1209)
09:45:39 riemann.1      |   at riemann.streams$match$stream__3514.invoke(streams.clj:1209)
09:45:39 riemann.1      |   at riemann.streams$default$stream__3731$fn__3742.invoke(streams.clj:1328)
09:45:39 riemann.1      |   at riemann.streams$default$stream__3731.invoke(streams.clj:1328)
09:45:39 riemann.1      |   at riemann.core$stream_BANG_$fn__4415.invoke(core.clj:19)
09:45:39 riemann.1      |   at riemann.core$stream_BANG_.invoke(core.clj:18)
09:45:39 riemann.1      |   at riemann.core$reaper$worker__4529$fn__4539.invoke(core.clj:303)
09:45:39 riemann.1      |   at riemann.core$reaper$worker__4529.invoke(core.clj:297)
09:45:39 riemann.1      |   at riemann.service.ThreadService$thread_service_runner__1973$fn__1974.invoke(service.clj:71)
09:45:39 riemann.1      |   at riemann.service.ThreadService$thread_service_runner__1973.invoke(service.clj:70)
09:45:39 riemann.1      |   at
09:45:39 riemann.1      |   at

Could someone please help me? Thanks.

Source: (StackOverflow)

Riemann - Build a stream dynamically from a map

I have the following function which gets a map with service name and threshold. It checks if the service crossed a defined threshold and then calls multiple downstream children on the event.

(defn tc
  [s & children]
     (and (service (:service_name s)) (not (expired? event)))
       (by [:host :service]
         (where (> metric (:threshold s)
           (with :state "critical" 
             (apply sdo children)))))))

I would like to build a stream dynamically using a vector of maps:

(def services [{:service "cpu/usage" :threshold 90}
               {:service "memory/usage" :threshold 90}])

When trying to run it in a stream i'm getting the following warning:

  (doseq [s services] (tc s prn)))

WARN [2015-01-05 14:27:07,187] Thread-15 - riemann.core - instrumentation service caught
  at riemann.core$stream_BANG_$fn__11140.invoke(core.clj:19)
  at riemann.core$stream_BANG_.invoke(core.clj:18)
  at riemann.core$instrumentation_service$measure__11149.invoke(core.clj:57)
  at riemann.service.ThreadService$thread_service_runner__8782$fn__8783.invoke(service.clj:66)
  at riemann.service.ThreadService$thread_service_runner__8782.invoke(service.clj:65)

It works, if i run the streams function inside the doseq. This one works and gives the following output:

(doseq [s services]
  (streams (tc s prn)))

#riemann.codec.Event{:host "testhost", :service "memory/usage", :state "critical", :description nil, :metric 91.0, :tags nil, :time 1420460856, :ttl 60.0}

Source: (StackOverflow)

Riemann: triggering alerts with changed-state

I'm new to Riemann and clojure. I'm trying to configure alerts based on changed states. But the states never seem to be updated/indexed. So when I get to the changed-state block, state is nil. I can add the alerts within the splitp block, but that seems redundant. Maybe we will want different types of notifications between critical and warnings, but for now, I'd like to see this work (if possible).

(let [index (default :ttl 20 (index))]
    (where (not (state "expired"))
      (where (service "load/load/shortterm")
        (splitp < metric
          0.05 (with :state "critical" index)
          0.02 (with :state "warning" index)
               (with :state "ok" index))
      #(info %)
      (changed-state {:init "ok"}
        (stable 60 :state
          #(info "CHANGED STATE" %)
          (email "")))

Thanks for any help!

Riemann v0.2.9, collectd v5.5.0, OS CentOS 6.5

Source: (StackOverflow)

description event getting nil in riemann

I am trying to send email alert from logstash using Riemann. My email should get triggered on meeting certain criteria. I wrote Riemann config to send email alert for that I am sending certain events from logstash, I have hardcoded :description field but in my Riemann server I am seeing description as nil always. I don’t know where I am going wrong.

Riemann Config

(let [host ""]
  (tcp-server {:host host})
  (udp-server {:host host})
  (ws-server  {:host host}))

  ;Create index and print the values indexed
  (let [eindex (default :ttl 300 (update-index (index)))])

  ;Index event for reserve webservice failure
  (let [email (mailer{…….})]

    (where (service "e_log")
          (fn [events]
           (let [count-of-failures (count (filter #(re-find #"system space*" (:description %)) events))]        ;Calculate the count for matched value
                {:status "Failure"
                 :metric  count-of-failures 
                 :total-fail (>= count-of-failures 1)})))

          (where (and (= (:status event) "Failure")
                      (:total-fail event))

            (email ""))prn)))))

Logstash Config

             riemann_event => { "service" => "e_log"
"description" => "system space communication"
"metric" => "%{metric}"
"ttl" => "%{ttl}"                                                                                 

In my Riemann server I am seeing :description field as nil always so that :total-fail is false always.

Riemann Server

riemann.codec.Event{:host nil, :service nil, :state nil, :description nil, :metric 0, :tags nil, :time 1447406529, :ttl nil, :status "Failure", :total-fail false}
riemann.codec.Event{:host nil, :service nil, :state nil, :description nil, :metric 0, :tags nil, :time 1447406529, :ttl nil, :status "Failure", :total-fail false}
riemann.codec.Event{:host nil, :service nil, :state nil, :description nil, :metric 0, :tags nil, :time 1447406529, :ttl nil, :status "Failure", :total-fail false}
riemann.codec.Event{:host nil, :service nil, :state nil, :description nil, :metric 0, :tags nil, :time 1447406529, :ttl nil, :status "Failure", :total-fail false}
riemann.codec.Event{:host nil, :service nil, :state nil, :description nil, :metric 0, :tags nil, :time 1447406529, :ttl nil, :status "Failure", :total-fail false}

Thanks in advance

Source: (StackOverflow)

Need help in optimising clojure statement

I'm very new to clojure and need to set up a riemann config so that it would be easy to edit/add new conditions. What we have now:

(defn tell-ops 
    (by [:service]
        (throttle 3 360
            (rollup 2 360
                (email to)))))
([to channel]
    (by [:service]
        (throttle 3 360
            (rollup 2 360
                (slacker channel)
                (email to))))))


(where (state "FATAL")
    (where (service #"^Serv1")
            (tell-ops "" "#dev-ops1"))

    (where (service #"^Serv2")
            (tell-ops ""))


moreover, it lacks the default statement, something like if nothing matches, tell-ops ""

I think I need something like a top level struct

(def services 
 [{:regex #"^serv1" :mail "" :channel "#serv1"} 
  {:regex #"serv2$" :mail ""} ])

so that it would be easy to add new ones. But I have no idea how to loop throuth this array considering absence of :channel in the second case and making a "default call" if none of the regexes matches

Thanks in advance

Source: (StackOverflow)

Report CPU as always ok with Riemann

We're using Riemann and Riemann-health to monitor our servers. However now I get quite a lot of CPU critical warnings, because the CPU peaked for a very short time - This is nothing I even need to know about I think. From my understanding, a constant high CPU usage will increase the load avg, which will be reported as well and sounds way more useful.

I don't want to disable reporting the CPU, just every level should be considered to be ok. If possible, I'd like to change the events on the Riemann server, so I don't have to change all the servers.

Here our Riemann config:

Source: (StackOverflow)