smallseo.info

riemann

A network event stream processing system, in Clojure. Riemann - A network monitoring system

Riemann Context for Hadoop to send metrics to Riemann using metrics2 interface

Is there a library which can be integrated with the different Hadoop components (Namenode, datanode, jobtracker, tasktracker) as well as the Hadoop 2 components (Resource Manager) to send metrics to Riemann?


Source: (StackOverflow)

riemann.io add jar to classpath

I have written custom clojure functions that I want to use in my riemann configuration. I am using leiningen to build jar file (with dependencies) containing my functions. What is the right way to include this jar file in the classpath when starting riemann ?


Source: (StackOverflow)

Count riemann events in given time window

In riemann config for specific service I'm trying to assign to all its events metric=1, sum them within 5sec and send the result to influxdb.

I gave up with following:

(streams
  (where (service "offers")
    (fixed-time-window 5
      (smap folds/sum (with :metric 1 index))))
  influx)

it doesn't really work, events stored in influx do not match this rule. any hint?


Source: (StackOverflow)

How can I use clj-http in riemann.config

I use riemann and now I write my riemann.config.

I want to use clj-http post all events from riemann stream to my web server. But I don't know how to import clj-http from riemann.jar.

I code (:use clj-http.client) or (:require [clj-http.client :as client]) in riemann.config but got error:

java.lang.ClassNotFoundException: clj-http.client

Could anyone help me ?


Source: (StackOverflow)

Clojure Dashboard query

I am trying to show a graph on riemann-dashboard using query "pingDelay > 0" .

I already have indexed my data using following code

(let [index (index)]
  (defn write-dht-metric [e]
    (let [dhtstate (re-find #"dht_status: health\.(\S+), msg count (\d+) \((\d+) bytes\).*peak \{ping = (\d+)" (:pgmsg e))]
      (if (not= dhtstate nil)
        (do
          (prn "RESULT>" dhtstate)
          (index {:host "dht-info"
                  :service (:service e)
                  :time (unix-time)
                  :dhtStatus (get dhtstate 1)
                  :msgCount (get dhtstate 2)
                  :pingDelay (get dhtstate 3)}
            )
          )
        )
      )
    )
  )

However, I am not getting anything on graph. Earlier, I thought that perhaps its because my "pingDelay" is in string "12345", so, i also tried ":pingDelay #(Long. (get dhtstate 3))" without any success.

Can anyone please help me about what I must do to make it work?

Regards


Source: (StackOverflow)

Riemann Dashboard not outputting internal Riemann metrics

I'm new to Riemann and also new to ruby and Clojure as well. I'm trying to output the internal riemann events via (streams prn) in my riemann.config file. I currently see messages being printed out in the terminal from where I launched riemann.

Ex:

#riemann.codec.Event{:host "localhost.localdomain", :service "riemann server ws 0.0.0.0:5556 in latency 0.999", :state "ok", :description nil, :metric nil, :tags nil, :time 283857867047/200, :ttl 20}

In my dashboard however I'm unable to get these to print to any sort of log or gauge.

I tried using the following as a service =~ "riemann %" from here

I get an orange message displaying 1 socket errors; check the server field above or a large question market above the title.

Not sure what else to try or do from here to identify what went wrong.


Source: (StackOverflow)

ALerting in Riemann?

I am using ELK (logstash, ES, Kibana) stack for log analysis and Riemann for alerting. I have logs in which users is one of the fields parsed by logstash and I send the events to riemann from riemann output plugin.

Logstash parses logs and user is one of the field. Eg: logs parsed

Timestamp              user     command-name
 2014-06-07...         root      sh ./scripts/abc.sh
 2014-06-08...         sid       sh ./scripts/xyz.sh
 2014-06-08...         abc       sh ./scripts/xyz.sh
 2014-06-09...         root      sh ./scripts/xyz.sh

Logstash:

riemann {
    riemann_event => {
        "service"     => "logins"
        "unique_user" => "%{user}"
    }
}

So users values will be like: root, sid, abc, root, sid, def, etc....

So I split stream by user i.e one stream for each unique user. Now, I want to alert when number of unique users count go more than 3. I wrote the following but it's not achieving my purpose.

Riemann:

(streams

 (where (service "logins")
  (by :unique_user
    (moving-time-window 3600 
     (smap (fn [events]
      (let
        [users (count events)]
         (if (> users 3)
          (email "abc@gmail.com")       
     ))))))))

I am new to Riemann and clojure. Any help is appreciated.


Source: (StackOverflow)

clojure.lang.LazySeq cannot be cast to clojure.lang.IFn

I'm new to Riemann and Clojure. All I want to do is to send email notifications to three email groups when some service's TTL is expired. I created some sort of config file where I store a list of emails:

{
  :email_group_1 (
                  "first@example.com"
                  "second@example.ru"
                 )
  :email_group_2 (
                  "third@example.com"
                 )
}

My riemann config looks like this:

(logging/init {:console true})
(import org.apache.log4j.Level)
(logging/set-level Level/DEBUG)

(require '[clojure.java.io :as io])
(import '[java.io PushbackReader])

(let [host "0.0.0.0"]
  (tcp-server {:host host :port 60001})
  (udp-server {:host host})
  (ws-server  {:host host :port 60003}))
(repl-server  {:host "127.0.0.1"})

(def cwd (System/getProperty "user.dir"))

(def emails
  (with-open [r (io/reader (str cwd "/etc/emails.clj"))]
             (read (PushbackReader. r))))

(periodically-expire 5)

(def email (mailer))

(defn notify [& egroups]
  (for [egroup egroups]
    (rollup 1 60 (apply email (emails egroup)))))

(let [index (index)]
  (streams
    (default :ttl 60
      index

      (expired
          (where (service "service_connect_active")
                    #(info "expired" %)
                    (notify :email_group_1 :email_group_2))))))

Code looks good (for me), but when this service is expired I get the following error:

09:45:39 riemann.1      | INFO [2015-05-08 10:45:39,313] Thread-5 - riemann.config - expired {:ttl 60, :time 357766884827/250, :state expired, :service service_connect_active, :host ava.local}
09:45:39 riemann.1      | WARN [2015-05-08 10:45:39,319] Thread-5 - riemann.config - clojure.lang.LazySeq@841649b8 threw
09:45:39 riemann.1      | java.lang.ClassCastException: clojure.lang.LazySeq cannot be cast to clojure.lang.IFn
09:45:39 riemann.1      |   at riemann.config$eval66$stream__70$fn__75.invoke(riemann.development.config:34)
09:45:39 riemann.1      |   at riemann.config$eval66$stream__70.invoke(riemann.development.config:45)
09:45:39 riemann.1      |   at riemann.streams$match$stream__3514$fn__3525.invoke(streams.clj:1209)
09:45:39 riemann.1      |   at riemann.streams$match$stream__3514.invoke(streams.clj:1209)
09:45:39 riemann.1      |   at riemann.streams$default$stream__3731$fn__3742.invoke(streams.clj:1328)
09:45:39 riemann.1      |   at riemann.streams$default$stream__3731.invoke(streams.clj:1328)
09:45:39 riemann.1      |   at riemann.core$stream_BANG_$fn__4415.invoke(core.clj:19)
09:45:39 riemann.1      |   at riemann.core$stream_BANG_.invoke(core.clj:18)
09:45:39 riemann.1      |   at riemann.core$reaper$worker__4529$fn__4539.invoke(core.clj:303)
09:45:39 riemann.1      |   at riemann.core$reaper$worker__4529.invoke(core.clj:297)
09:45:39 riemann.1      |   at riemann.service.ThreadService$thread_service_runner__1973$fn__1974.invoke(service.clj:71)
09:45:39 riemann.1      |   at riemann.service.ThreadService$thread_service_runner__1973.invoke(service.clj:70)
09:45:39 riemann.1      |   at clojure.lang.AFn.run(AFn.java:22)
09:45:39 riemann.1      |   at java.lang.Thread.run(Thread.java:745)

Could someone please help me? Thanks.


Source: (StackOverflow)

Riemann - Build a stream dynamically from a map

I have the following function which gets a map with service name and threshold. It checks if the service crossed a defined threshold and then calls multiple downstream children on the event.

(defn tc
  [s & children]
   (where
     (and (service (:service_name s)) (not (expired? event)))
       (by [:host :service]
         (where (> metric (:threshold s)
           (with :state "critical" 
             (apply sdo children)))))))

I would like to build a stream dynamically using a vector of maps:

(def services [{:service "cpu/usage" :threshold 90}
               {:service "memory/usage" :threshold 90}])

When trying to run it in a stream i'm getting the following warning:

(streams
  (doseq [s services] (tc s prn)))

WARN [2015-01-05 14:27:07,187] Thread-15 - riemann.core - instrumentation service caught
java.lang.NullPointerException
  at riemann.core$stream_BANG_$fn__11140.invoke(core.clj:19)
  at riemann.core$stream_BANG_.invoke(core.clj:18)
  at riemann.core$instrumentation_service$measure__11149.invoke(core.clj:57)
  at riemann.service.ThreadService$thread_service_runner__8782$fn__8783.invoke(service.clj:66)
  at riemann.service.ThreadService$thread_service_runner__8782.invoke(service.clj:65)
  at clojure.lang.AFn.run(AFn.java:22)
  at java.lang.Thread.run(Thread.java:701)

It works, if i run the streams function inside the doseq. This one works and gives the following output:

(doseq [s services]
  (streams (tc s prn)))

#riemann.codec.Event{:host "testhost", :service "memory/usage", :state "critical", :description nil, :metric 91.0, :tags nil, :time 1420460856, :ttl 60.0}

Source: (StackOverflow)

Riemann: triggering alerts with changed-state

I'm new to Riemann and clojure. I'm trying to configure alerts based on changed states. But the states never seem to be updated/indexed. So when I get to the changed-state block, state is nil. I can add the alerts within the splitp block, but that seems redundant. Maybe we will want different types of notifications between critical and warnings, but for now, I'd like to see this work (if possible).

(let [index (default :ttl 20 (index))]
  (streams
    index
    (where (not (state "expired"))
      (where (service "load/load/shortterm")
        (splitp < metric
          0.05 (with :state "critical" index)
          0.02 (with :state "warning" index)
               (with :state "ok" index))
      )
      #(info %)
      (changed-state {:init "ok"}
        (stable 60 :state
          #(info "CHANGED STATE" %)
          (email "user@host.com")))
    )
  )
)

Thanks for any help!

Riemann v0.2.9, collectd v5.5.0, OS CentOS 6.5


Source: (StackOverflow)

description event getting nil in riemann

I am trying to send email alert from logstash using Riemann. My email should get triggered on meeting certain criteria. I wrote Riemann config to send email alert for that I am sending certain events from logstash, I have hardcoded :description field but in my Riemann server I am seeing description as nil always. I don’t know where I am going wrong.

Riemann Config

(let [host "127.0.0.1"]
  (tcp-server {:host host})
  (udp-server {:host host})
  (ws-server  {:host host}))

  ;Create index and print the values indexed
  (let [eindex (default :ttl 300 (update-index (index)))])


  ;Index event for reserve webservice failure
  (let [email (mailer{…….})]

  (streams
    (where (service "e_log")
      (fixed-time-window
        1 
        (smap
          (fn [events]
           (let [count-of-failures (count (filter #(re-find #"system space*" (:description %)) events))]        ;Calculate the count for matched value
               (event
                {:status "Failure"
                 :metric  count-of-failures 
                 :total-fail (>= count-of-failures 1)})))

          (where (and (= (:status event) "Failure")
                      (:total-fail event))

            (email "dfbnn@gmail.com"))prn)))))

Logstash Config

    riemann{
        host=>localhost
             riemann_event => { "service" => "e_log"
"description" => "system space communication"
"metric" => "%{metric}"
"ttl" => "%{ttl}"                                                                                 
                          }                                        
                    }

In my Riemann server I am seeing :description field as nil always so that :total-fail is false always.

Riemann Server

riemann.codec.Event{:host nil, :service nil, :state nil, :description nil, :metric 0, :tags nil, :time 1447406529, :ttl nil, :status "Failure", :total-fail false}
riemann.codec.Event{:host nil, :service nil, :state nil, :description nil, :metric 0, :tags nil, :time 1447406529, :ttl nil, :status "Failure", :total-fail false}
riemann.codec.Event{:host nil, :service nil, :state nil, :description nil, :metric 0, :tags nil, :time 1447406529, :ttl nil, :status "Failure", :total-fail false}
riemann.codec.Event{:host nil, :service nil, :state nil, :description nil, :metric 0, :tags nil, :time 1447406529, :ttl nil, :status "Failure", :total-fail false}
riemann.codec.Event{:host nil, :service nil, :state nil, :description nil, :metric 0, :tags nil, :time 1447406529, :ttl nil, :status "Failure", :total-fail false}

Thanks in advance


Source: (StackOverflow)

Need help in optimising clojure statement

I'm very new to clojure and need to set up a riemann config so that it would be easy to edit/add new conditions. What we have now:

(defn tell-ops 
([to]
    (by [:service]
        (throttle 3 360
            (rollup 2 360
                slackerDefault
                (email to)))))
([to channel]
    (by [:service]
        (throttle 3 360
            (rollup 2 360
                (slacker channel)
                (email to))))))

    ............

(where (state "FATAL")
    (where (service #"^Serv1")
            (tell-ops "dev.ops1@foo.com" "#dev-ops1"))

    (where (service #"^Serv2")
            (tell-ops "dev.ops2@bar.com"))
   ....

)

moreover, it lacks the default statement, something like if nothing matches, tell-ops "admin@fo.bar"

I think I need something like a top level struct

(def services 
 [{:regex #"^serv1" :mail "foo@bar.com" :channel "#serv1"} 
  {:regex #"serv2$" :mail "foo@baz.com"} ])

so that it would be easy to add new ones. But I have no idea how to loop throuth this array considering absence of :channel in the second case and making a "default call" if none of the regexes matches

Thanks in advance


Source: (StackOverflow)

Report CPU as always ok with Riemann

We're using Riemann and Riemann-health to monitor our servers. However now I get quite a lot of CPU critical warnings, because the CPU peaked for a very short time - This is nothing I even need to know about I think. From my understanding, a constant high CPU usage will increase the load avg, which will be reported as well and sounds way more useful.

I don't want to disable reporting the CPU, just every level should be considered to be ok. If possible, I'd like to change the events on the Riemann server, so I don't have to change all the servers.

Here our Riemann config: https://gist.github.com/iGEL/e352764a8c559440c851


Source: (StackOverflow)

Custom body message in riemann email

I am trying to create a custom message in the body section of email using riemann. I couldn't append the field dynamically.

Riemann config:

(let [email (mailer 
              {:host "XXXXX" :port XX :user "XXX" :pass "XXX" :auth "true"
               :subject (fn [events] "Team")
               :body (fn [events] 
                       (apply str "Hello Team, now the time is" (:timestamp event) "Thank You!"))
               :from "xxx@gmail.com"})]

My output:

Hello Team, now the time is Thank You!

My expected output:

Hello Team, now the time is 12:13:45 Thank You!.

My timestamp not getting appended in the :body.


Source: (StackOverflow)

riemann email exception with SMTP

I was able to set up riemann with SMTP. Riemann appears to send the email out when the condition is met but I observe an exception (copied below) - any insight on how to troubleshoot/fix this will be appreciated.

Config

    (def email (mailer {:host "xxx.xxx.xxx.xxx"
                        :port "xxxx"
                        :user "user@somewhere.com"
                        :pass "user12345"
                        :from "user@somewhere.com"}))

(streams
 (where (and (service #"^riemann netty execution-handler"))
        (email "user@somewhere.com")))

Error:

#riemann.codec.Event{:host "ubuntu-3", :service "riemann netty execution-handler threads active", :state "ok", :description nil, :metric 0, :tags nil, :time 348380111059/250, :ttl 20000}
WARN [2014-02-27 12:00:44,278] Thread-10 - riemann.config - riemann.email$mailer$make_stream__16773$stream__16774@4e9c33e9 threw
java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Number
        at postal.smtp$smtp_send_STAR_.invoke(smtp.clj:33)
        at postal.smtp$smtp_send.doInvoke(smtp.clj:58)
        at clojure.lang.RestFn.invoke(RestFn.java:423)
        at postal.core$send_message.invoke(core.clj:35)
        at riemann.email$email_event.invoke(email.clj:18)
        at riemann.email$mailer$make_stream__16773$stream__16774.invoke(email.clj:69)
        at riemann.config$eval40$stream__41$fn__46.invoke(riemann.config:44)
        at riemann.config$eval40$stream__41.invoke(riemann.config:44)
        at riemann.core$stream_BANG_$fn__10513.invoke(core.clj:19)
        at riemann.core$stream_BANG_.invoke(core.clj:18)
        at riemann.core$instrumentation_service$measure__10522.invoke(core.clj:56)
        at riemann.service.ThreadService$thread_service_runner__8329$fn__8330.invoke(service.clj:64)
        at riemann.service.ThreadService$thread_service_runner__8329.invoke(service.clj:63)
        at clojure.lang.AFn.run(AFn.java:24)
        at java.lang.Thread.run(Thread.java:744)

Source: (StackOverflow)