Monitorix - a better way to monitor a server

Monitorix definitely is an easy and convenient way to monitor a server. Look at this pictures of how I monitor my Raspberry Pi:

It features an easy way to monitor daily, weekly, monthly and yearly periods.

Graphs include:

• System load
• Kernel usage
• Filesystem usage
• Disk I/O
• Network (global and per service/port. And I want to emphasize that - per service and per port - very cool)
• Users
• Sensors
• and a bunch more which I turned off :)

Every graph in there is clickable and gives you a bigger picture of one of your interest.

Why Monitorix is so cool?

Monirorix is cool because:

• It's free and opensource. I (personally) don't care much about the later, but some of us like that
• Once installed, it can be accessed from any web browser. I monitor my server even from phone.
• It's lightweight
• It doesn't require a web server and can be configured either to use a web-server or to run in a stand-alone mode
• It shows the information in graphs which are configurable and easy to read
• It can be configured to fire alarms on certain conditions (i.e. I use an alarm to send me an email once the System Load is above 75% for the last 15 minutes. The second alarm I use is when the free space on / reaches 75%)
• It has a built-in Basic Access Authentication which is a very handy to have if combined with the access from the Internet
• It can automatically send email reports (daily, weekly, monthly, early) with all the graphs in there

How does Monitorix work?

Monitorix is a bunch of Perl scripts. It can be installed either from the official web site or as a package on many modern Linux/Unix distributions.

It reads data from various places and saves it in database files. It reads frequently enough, though. And writes database files frequently enough as well. So, that's not good to keep on running on a hard drive or SSD all the time. The later wears off and the former is too slow to operate fast.

Here is the good news:

Monitorix causes too much of Disk I/O - I wanna optimize it

Anything-sync-daemon is your best friend there. A good fellow Graysky developed it for us and published on his GitHub page. Read up the documentation on Arch wiki - it is pretty straight forward. Few words - it works by pushing a desired directory into tmpfs, which is essentially, your RAM. That really sky rockets your Monitorix. It works orders of magnitude faster. And don't worry, your data survives reboots. When the daemon starts it copies a directory into the RAM and when it stops, it transfers the data back to the drive.

Too much RAM is consumed, I wanna optimize that

That is also achievable. Out of the box Monitorix eats more than 40Mb. While it may sound goofy for modern servers, it doesn't sound the same when you want to install it on a BeagelBone or Raspberry Pi like I did. After tuning my installation consumes only 19Mb.

To cut some memory off Monitorix which is never used on purpose back to the system you need to open your monitorix.conf:

# vim /etc/monitorix.conf

• Go to <graph_enable> section and turn everything you don't want to see to "n" state.
• For each of the sections which stood in "y" check man monitorix.conf or the official documentation and see if the section has a list associated with it, i.e. fs is a good example - the smaller the list, the smaller the resulting db file will be. Warning from man:

  WARNING: Every time the number of groups in this option changes, Monitorix will resize the fs.rrd file accordingly, removing all historical data.

Alerts

Alerts are implemented as hooks to shell commands. The end-point can be any, starting from sh, Python and ending wherever your imagination can bring you. Alerts are really easy to use and a good example is shown right here in Automated Email Alerts.

Finally

Finally, I'm giving away my monitorix.conf which I tuned for Raspberry Pi. Pay extra attention to alerts subsection under system and under fs which you most likely will want to change to your own hooks.

SSH - a life behind Screen

Screen - Screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shell.

This post is not only SSH related, but it can be very nicely utilized while working with the local terminal. It is good when you're on Windows host with Putty and don't want to create enormous amount of windows. Also this is very useful, if you don't want to keep your ssh session open while a big chunk of job is being executed.

Here is a picture of my screen session:

Some points to pay attention to:

The very last line belongs to screen. It says I have 5 sessions (or tabs, or windows, pick your name).

users:1(5)	1 user with 5 open sessions
uptime:2.0	2 days of uptime
mem:387	megabytes free
cpu:18.17	cpu load
Mon, 08/30	date and time (truncated because of the size of the window)

All what you see on the last line is configurable and can be set to show different values. Say, for example, you might want to see your wifi status, or the CPU temperature.

All you need to do to get this is to install screen, and insert this into your ~/.screenrc

1. # ~/.screenrc
2. altscreen on                # Return screen to original state when running
3.                             # ANSI programs like less and vi
5. # turn off startup message
6. startup_message off
8. # detach on hangup
9. autodetach on
11. # emulate .logout message
12. pow_detach_msg "Screen session of \$LOGNAME \$:cr:\$:nl:ended."
14. # Add stuff to xterm (and cousins) title bars.  This is a moderate abuse of the
15. # hardstatus feature--it just puts the hardstatus stuff into an xterm title
16. # bar.
18. setenv DISPLAY :0
19. setenv TERM screen-256color
21. backtick 1  10   0 /home/sealemar/etc/bin/screen/free.sh mem
22. backtick 2  10   0 /home/sealemar/etc/bin/screen/free.sh swap
23. backtick 3   7   7 /home/sealemar/etc/bin/screen/cpu_load.py
24. backtick 4 300   0 /home/sealemar/etc/bin/screen/uptime.sh
25. backtick 5  60   0 /home/sealemar/etc/bin/screen/date_time.sh
26. backtick 6  15   0 /home/sealemar/etc/bin/screen/users_logged_in.sh
27. #backtick 4  15  15 /home/sealemar/etc/bin/battery_life.pl --nocolor
28. #backtick 4  30  30 /home/sealemar/etc/bin/screen/wifi.sh
29. # Set the caption on the bottom line
30. caption always
31. #caption string "%{= kw}%-w%{= BW}%n %t%{-}%+w %-= wifi:%4` mem:%1` swap:%2` cpu:%3` - %LD %d %LM %Y - %c"
32. caption string "%{= kw}%-w%{= BW}%n %t%{-}%+w %-= users:%6` uptime:%4`d mem:%1` swap:%2` cpu:%3` - %5`"
34. # use xterm scrollback mechanism
35. info xterm* ti@:te@
37. # 256 color xterm
38. attrcolor b ".I"    # allow bold colors - necessary for some reason
40. # don't use visual bell
41. vbell off
42. # replace ctrl-A by ctrl-O
43. escape ^Oo
44. # set a big scrolling buffer
45. defscrollback 20000
47. hardcopydir ~/.cache/screen
49. # set how screen will highlight the text
50. sorendition =r
52. # fix the residual text
53. #altscreen on
55. bind ' ' windowlist -b
57. # resizing
58. bind = resize =
59. bind + resize +1
60. bind - resize -1
61. bind _ resize max
63. bind ^A eval "split" "focus" "next" # ^O-^A to split the window into a new region
64. bind ^X remove                      # ^O-^X to remove the current region
65. bindkey -k k5 focus                 # F5 to cycle through split regions
67. bindkey -k k6 prev          # F6 for previous window
68. bindkey -k k7 next          # F7 for next window
69. bindkey -k k8 copy          # F8 to enter scroll mode
71. # create some screens
72. screen -t booo1 1 bash
73. screen -t booo2 2 bash
74. screen -t booo3 3 bash
75. screen -t booo4 4 bash
76. screen -t Build 5 bash

Visible stuff is configured on lines 71-76 (sessions and their names which are created when screen start up) and lines 21-32 (counters, date time, etc.) I'm not going to elaborate on what every line does. The most interesting to tune, however, is backticks. It's obvious what it does from the documentation. Also, if you have any questions, don't hesitate to ask, I will do my best to answer everything.

Line #32 outputs a caption screen where user-defined backticks appear as a number between % and `, i.e "%6`"

The command below starts screen:

screen -dRR

which detaches screen session from wherever it was last opened from and attaches to it. If no screen session has been opened, a new one is created. Check man screen for a list of options, but that one above is the best I've found so far.

Here is a list of commands and shortcuts to use with the configuration above:

Command / Shortcut	Meaning
screen -dRR	Detaches existing screen session and attaches to it or starts a new session
F6	Previous tab
F7	Next tab
F8	Select text
<Ctrl+O><Ctrl+]>	Paste screen selection
<Ctrl+O><Ctrl+A>	Split window
<Ctrl+O><Ctrl+X>	Close split window
F5	Cycle through split windows

One big plus of "ssh through screen" over "ssh without screen" is that you may start a job which takes hours to execute and then you can close your ssh client without worrying that the job will terminate and your session will end, because it won't. When you feel like you want to check how it goes, you just connect from the same place, or a different one, it doesn't matter, and attach to the existing screen session - that simple. If you are in the middle of something and you have a lot of stuff open, but you have to go right now and you don't want to close everything. It's all right, you don't have to, just close your ssh session. Next time, reattach to the screen with "screen -dRR" - done - everything is where you have left it.

Backticks

I publish all of my backticks with a right free to use, but without any warranties. References are appreciated.

date_time.sh

1. /usr/bin/date +"%a, %D %H:%M

free.sh

1. #!/bin/bash
2. case "$1" in
3.     "mem")  free -m | grep buffers/cache | awk '{print $4}' ;;
4.     "swap") free -m | grep Swap | awk '{print $4}' ;;
5. esac

cpu_load.py

1. #!/usr/bin/env python2
2. #
3. # developed by Sergey Markelov (2013)
5. import os, sys, time
7. PIPE_WRITE_INTERVAL = 7
8. PIPE_FILENAME = "/dev/shm/cpu_load.pipe"
9. SINGLETON_FILE = "/dev/shm/cpu_load.py.singleton"
11. def getStat():
12.     with open("/proc/stat", "r") as fd:
13.         times = fd.readline().split()[1:5]
15.     return([int(t) for t in times])
17. def cpuLoad(interval):
18.     times1 = getStat()
19.     time.sleep(interval)
20.     times2 = getStat()
22.     for i in range(len(times1)):
23.         times2[i] -= times1[i]
25.     usage = times2[0] + times2[1] + times2[2]
26.     total = usage + times2[3]
28.     return(100 * float(usage) / total)
30. def writeData():
31. # ensure this is the only instance
32.     try:
33.         open(SINGLETON_FILE, "r").close()
34.     except IOError:
35.         open(SINGLETON_FILE, "w").close()
36.     else:
37.         sys.exit(2)
39.     try:
40.         while(True):
41.             load = cpuLoad(PIPE_WRITE_INTERVAL)
42.             with open(PIPE_FILENAME, "w") as fd:
43.                 fd.write(str(load))
45.     except KeyboardInterrupt:
46.         pass
47.     finally:
48.         os.remove(PIPE_FILENAME)
49.         os.remove(SINGLETON_FILE)
51. def readData():
52.     try:
53.         pipe = open(PIPE_FILENAME, "r")
54.     except IOError:
55.         print "N/A"
56.     else:
57.         print "%6.2f" % float(pipe.readline())
58.         pipe.close()
60. if __name__ == "__main__":
61.     try:
62.         if(len(sys.argv) < 2 or sys.argv[1] != "write"):
63.             readData()
64.         else:
65.             writeData()
67.     except Exception as e:
68.         print("An unexpected exception has occured: %s" % str(e))
69.         sys.exit(1)

uptime.sh

1. #!/bin/sh
3. # prints uptime as fraction of days
4. # Example:
5. # 3.5
6. # means uptime is 3.5 days
8. /usr/bin/awk '{printf "%0.1f\n", $1 / 3600 / 24}' /proc/uptime

users_logged_in.sh

1. #!/bin/sh
3. # prints
4. # total_users(total_sessions)
5. # Example:
6. # 2(5)
7. # means 2 users and 5 sessions in total
9. PATH=/usr/bin
11. users=`w -h | awk '{print $1}' | sort | uniq | wc -l`
12. sessions=`w -h | awk '{print $1}' | wc -l`
14. echo "${users}(${sessions})"