Thursday, December 18, 2014

Redmine plus S3 - Secure projects

Prelude

Many developers have been in a position where one way or another they were using a project management tool such as Microsoft TFS, JIRA or others. Some of them are better than others and many of them are paid or require certain components. Many of them are very hardly customizable and many more are not a good fit for a personal project or when the costs should stay low. The best option of all of the low cost alternatives is one which is free. And wide spread. And which has a big community.

Redmine

Redmine is a flexible project management web application. Written using the Ruby on Rails framework, it is cross-platform and cross-database. Redmine is open source and released under the terms of the GNU General Public License v2 (GPL).
Redmine is easy to use, easy to extend and easy to customize software. With a basic skills of CSS it takes very little time to give it a new appearance if you can not find anything suitable out of the plenty themes available for free on the Internet.

Dump

Every developer can recall at least some cases when he has done something and then a catastrophe has happened - he wrongfully ruined his work himself, the software glitch corrupted the data in the database, hardware glitch bricked the hard drive and the information was gone. All of those are examples of a disaster which can hardly be foreseen, but which always can be prevented. Dumping the information regularly will significantly lower if not absolutely remove all consequences of data corruption. Dumps can be local and remote. While local dumps will help you in a situation when the data is corrupted, the remote dumps are essential if the system goes down completely. The best way to dump data is an automated way.

S3

Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage. Amazon S3 is easy to use, with a simple web services interface to store and retrieve any amount of data from anywhere on the web. With Amazon S3, you pay only for the storage you actually use. There is no minimum fee and no setup cost.
S3 is the best option as of the end of 2014 to have you data preserved remotely at low or now cost at all.

Redmine-Dump

The idea is to dump data every day locally and send a weekly backup to S3. The best option is to create a new user specifically for backup reasons with a very restricted permissions.

S3 IAM User policy

From AWS IAM console create an individual IAM user and attach the following policy. You may want to change bucket_name and Version
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::bucket_name/dump/*"
    }
  ]
}
The Resource should be the place on S3 where you want the backups to go. That same resource path should appear in s3-dump.sh S3_DUMP_DIR variable

credentials.dump.sh

credentials.dump.sh is where you define your AWS credentials for the user you've created a few steps above.

$ cat ${HOME}/.config/aws/credentials.dump.sh
export AWS_ACCESS_KEY_ID="A...Q"
export AWS_SECRET_ACCESS_KEY="E...S"

Other things to override

  • common.sh REDMINE_DUMP_DIR - where you want to have your backups at
  • dump.sh REDMINE_HOME - where your Redmine is located
  • s3-dump.sh AWS_REGION - the AWS region you specified when you created an S3 bucket
  • s3-dump.sh S3_DUMP_DIR - where the backups should go to in S3

crontab

Cron is a daemon to have a job to start at a certain day and time.
The corresponding crontab can look like this
$ crontab -l
# 10:00 AM UTC every day - Dump Redmine
0 10 * * * /home/ec2-user/etc/bin/redmine/redmine-dump/dump.sh >> /home/ec2-user/dump/redmine/redmine-dump.log 2>&1

# 11:30 AM UTC every Sunday - Dump Redmine to S3
30 11 * * 0 /home/ec2-user/etc/bin/redmine/redmine-dump/s3-dump.sh >> /home/ec2-user/dump/redmine/redmine-dump.log 2>&1

Friday, August 1, 2014

Браво! downloader

It turned out I like Russian Rock'n'Roll band 'Браво'. You may check it out here: Браво! official website.
The guys made all of the songs available for a free download. The quality is not so good, but hey, it's free.
I also found myself very lazy on clicking each and every link to download all of the tracks. If you feel the same - here is a Python script for your convenience which I've just developed: download.py.
It will download all of the tracks and put them under appropriate folder structure: album/track###.mp3
If it ever stops / fails to fetch a file, you may rerun the script - it is smart enough to understand what files have been downloaded so far, provided that they were not moved and the folder structure hasn't changed.
Enjoy!

Sunday, February 2, 2014

BinaryClock events

In two of my previous posts I was telling about a feature of the BinaryClock which I referred to as events. This is a very important feature which makes BinaryClock personal. It's better to describe it with example.

Example:

Say, today is December 23, 2013. The next event from today would be Christmas which is celebrated on December 25. If you are at the default screen which displays time in binary (refer to the table on this page) and you press button #4 (show event) then you'll see a sliding message "Chrismas - Dec 25 2013 Thursday", now if you press button #2 (show event year info) it will slide "2014 years - started in 0".

Another example:

Your friend Bob is having birthday tomorrow. Today is April 2, 2014. If you press button #4 (show event), you'll see a sliding message "Bob's birthday - Apr 3 2014" if you press button #2 while the event message is sliding, you'll see "25 years - started in 1989" (if Bob's turning 25).

Events can be set in three ways.

Month and day

Example is Christmas which is always celebrated on December 25. To add an event of this type, go to lib/clock_event_personal.c and add
clock_event_initDayOfMonth (25, DECEMBER, 0, "Christmas")
clock_event_initDayOfMonth() is a macro from lib/clock_event.h. Here is its prototype:
// @brief Initializer for an event which is set with month and day
// @param day day of month
// @param month month of the event
// @param year year when the event first occurred
// @param name name of the event
//
#define clock_event_initDayOfMonth(day, month, year, name)

Month and week

Thanksgiving in the US is celebrated on the fourth Thursday of November. To add an event of that type, go to lib/clock_event.personal.c and add
clock_event_initDayOfWeek (THURSDAY, 3, WEEK_FROM_START, NOVEMBER, 1574, "Thanksgiving")
clock_event_initDayOfWeek() is a macro from lib/clock_event.h
// @brief Initializer for an event which is set with
// day of week, week of month, and month
// @param dayOfWeek day of week. See macros in lib/date_time.h
// @param weekOfMonth week of the month. [0..3]
// @param fromBeginningOfMonth whether the _weekOfMonth_ should be counted from the beginning of the month
//          This can be any number or TRUE / FALSE which can be treated in the boolean context
//
// @param month month of the event
// @param year year when the event first occurred
// @param name name of the event
//
#define clock_event_initDayOfWeek(dayOfWeek, weekOfMonth, fromBeginningOfMonth, month, year, name)
weekOfMonth is a number from 0 to 3. There are not more than 4 full weeks in a month. 0 corresponds to the first week, 3 to the fourth. fromBeginningOfMonth is a flag which can be set to TRUE or FALSE. It's easier to understand with example. Thanksgiving is the fourth Thursday of November. In other words it is the fourth thursday from the beginning of the month November. Another example is SysAdmin's day, which is the last Friday of July:
clock_event_initDayOfWeek (FRIDAY, 0, WEEK_FROM_END, JULY, 2000, "SysAdmin's Day")

Day of year

The last type of events is specified by day of year. Example is Programmer's day which is celebrated on the 256 day of a year.
clock_event_initDayOfYear (256, 2009, "Programmer's day")
clock_event_initDayOfYear() is a macro from lib/clock_event.h
// @brief Initializer for an event which is set with day of the year
// @param dayOfYear day of year
// @param year year when the event first occurred
// @param name name of the event
//
#define clock_event_initDayOfYear(dayOfYear, year, name)

Finally

Once you have all your events specified in lib/clock_event_personal.c, don't forget to change CLOCK_EVENTS_SIZE macro in lib/clock_event_personal.h to the size of the event's list. If you have 13 events, CLOCK_EVENTS_SIZE should be 13.

After you have all that ready, you may go to the project root and make the target (compile):
$ cd BinaryClock
$ make
If there is no errors, you may see the result in emulator. From the project root run
$ emulator/build/bin/terminal-binary-clock
Once you're satisfied with result, you can move on and upload the software to the real BinaryClock hardware. Plug FTDI to the BinaryClock and do
$ cd arduino
$ make upload

Saturday, February 1, 2014

BinaryClock Hardware

In the previous post I was talking about the software which runs on BinaryClock. Here I want to talk about the hardware. Here is how my version looks like. It doesn't mean if you want to assemble one you have to position all the parts in exact the same way.

BinaryClock default screen

Programming BinaryClock with FTDI cable

BinaryClock sliding decimal time
Schematics

Prototype video


You may find schematics and photos along with the source code of BinaryClock on my GitHub project page. The schematics and the photos are under doc/ folder. I hope those are pretty self-explanatory.

Instructions

The table below explains how push buttons control BinaryClock.

#ScreenDescriptionButtons behavior
1234
1 First Slides Welcome Message X (2) Show time in binary X X
2 Default Shows time in binary (3) Slide time in decimal (4) Set time (5) Show date in binary (6) Show events
3 Time in decimal Slides time from right to left (2) Show time in binary X (7) Slide date in decimal (6) Show events
4 Set time Set time in binary X Set next Decrease Increase
5 Date in binary Displays date in binary (7) Slide date in decimal (8) Set date (2) Show time (6) Show events
6 Show events Slides events from the next from today to the furthest from today (2) Show time (9) Show event year information Previous event Next event
7 Date in decimal Slides date from right to left (5) Show date in binary X (2) Show time (6) Show events
8 Set date Set date in binary X Set next Decrease Increase
9 Event year information Show event year information X Skip X X

Example: you are at screen #5 which displays date in binary. If you press the first button you will go to the screen #7 which slides date in binary. If you press the second button you will go to the screen #8 which sets the date. If you press the third button you will go to the screen #2 which displays time in binary. If you press the button number 4 you will go to the screen #6 which shows events. See the video above. More on events is on the next post.

How It Works

ATMEGA328p chip is what drives the BinaryClock. The chip runs the software. One 74HC595 (which is known as shift register) drives rows of the LED array, the other one drives columns. The output pins of the one which drives rows connect to the LED matrix through 260Ω resistors. 74HC595 which drives columns connects to LED array through ULN2803A which is also known as Darlington Transistor Array. Four push buttons are connected to ATMEGA328p which control the BinaryClock behavior.

Shift registers are there to reduce the number of pins used on ATMEGA328p. If two 74HC595 were not there, 16 pins on ATMEGA328p would be needed to achieve the same. Shift registers as the name suggests shift bits with each clock tick. The wiki will tell more about that. A good thing about shift registers is that they can be placed in series. So it doesn't matter how many shift registers you put in series, the same number of pins on the microcontroller can be used. Resistors are there to not over stress the LED matrix. If too much current goes through LED it burns brighter, but faster. Darlington array is there not to over stress the columns shift register. The current goes through the rows to the columns and further on. To light up an LED on the array the column needs to be grounded and the current should be applied to the row. You may notice that there are only 8 pins which drive rows and 8 pins which drive columns, so if you apply current to 8 rows and ground 8 columns, all the rows and columns will be lit. To display a pattern you will want to travel through the rows and display the columns or vise versa. More on the topic is here. BinaryClock software scans rows and uses walking pattern (as discussed on the last link) on columns. Thus at one point of time all the LEDs in a column may be lit and too much current will go to the columns pin. Darlington Array protects columns shift register from that over stress.

Finally in the top left corner you may notice legs. Those are legs to connect BinaryClock to FTDI which can be connected to a PC. In other words, that is the interface to program BinaryClock.

You may notice some other parts on the schematics, such as 16Mhz crystal oscillator, capacitors and a resistor on the first pin of ATMEGA328p. I'm not discussing those parts here. Please, refer to this page to better understand their purpose.

Optimizations & Improvements & Further thoughts

Maxim display driver may be used to reduce the number of parts significantly. If it was used instead, I wouldn't have two shift registers and Darlington Array on the board. And I would have only one resistor instead of eight (on rows) although it would be of a different value. Atmega chip has 32kb memory. It may seem like not a huge amount, but compilation of BinaryClock for it reports this:
AVR Memory Usage
----------------
Device: atmega328p

Program:   12306 bytes (37.6% Full)
(.text + .data + .bootloader)

Data:       1269 bytes (62.0% Full)
(.data + .bss + .noinit)
So, there are a lot of space left unused. You may want to extend the board with some sensors and then display the information from the sensors on the LED matrix.

Don't forget to edit lib/clock_event_personal.c with your personal events.

BinaryClock - a binary clock DIY

Idea

Here and there on the Internet you may see a geeky device called binary clock. The implementation can be different, but the idea behind is somewhat similar - it is an ordinary digital clock, but showing the time in binary format instead of decimal. After clicking the links you want one for yourself. Here I will tell you what you will need to create one with parts available at almost no cost on eBay.

There are a number of places where you can order a simple device which shows time. My idea was a clock which could show time, date and which could read decimal if I ever want to present that to somebody. Such a device wasn't out there, unfortunately, but looking retrospectively I'm positive it all was for the better because that led me to Arduino.

Arduino is an open-source electronics prototyping platform based on flexible, easy-to-use hardware and software. It's intended for artists, designers, hobbyists and anyone interested in creating interactive objects or environments.

Every Arduino device requires two things - the hardware and the software.

Functionality

The BinaryClock can:
  1. Show time in binary
  2. Slide time in decimal
  3. Show date in binary
  4. Slide date in decimal
  5. Set up time in binary
  6. Set up date in binary
  7. Slide user-defined events from the closest to the furthest
  8. Slide history information about an event such as when an event was first started and how many years the event is being celebrated
To better understand how it works, refer to the next post, where I discuss the hardware. Events are described here.

Software

Most of the time to write a program you don't have to have a device which it is intended to be used on. This code is no exception. Just by spending a little bit more time on design step a developer can get rid of a series of headaches later.

The source tree is organized in following way:

arduino/Arduino related code
doc/Documentation and schematic / PCB layout
emulator/PC emulator
include/A code which is common for arduino/emulator/lib/test
lib/BinaryClock library - the clock source
test/Tests
.ycm_extra_conf.pyI use YouCompleteMe which is a very handy plugin for ViM
LICENSEThe license
MakefileThe Makefile to build BinaryClock
project.vimA ViM project file - read on :)

The entire source code is published on my GitHub under Apache 2.0 license.

Tools I will need

This is a list of tools you will (or may) need to work with the code. This list will be described in more details further.
For the emulator and tests:
  • gcc / clang (at least one of these comes with every Unix based OS - Linux - gcc, MacOs - clang)
For the emulator:
  • ncurses (comes with every Unix based OS)
For the vimproject (optional - read on):
For the schematic:

Arduino

Under Arduino folder you will see only two files binary_clock.ino and Makefile.

binary_clock.ino - the code - is less than 200 lines long, half of which is comments. That's because all the functionality is hosted in lib/. This design makes easier testing and reusing the code (see emulator and test below). binary_clock.ino implements the code which is required by Arduino only, such as what pins the display sits on and what pins the buttons are on along with the glue code to bind the BinaryClock to the hardware.

Arduino website popularizes Arduino IDE. While it's ok for playing around, you may find yourself annoyed with the lack of functionality pretty fast. Luckily, there are ways to get around that. I personally like ViM. If you've never used ViM or Emacs - try one of them. Both have a very steep learning curve, but at the end of the day it worth every second. The next thing is Make. It can suit many needs. Just go read about it. To write a good Makefile from scratch is a challenge as the Make system has a very steep learning curve as well. And learning how to do it worth every second as well :). However, in many cases it's just easier to adopt an existing code, than to write your own, especially if it is well supported. And Makefile by Sudar / Martin Oldfield is a very good example of that. I took his Makefile as a basis.

Makefile under arduino folder of the project is definitions for Sudar's Makefile with some extra functionality. It has an error macro in the beginning which checks if Sudar's Makefile is installed. make will throw that error if it is missing.

To compile arduino code, go to arduino folder and run
$ cd BinaryClock/arduino
$ make
To upload the code to a device, run
$ make upload
If you use arduino as ISP programmer, run
$ make ispload
Note: if you don't have colorgcc installed, the Makefile may not work. If it doesn't, just comment two lines of code with "Override Arduino.mk defaults to point to my soft links" above them.

doc

Doc directory contains schematic which comes as a pdf, png and Fritzing project. And as far as the Fritzing project is concerned, there are two parts: FTDI Basic Male Header and Led Matrix LD1088-BS, which you may need to view the project. These parts are not needed for the pdf and the png, though.

emulator

As the name tells, the directory contains emulator related code. Like in the case of Arduino code, emulator is no more than the bindings and the glue code to work with the BinaryClock library (lib/).

Emulator uses ncurses library to emulate the clock in the terminal.


The emulator displays the time the same way it would be displayed on the real piece of hardware. The time follows the standard time representation on any electronic clock - from the left to the right one bar two pixels wide represents hours, minutes and seconds. And the one above reads 12:34:06. BinaryClock uses 24 hour format to eliminate ambiguity. Thus 12 is always 12pm. 12am would have 0 in the first column :)


One more example. On the emulator screen above it reads 23:59:55, which is 11:59pm.

To start the emulator, first you'll have to compile the code. From the project root do
$ make
This will compile everything - the library, Arduino code, the tests and the emulator. If you want to compile emulator only, do this:
$ cd BinaryClock/emulator
$ make
However it will still compile the library code, because it is the core of the BinaryClock. After the compiler has done its job, run
$ emulator/build/bin/terminal-binary-clock
From the project root. The emulator will show up. It will display a welcome message and then it will switch to the time screen. On the right you may notice numbers from 1 to 4 and the text further to the right after each of the numbers. The numbers are buttons. By pressing a corresponding button on the keyboard you execute a behavior associated with the button. On the real device, the buttons are simple push buttons. So, if you press 1, the emulator will show the time in text. BinaryClock has only 8x8 LEDs screen and the text is displayed by sliding it from right to left, the same way it slides on TV during the news hour.


The emulator reads the current time set on your PC when it starts, so you don't have to adjust it. On the real device the time most likely will need to be adjusted.

include

Include folder contains common functionality which is used from all other places. common.h and logger.h are included almost with every *.c unit. Makefile.include is included in the other Makefiles under lib, emulator, test.

lib

lib is a core of the BinaryClock. It is the code which BinaryClock runs. The modules are:

clockBinaryClock API
clock_alphabetCharacters which BinaryClock supports
clock_buttonFunctionality to work with the clock buttons
clock_eventEvents specific code
clock_event_personalThat's where you may set your own events which the clock will know of
clock_externA set of functions which need to be implemented by the user of the library. The pointers need to be assigned with the user-specific implementation
clock_mainThe only functionality to initialize and to update the clock state
clock_stateBinaryClock state object - the only object to pass around to the clock
clock_timeA mechanism to update BinaryClock time
date_timeInternal date-time code

The library is compiled into libclock.a which later can be referenced during the compilation of the binaries, such as emulator or test.

test

Contains a bunch of unit tests to test every external function from the lib. That proves that the BinaryClock would behave the way the developer wanted it to behave. test_ut is a light-weight test framework written especially for the project. It futures assert-like functionality which you may be familiar with from the modern test frameworks.

Run
$ cd BinaryClock
$make check
The example output will look like
--- main.c:48 (main) - a test suite started ---
Starting main.c:main --- [ut_clock]
--- ut_clock.c:252 (ut_clock) - a test suite started ---
Starting ut_clock.c:ut_clock --- [clock_slidePattern() returns correct result]
Starting ut_clock.c:ut_clock --- [clock_drawPattern() correct]
Starting ut_clock.c:ut_clock --- [clock_displayBinaryNumber() correct]
Starting ut_clock.c:ut_clock --- [clock_slideText() correct]
--- ut_clock.c:252 (ut_clock) - a test suite completed --- 4 test(s) ran. 4 succeded, 0 failed
Starting main.c:main --- [ut_clock_time]
--- ut_clock_time.c:76 (ut_clock_time) - a test suite started ---
Starting ut_clock_time.c:ut_clock_time --- [clock_updateUptimeMillis() returns correct delta]
Starting ut_clock_time.c:ut_clock_time --- [clock_updateUptimeMillis() handles integers overflows correctly]
--- ut_clock_time.c:76 (ut_clock_time) - a test suite completed --- 2 test(s) ran. 2 succeded, 0 failed
Starting main.c:main --- [ut_date_time]
...
 ...truncated...
...
Starting main.c:main --- [ut_clock_alphabet]
--- ut_clock_alphabet.c:67 (ut_clock_alphabet) - a test suite started ---
Starting ut_clock_alphabet.c:ut_clock_alphabet --- [clock_alphabet_getIndexByCharacter() correct]
Starting ut_clock_alphabet.c:ut_clock_alphabet --- [clock_alphabet_getIndexByCharacter() returns ERANGE and correct index]
--- ut_clock_alphabet.c:67 (ut_clock_alphabet) - a test suite completed --- 2 test(s) ran. 2 succeded, 0 failed
Starting main.c:main --- [ut_clock_event]
--- ut_clock_event.c:330 (ut_clock_event) - a test suite started ---
Starting ut_clock_event.c:ut_clock_event --- [clock_event_initDayOfMonth() is correct]
Starting ut_clock_event.c:ut_clock_event --- [clock_event_initDayOfWeek() is correct]
Starting ut_clock_event.c:ut_clock_event --- [clock_event_initDayOfYear() is correct]
Starting ut_clock_event.c:ut_clock_event --- [clock_event_getEventDetails() is correct]
Starting ut_clock_event.c:ut_clock_event --- [clock_event_initList() is correct]
Starting ut_clock_event.c:ut_clock_event --- [clock_event_updateList() is correct]
Starting ut_clock_event.c:ut_clock_event --- [clock_event_findClosestFromList() is correct]
Starting ut_clock_event.c:ut_clock_event --- [clock_event_yearInfoToStr() is correct]
--- ut_clock_event.c:330 (ut_clock_event) - a test suite completed --- 8 test(s) ran. 8 succeded, 0 failed
--- main.c:48 (main) - a test suite completed --- 6 test(s) ran. 6 succeded, 0 failed

project.vim

This you will only need if you are a ViM geek as I am.
$ alias gvimproject
alias gvimproject='gvim --cmd "source project.vim"'
I have the above alias in my environment and whenever I run gvimproject the shell starts an instance of GViM for me and preloads project.vim from the current folder.

This particular project.vim initializes path variable in ViM essentially telling it where the source code is located. So even if you work with the code under emulator folder and want to do gf (goto file) on a filename which refers to the code in lib, the Vim will know what you are trying to do. It also sets the appropriate ctags for you as you open up a *.c or an *.h file. It rebuilds the appropriate ctags file when you work with the code, so that when you press ctrl+] the tags are always fresh. And finally project.vim saves your current session when you leave Vim and automatically loads the last session when you start Vim. So, you don't have to remember which file was the last one that you worked with, what windows were opened, what line you were at, window size and position, etc.

Saturday, October 12, 2013

Monitorix - a better way to monitor a server

Monitorix definitely is an easy and convenient way to monitor a server. Look at this pictures of how I monitor my Raspberry Pi:





It features an easy way to monitor daily, weekly, monthly and yearly periods.

Graphs include:
  • System load
  • Kernel usage
  • Filesystem usage
  • Disk I/O
  • Network (global and per service/port. And I want to emphasize that - per service and per port - very cool)
  • Users
  • Sensors
  • and a bunch more which I turned off :)
Every graph in there is clickable and gives you a bigger picture of one of your interest.

Why Monitorix is so cool?

Monirorix is cool because:
  • It's free and opensource. I (personally) don't care much about the later, but some of us like that
  • Once installed, it can be accessed from any web browser. I monitor my server even from phone.
  • It's lightweight
    • It doesn't require a web server and can be configured either to use a web-server or to run in a stand-alone mode
  • It shows the information in graphs which are configurable and easy to read
  • It can be configured to fire alarms on certain conditions (i.e. I use an alarm to send me an email once the System Load is above 75% for the last 15 minutes. The second alarm I use is when the free space on / reaches 75%)
  • It has a built-in Basic Access Authentication which is a very handy to have if combined with the access from the Internet
  • It can automatically send email reports (daily, weekly, monthly, early) with all the graphs in there

How does Monitorix work?

Monitorix is a bunch of Perl scripts. It can be installed either from the official web site or as a package on many modern Linux/Unix distributions.
It reads data from various places and saves it in database files. It reads frequently enough, though. And writes database files frequently enough as well. So, that's not good to keep on running on a hard drive or SSD all the time. The later wears off and the former is too slow to operate fast.
Here is the good news:

Monitorix causes too much of Disk I/O - I wanna optimize it

Anything-sync-daemon is your best friend there. A good fellow Graysky developed it for us and published on his GitHub page. Read up the documentation on Arch wiki - it is pretty straight forward. Few words - it works by pushing a desired directory into tmpfs, which is essentially, your RAM. That really sky rockets your Monitorix. It works orders of magnitude faster. And don't worry, your data survives reboots. When the daemon starts it copies a directory into the RAM and when it stops, it transfers the data back to the drive.

Too much RAM is consumed, I wanna optimize that

That is also achievable. Out of the box Monitorix eats more than 40Mb. While it may sound goofy for modern servers, it doesn't sound the same when you want to install it on a BeagelBone or Raspberry Pi like I did. After tuning my installation consumes only 19Mb.

To cut some memory off Monitorix which is never used on purpose back to the system you need to open your monitorix.conf:
# vim /etc/monitorix.conf
  • Go to <graph_enable> section and turn everything you don't want to see to "n" state.
  • For each of the sections which stood in "y" check man monitorix.conf or the official documentation and see if the section has a list associated with it, i.e. fs is a good example - the smaller the list, the smaller the resulting db file will be. Warning from man:
    WARNING: Every time the number of groups in this option changes, Monitorix will resize the fs.rrd file accordingly, removing all historical data.

Alerts

Alerts are implemented as hooks to shell commands. The end-point can be any, starting from sh, Python and ending wherever your imagination can bring you. Alerts are really easy to use and a good example is shown right here in Automated Email Alerts.

Finally

Finally, I'm giving away my monitorix.conf which I tuned for Raspberry Pi. Pay extra attention to alerts subsection under system and under fs which you most likely will want to change to your own hooks.

Wednesday, October 2, 2013

SSH - a life behind Screen

Screen - Screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shell.

This post is not only SSH related, but it can be very nicely utilized while working with the local terminal. It is good when you're on Windows host with Putty and don't want to create enormous amount of windows. Also this is very useful, if you don't want to keep your ssh session open while a big chunk of job is being executed.

Here is a picture of my screen session:


Some points to pay attention to:

The very last line belongs to screen. It says I have 5 sessions (or tabs, or windows, pick your name).

users:1(5)1 user with 5 open sessions
uptime:2.02 days of uptime
mem:387megabytes free
cpu:18.17cpu load
Mon, 08/30date and time (truncated because of the size of the window)

All what you see on the last line is configurable and can be set to show different values. Say, for example, you might want to see your wifi status, or the CPU temperature.

All you need to do to get this is to install screen, and insert this into your ~/.screenrc


  1. # ~/.screenrc
  2. altscreen on                # Return screen to original state when running
  3.                             # ANSI programs like less and vi
  4. # turn off startup message
  5. startup_message off
  6. # detach on hangup
  7. autodetach on
  8. # emulate .logout message
  9. pow_detach_msg "Screen session of \$LOGNAME \$:cr:\$:nl:ended."
  10. # Add stuff to xterm (and cousins) title bars.  This is a moderate abuse of the
  11. # hardstatus feature--it just puts the hardstatus stuff into an xterm title
  12. # bar.
  13. setenv DISPLAY :0
  14. setenv TERM screen-256color
  15. backtick 1  10   0 /home/sealemar/etc/bin/screen/free.sh mem
  16. backtick 2  10   0 /home/sealemar/etc/bin/screen/free.sh swap
  17. backtick 3   7   7 /home/sealemar/etc/bin/screen/cpu_load.py
  18. backtick 4 300   0 /home/sealemar/etc/bin/screen/uptime.sh
  19. backtick 5  60   0 /home/sealemar/etc/bin/screen/date_time.sh
  20. backtick 6  15   0 /home/sealemar/etc/bin/screen/users_logged_in.sh
  21. #backtick 4  15  15 /home/sealemar/etc/bin/battery_life.pl --nocolor
  22. #backtick 4  30  30 /home/sealemar/etc/bin/screen/wifi.sh
  23. # Set the caption on the bottom line
  24. caption always
  25. #caption string "%{= kw}%-w%{= BW}%n %t%{-}%+w %-= wifi:%4` mem:%1` swap:%2` cpu:%3` - %LD %d %LM %Y - %c"
  26. caption string "%{= kw}%-w%{= BW}%n %t%{-}%+w %-= users:%6` uptime:%4`d mem:%1` swap:%2` cpu:%3` - %5`"
  27. # use xterm scrollback mechanism
  28. info xterm* ti@:te@
  29. # 256 color xterm
  30. attrcolor b ".I"    # allow bold colors - necessary for some reason
  31. # don't use visual bell
  32. vbell off
  33. # replace ctrl-A by ctrl-O
  34. escape ^Oo
  35. # set a big scrolling buffer
  36. defscrollback 20000
  37. hardcopydir ~/.cache/screen
  38. # set how screen will highlight the text
  39. sorendition =r
  40. # fix the residual text
  41. #altscreen on
  42. bind ' ' windowlist -b
  43. # resizing
  44. bind = resize =
  45. bind + resize +1
  46. bind - resize -1
  47. bind _ resize max
  48. bind ^A eval "split" "focus" "next" # ^O-^A to split the window into a new region
  49. bind ^X remove                      # ^O-^X to remove the current region
  50. bindkey -k k5 focus                 # F5 to cycle through split regions
  51. bindkey -k k6 prev          # F6 for previous window
  52. bindkey -k k7 next          # F7 for next window
  53. bindkey -k k8 copy          # F8 to enter scroll mode
  54. # create some screens
  55. screen -t booo1 1 bash
  56. screen -t booo2 2 bash
  57. screen -t booo3 3 bash
  58. screen -t booo4 4 bash
  59. screen -t Build 5 bash

Visible stuff is configured on lines 71-76 (sessions and their names which are created when screen start up) and lines 21-32 (counters, date time, etc.) I'm not going to elaborate on what every line does. The most interesting to tune, however, is backticks. It's obvious what it does from the documentation. Also, if you have any questions, don't hesitate to ask, I will do my best to answer everything.

Line #32 outputs a caption screen where user-defined backticks appear as a number between % and `, i.e "%6`"

The command below starts screen:
screen -dRR
which detaches screen session from wherever it was last opened from and attaches to it. If no screen session has been opened, a new one is created. Check man screen for a list of options, but that one above is the best I've found so far.

Here is a list of commands and shortcuts to use with the configuration above:

Command / ShortcutMeaning
screen -dRRDetaches existing screen session and attaches to it or starts a new session
F6Previous tab
F7Next tab
F8Select text
<Ctrl+O><Ctrl+]>Paste screen selection
<Ctrl+O><Ctrl+A>Split window
<Ctrl+O><Ctrl+X>Close split window
F5Cycle through split windows

One big plus of "ssh through screen" over "ssh without screen" is that you may start a job which takes hours to execute and then you can close your ssh client without worrying that the job will terminate and your session will end, because it won't. When you feel like you want to check how it goes, you just connect from the same place, or a different one, it doesn't matter, and attach to the existing screen session - that simple. If you are in the middle of something and you have a lot of stuff open, but you have to go right now and you don't want to close everything. It's all right, you don't have to, just close your ssh session. Next time, reattach to the screen with "screen -dRR" - done - everything is where you have left it.

Backticks

I publish all of my backticks with a right free to use, but without any warranties. References are appreciated.

date_time.sh


  1. /usr/bin/date +"%a, %D %H:%M

free.sh

  1. #!/bin/bash
  2. case "$1" in
  3.     "mem")  free -m | grep buffers/cache | awk '{print $4}' ;;
  4.     "swap") free -m | grep Swap | awk '{print $4}' ;;
  5. esac

cpu_load.py

  1. #!/usr/bin/env python2
  2. #
  3. # developed by Sergey Markelov (2013)
  4. import os, sys, time
  5. PIPE_WRITE_INTERVAL = 7
  6. PIPE_FILENAME = "/dev/shm/cpu_load.pipe"
  7. SINGLETON_FILE = "/dev/shm/cpu_load.py.singleton"
  8. def getStat():
  9.     with open("/proc/stat", "r") as fd:
  10.         times = fd.readline().split()[1:5]
  11.     return([int(t) for t in times])
  12. def cpuLoad(interval):
  13.     times1 = getStat()
  14.     time.sleep(interval)
  15.     times2 = getStat()
  16.     for i in range(len(times1)):
  17.         times2[i] -= times1[i]
  18.     usage = times2[0] + times2[1] + times2[2]
  19.     total = usage + times2[3]
  20.     return(100 * float(usage) / total)
  21. def writeData():
  22. # ensure this is the only instance
  23.     try:
  24.         open(SINGLETON_FILE, "r").close()
  25.     except IOError:
  26.         open(SINGLETON_FILE, "w").close()
  27.     else:
  28.         sys.exit(2)
  29.     try:
  30.         while(True):
  31.             load = cpuLoad(PIPE_WRITE_INTERVAL)
  32.             with open(PIPE_FILENAME, "w") as fd:
  33.                 fd.write(str(load))
  34.     except KeyboardInterrupt:
  35.         pass
  36.     finally:
  37.         os.remove(PIPE_FILENAME)
  38.         os.remove(SINGLETON_FILE)
  39. def readData():
  40.     try:
  41.         pipe = open(PIPE_FILENAME, "r")
  42.     except IOError:
  43.         print "N/A"
  44.     else:
  45.         print "%6.2f" % float(pipe.readline())
  46.         pipe.close()
  47. if __name__ == "__main__":
  48.     try:
  49.         if(len(sys.argv) < 2 or sys.argv[1] != "write"):
  50.             readData()
  51.         else:
  52.             writeData()
  53.     except Exception as e:
  54.         print("An unexpected exception has occured: %s" % str(e))
  55.         sys.exit(1)

uptime.sh


  1. #!/bin/sh
  2. # prints uptime as fraction of days
  3. # Example:
  4. # 3.5
  5. # means uptime is 3.5 days
  6. /usr/bin/awk '{printf "%0.1f\n", $1 / 3600 / 24}' /proc/uptime

users_logged_in.sh


  1. #!/bin/sh
  2. # prints
  3. # total_users(total_sessions)
  4. # Example:
  5. # 2(5)
  6. # means 2 users and 5 sessions in total
  7. PATH=/usr/bin
  8. users=`w -h | awk '{print $1}' | sort | uniq | wc -l`
  9. sessions=`w -h | awk '{print $1}' | wc -l`
  10. echo "${users}(${sessions})"