Thursday, December 18, 2014

Redmine plus S3 - Secure projects

Prelude

Many developers have been in a position where one way or another they were using a project management tool such as Microsoft TFS, JIRA or others. Some of them are better than others and many of them are paid or require certain components. Many of them are very hardly customizable and many more are not a good fit for a personal project or when the costs should stay low. The best option of all of the low cost alternatives is one which is free. And wide spread. And which has a big community.

Redmine

Redmine is a flexible project management web application. Written using the Ruby on Rails framework, it is cross-platform and cross-database. Redmine is open source and released under the terms of the GNU General Public License v2 (GPL).
Redmine is easy to use, easy to extend and easy to customize software. With a basic skills of CSS it takes very little time to give it a new appearance if you can not find anything suitable out of the plenty themes available for free on the Internet.

Dump

Every developer can recall at least some cases when he has done something and then a catastrophe has happened - he wrongfully ruined his work himself, the software glitch corrupted the data in the database, hardware glitch bricked the hard drive and the information was gone. All of those are examples of a disaster which can hardly be foreseen, but which always can be prevented. Dumping the information regularly will significantly lower if not absolutely remove all consequences of data corruption. Dumps can be local and remote. While local dumps will help you in a situation when the data is corrupted, the remote dumps are essential if the system goes down completely. The best way to dump data is an automated way.

S3

Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage. Amazon S3 is easy to use, with a simple web services interface to store and retrieve any amount of data from anywhere on the web. With Amazon S3, you pay only for the storage you actually use. There is no minimum fee and no setup cost.
S3 is the best option as of the end of 2014 to have you data preserved remotely at low or now cost at all.

Redmine-Dump

The idea is to dump data every day locally and send a weekly backup to S3. The best option is to create a new user specifically for backup reasons with a very restricted permissions.

S3 IAM User policy

From AWS IAM console create an individual IAM user and attach the following policy. You may want to change bucket_name and Version
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::bucket_name/dump/*"
    }
  ]
}
The Resource should be the place on S3 where you want the backups to go. That same resource path should appear in s3-dump.sh S3_DUMP_DIR variable

credentials.dump.sh

credentials.dump.sh is where you define your AWS credentials for the user you've created a few steps above.

$ cat ${HOME}/.config/aws/credentials.dump.sh
export AWS_ACCESS_KEY_ID="A...Q"
export AWS_SECRET_ACCESS_KEY="E...S"

Other things to override

  • common.sh REDMINE_DUMP_DIR - where you want to have your backups at
  • dump.sh REDMINE_HOME - where your Redmine is located
  • s3-dump.sh AWS_REGION - the AWS region you specified when you created an S3 bucket
  • s3-dump.sh S3_DUMP_DIR - where the backups should go to in S3

crontab

Cron is a daemon to have a job to start at a certain day and time.
The corresponding crontab can look like this
$ crontab -l
# 10:00 AM UTC every day - Dump Redmine
0 10 * * * /home/ec2-user/etc/bin/redmine/redmine-dump/dump.sh >> /home/ec2-user/dump/redmine/redmine-dump.log 2>&1

# 11:30 AM UTC every Sunday - Dump Redmine to S3
30 11 * * 0 /home/ec2-user/etc/bin/redmine/redmine-dump/s3-dump.sh >> /home/ec2-user/dump/redmine/redmine-dump.log 2>&1

Friday, August 1, 2014

Браво! downloader

It turned out I like Russian Rock'n'Roll band 'Браво'. You may check it out here: Браво! official website.
The guys made all of the songs available for a free download. The quality is not so good, but hey, it's free.
I also found myself very lazy on clicking each and every link to download all of the tracks. If you feel the same - here is a Python script for your convenience which I've just developed: download.py.
It will download all of the tracks and put them under appropriate folder structure: album/track###.mp3
If it ever stops / fails to fetch a file, you may rerun the script - it is smart enough to understand what files have been downloaded so far, provided that they were not moved and the folder structure hasn't changed.
Enjoy!

Sunday, February 2, 2014

BinaryClock events

In two of my previous posts I was telling about a feature of the BinaryClock which I referred to as events. This is a very important feature which makes BinaryClock personal. It's better to describe it with example.

Example:

Say, today is December 23, 2013. The next event from today would be Christmas which is celebrated on December 25. If you are at the default screen which displays time in binary (refer to the table on this page) and you press button #4 (show event) then you'll see a sliding message "Chrismas - Dec 25 2013 Thursday", now if you press button #2 (show event year info) it will slide "2014 years - started in 0".

Another example:

Your friend Bob is having birthday tomorrow. Today is April 2, 2014. If you press button #4 (show event), you'll see a sliding message "Bob's birthday - Apr 3 2014" if you press button #2 while the event message is sliding, you'll see "25 years - started in 1989" (if Bob's turning 25).

Events can be set in three ways.

Month and day

Example is Christmas which is always celebrated on December 25. To add an event of this type, go to lib/clock_event_personal.c and add
clock_event_initDayOfMonth (25, DECEMBER, 0, "Christmas")
clock_event_initDayOfMonth() is a macro from lib/clock_event.h. Here is its prototype:
// @brief Initializer for an event which is set with month and day
// @param day day of month
// @param month month of the event
// @param year year when the event first occurred
// @param name name of the event
//
#define clock_event_initDayOfMonth(day, month, year, name)

Month and week

Thanksgiving in the US is celebrated on the fourth Thursday of November. To add an event of that type, go to lib/clock_event.personal.c and add
clock_event_initDayOfWeek (THURSDAY, 3, WEEK_FROM_START, NOVEMBER, 1574, "Thanksgiving")
clock_event_initDayOfWeek() is a macro from lib/clock_event.h
// @brief Initializer for an event which is set with
// day of week, week of month, and month
// @param dayOfWeek day of week. See macros in lib/date_time.h
// @param weekOfMonth week of the month. [0..3]
// @param fromBeginningOfMonth whether the _weekOfMonth_ should be counted from the beginning of the month
//          This can be any number or TRUE / FALSE which can be treated in the boolean context
//
// @param month month of the event
// @param year year when the event first occurred
// @param name name of the event
//
#define clock_event_initDayOfWeek(dayOfWeek, weekOfMonth, fromBeginningOfMonth, month, year, name)
weekOfMonth is a number from 0 to 3. There are not more than 4 full weeks in a month. 0 corresponds to the first week, 3 to the fourth. fromBeginningOfMonth is a flag which can be set to TRUE or FALSE. It's easier to understand with example. Thanksgiving is the fourth Thursday of November. In other words it is the fourth thursday from the beginning of the month November. Another example is SysAdmin's day, which is the last Friday of July:
clock_event_initDayOfWeek (FRIDAY, 0, WEEK_FROM_END, JULY, 2000, "SysAdmin's Day")

Day of year

The last type of events is specified by day of year. Example is Programmer's day which is celebrated on the 256 day of a year.
clock_event_initDayOfYear (256, 2009, "Programmer's day")
clock_event_initDayOfYear() is a macro from lib/clock_event.h
// @brief Initializer for an event which is set with day of the year
// @param dayOfYear day of year
// @param year year when the event first occurred
// @param name name of the event
//
#define clock_event_initDayOfYear(dayOfYear, year, name)

Finally

Once you have all your events specified in lib/clock_event_personal.c, don't forget to change CLOCK_EVENTS_SIZE macro in lib/clock_event_personal.h to the size of the event's list. If you have 13 events, CLOCK_EVENTS_SIZE should be 13.

After you have all that ready, you may go to the project root and make the target (compile):
$ cd BinaryClock
$ make
If there is no errors, you may see the result in emulator. From the project root run
$ emulator/build/bin/terminal-binary-clock
Once you're satisfied with result, you can move on and upload the software to the real BinaryClock hardware. Plug FTDI to the BinaryClock and do
$ cd arduino
$ make upload

Saturday, February 1, 2014

BinaryClock Hardware

In the previous post I was talking about the software which runs on BinaryClock. Here I want to talk about the hardware. Here is how my version looks like. It doesn't mean if you want to assemble one you have to position all the parts in exact the same way.

BinaryClock default screen

Programming BinaryClock with FTDI cable

BinaryClock sliding decimal time
Schematics

Prototype video


You may find schematics and photos along with the source code of BinaryClock on my GitHub project page. The schematics and the photos are under doc/ folder. I hope those are pretty self-explanatory.

Instructions

The table below explains how push buttons control BinaryClock.

#ScreenDescriptionButtons behavior
1234
1 First Slides Welcome Message X (2) Show time in binary X X
2 Default Shows time in binary (3) Slide time in decimal (4) Set time (5) Show date in binary (6) Show events
3 Time in decimal Slides time from right to left (2) Show time in binary X (7) Slide date in decimal (6) Show events
4 Set time Set time in binary X Set next Decrease Increase
5 Date in binary Displays date in binary (7) Slide date in decimal (8) Set date (2) Show time (6) Show events
6 Show events Slides events from the next from today to the furthest from today (2) Show time (9) Show event year information Previous event Next event
7 Date in decimal Slides date from right to left (5) Show date in binary X (2) Show time (6) Show events
8 Set date Set date in binary X Set next Decrease Increase
9 Event year information Show event year information X Skip X X

Example: you are at screen #5 which displays date in binary. If you press the first button you will go to the screen #7 which slides date in binary. If you press the second button you will go to the screen #8 which sets the date. If you press the third button you will go to the screen #2 which displays time in binary. If you press the button number 4 you will go to the screen #6 which shows events. See the video above. More on events is on the next post.

How It Works

ATMEGA328p chip is what drives the BinaryClock. The chip runs the software. One 74HC595 (which is known as shift register) drives rows of the LED array, the other one drives columns. The output pins of the one which drives rows connect to the LED matrix through 260Ω resistors. 74HC595 which drives columns connects to LED array through ULN2803A which is also known as Darlington Transistor Array. Four push buttons are connected to ATMEGA328p which control the BinaryClock behavior.

Shift registers are there to reduce the number of pins used on ATMEGA328p. If two 74HC595 were not there, 16 pins on ATMEGA328p would be needed to achieve the same. Shift registers as the name suggests shift bits with each clock tick. The wiki will tell more about that. A good thing about shift registers is that they can be placed in series. So it doesn't matter how many shift registers you put in series, the same number of pins on the microcontroller can be used. Resistors are there to not over stress the LED matrix. If too much current goes through LED it burns brighter, but faster. Darlington array is there not to over stress the columns shift register. The current goes through the rows to the columns and further on. To light up an LED on the array the column needs to be grounded and the current should be applied to the row. You may notice that there are only 8 pins which drive rows and 8 pins which drive columns, so if you apply current to 8 rows and ground 8 columns, all the rows and columns will be lit. To display a pattern you will want to travel through the rows and display the columns or vise versa. More on the topic is here. BinaryClock software scans rows and uses walking pattern (as discussed on the last link) on columns. Thus at one point of time all the LEDs in a column may be lit and too much current will go to the columns pin. Darlington Array protects columns shift register from that over stress.

Finally in the top left corner you may notice legs. Those are legs to connect BinaryClock to FTDI which can be connected to a PC. In other words, that is the interface to program BinaryClock.

You may notice some other parts on the schematics, such as 16Mhz crystal oscillator, capacitors and a resistor on the first pin of ATMEGA328p. I'm not discussing those parts here. Please, refer to this page to better understand their purpose.

Optimizations & Improvements & Further thoughts

Maxim display driver may be used to reduce the number of parts significantly. If it was used instead, I wouldn't have two shift registers and Darlington Array on the board. And I would have only one resistor instead of eight (on rows) although it would be of a different value. Atmega chip has 32kb memory. It may seem like not a huge amount, but compilation of BinaryClock for it reports this:
AVR Memory Usage
----------------
Device: atmega328p

Program:   12306 bytes (37.6% Full)
(.text + .data + .bootloader)

Data:       1269 bytes (62.0% Full)
(.data + .bss + .noinit)
So, there are a lot of space left unused. You may want to extend the board with some sensors and then display the information from the sensors on the LED matrix.

Don't forget to edit lib/clock_event_personal.c with your personal events.

BinaryClock - a binary clock DIY

Idea

Here and there on the Internet you may see a geeky device called binary clock. The implementation can be different, but the idea behind is somewhat similar - it is an ordinary digital clock, but showing the time in binary format instead of decimal. After clicking the links you want one for yourself. Here I will tell you what you will need to create one with parts available at almost no cost on eBay.

There are a number of places where you can order a simple device which shows time. My idea was a clock which could show time, date and which could read decimal if I ever want to present that to somebody. Such a device wasn't out there, unfortunately, but looking retrospectively I'm positive it all was for the better because that led me to Arduino.

Arduino is an open-source electronics prototyping platform based on flexible, easy-to-use hardware and software. It's intended for artists, designers, hobbyists and anyone interested in creating interactive objects or environments.

Every Arduino device requires two things - the hardware and the software.

Functionality

The BinaryClock can:
  1. Show time in binary
  2. Slide time in decimal
  3. Show date in binary
  4. Slide date in decimal
  5. Set up time in binary
  6. Set up date in binary
  7. Slide user-defined events from the closest to the furthest
  8. Slide history information about an event such as when an event was first started and how many years the event is being celebrated
To better understand how it works, refer to the next post, where I discuss the hardware. Events are described here.

Software

Most of the time to write a program you don't have to have a device which it is intended to be used on. This code is no exception. Just by spending a little bit more time on design step a developer can get rid of a series of headaches later.

The source tree is organized in following way:

arduino/Arduino related code
doc/Documentation and schematic / PCB layout
emulator/PC emulator
include/A code which is common for arduino/emulator/lib/test
lib/BinaryClock library - the clock source
test/Tests
.ycm_extra_conf.pyI use YouCompleteMe which is a very handy plugin for ViM
LICENSEThe license
MakefileThe Makefile to build BinaryClock
project.vimA ViM project file - read on :)

The entire source code is published on my GitHub under Apache 2.0 license.

Tools I will need

This is a list of tools you will (or may) need to work with the code. This list will be described in more details further.
For the emulator and tests:
  • gcc / clang (at least one of these comes with every Unix based OS - Linux - gcc, MacOs - clang)
For the emulator:
  • ncurses (comes with every Unix based OS)
For the vimproject (optional - read on):
For the schematic:

Arduino

Under Arduino folder you will see only two files binary_clock.ino and Makefile.

binary_clock.ino - the code - is less than 200 lines long, half of which is comments. That's because all the functionality is hosted in lib/. This design makes easier testing and reusing the code (see emulator and test below). binary_clock.ino implements the code which is required by Arduino only, such as what pins the display sits on and what pins the buttons are on along with the glue code to bind the BinaryClock to the hardware.

Arduino website popularizes Arduino IDE. While it's ok for playing around, you may find yourself annoyed with the lack of functionality pretty fast. Luckily, there are ways to get around that. I personally like ViM. If you've never used ViM or Emacs - try one of them. Both have a very steep learning curve, but at the end of the day it worth every second. The next thing is Make. It can suit many needs. Just go read about it. To write a good Makefile from scratch is a challenge as the Make system has a very steep learning curve as well. And learning how to do it worth every second as well :). However, in many cases it's just easier to adopt an existing code, than to write your own, especially if it is well supported. And Makefile by Sudar / Martin Oldfield is a very good example of that. I took his Makefile as a basis.

Makefile under arduino folder of the project is definitions for Sudar's Makefile with some extra functionality. It has an error macro in the beginning which checks if Sudar's Makefile is installed. make will throw that error if it is missing.

To compile arduino code, go to arduino folder and run
$ cd BinaryClock/arduino
$ make
To upload the code to a device, run
$ make upload
If you use arduino as ISP programmer, run
$ make ispload
Note: if you don't have colorgcc installed, the Makefile may not work. If it doesn't, just comment two lines of code with "Override Arduino.mk defaults to point to my soft links" above them.

doc

Doc directory contains schematic which comes as a pdf, png and Fritzing project. And as far as the Fritzing project is concerned, there are two parts: FTDI Basic Male Header and Led Matrix LD1088-BS, which you may need to view the project. These parts are not needed for the pdf and the png, though.

emulator

As the name tells, the directory contains emulator related code. Like in the case of Arduino code, emulator is no more than the bindings and the glue code to work with the BinaryClock library (lib/).

Emulator uses ncurses library to emulate the clock in the terminal.


The emulator displays the time the same way it would be displayed on the real piece of hardware. The time follows the standard time representation on any electronic clock - from the left to the right one bar two pixels wide represents hours, minutes and seconds. And the one above reads 12:34:06. BinaryClock uses 24 hour format to eliminate ambiguity. Thus 12 is always 12pm. 12am would have 0 in the first column :)


One more example. On the emulator screen above it reads 23:59:55, which is 11:59pm.

To start the emulator, first you'll have to compile the code. From the project root do
$ make
This will compile everything - the library, Arduino code, the tests and the emulator. If you want to compile emulator only, do this:
$ cd BinaryClock/emulator
$ make
However it will still compile the library code, because it is the core of the BinaryClock. After the compiler has done its job, run
$ emulator/build/bin/terminal-binary-clock
From the project root. The emulator will show up. It will display a welcome message and then it will switch to the time screen. On the right you may notice numbers from 1 to 4 and the text further to the right after each of the numbers. The numbers are buttons. By pressing a corresponding button on the keyboard you execute a behavior associated with the button. On the real device, the buttons are simple push buttons. So, if you press 1, the emulator will show the time in text. BinaryClock has only 8x8 LEDs screen and the text is displayed by sliding it from right to left, the same way it slides on TV during the news hour.


The emulator reads the current time set on your PC when it starts, so you don't have to adjust it. On the real device the time most likely will need to be adjusted.

include

Include folder contains common functionality which is used from all other places. common.h and logger.h are included almost with every *.c unit. Makefile.include is included in the other Makefiles under lib, emulator, test.

lib

lib is a core of the BinaryClock. It is the code which BinaryClock runs. The modules are:

clockBinaryClock API
clock_alphabetCharacters which BinaryClock supports
clock_buttonFunctionality to work with the clock buttons
clock_eventEvents specific code
clock_event_personalThat's where you may set your own events which the clock will know of
clock_externA set of functions which need to be implemented by the user of the library. The pointers need to be assigned with the user-specific implementation
clock_mainThe only functionality to initialize and to update the clock state
clock_stateBinaryClock state object - the only object to pass around to the clock
clock_timeA mechanism to update BinaryClock time
date_timeInternal date-time code

The library is compiled into libclock.a which later can be referenced during the compilation of the binaries, such as emulator or test.

test

Contains a bunch of unit tests to test every external function from the lib. That proves that the BinaryClock would behave the way the developer wanted it to behave. test_ut is a light-weight test framework written especially for the project. It futures assert-like functionality which you may be familiar with from the modern test frameworks.

Run
$ cd BinaryClock
$make check
The example output will look like
--- main.c:48 (main) - a test suite started ---
Starting main.c:main --- [ut_clock]
--- ut_clock.c:252 (ut_clock) - a test suite started ---
Starting ut_clock.c:ut_clock --- [clock_slidePattern() returns correct result]
Starting ut_clock.c:ut_clock --- [clock_drawPattern() correct]
Starting ut_clock.c:ut_clock --- [clock_displayBinaryNumber() correct]
Starting ut_clock.c:ut_clock --- [clock_slideText() correct]
--- ut_clock.c:252 (ut_clock) - a test suite completed --- 4 test(s) ran. 4 succeded, 0 failed
Starting main.c:main --- [ut_clock_time]
--- ut_clock_time.c:76 (ut_clock_time) - a test suite started ---
Starting ut_clock_time.c:ut_clock_time --- [clock_updateUptimeMillis() returns correct delta]
Starting ut_clock_time.c:ut_clock_time --- [clock_updateUptimeMillis() handles integers overflows correctly]
--- ut_clock_time.c:76 (ut_clock_time) - a test suite completed --- 2 test(s) ran. 2 succeded, 0 failed
Starting main.c:main --- [ut_date_time]
...
 ...truncated...
...
Starting main.c:main --- [ut_clock_alphabet]
--- ut_clock_alphabet.c:67 (ut_clock_alphabet) - a test suite started ---
Starting ut_clock_alphabet.c:ut_clock_alphabet --- [clock_alphabet_getIndexByCharacter() correct]
Starting ut_clock_alphabet.c:ut_clock_alphabet --- [clock_alphabet_getIndexByCharacter() returns ERANGE and correct index]
--- ut_clock_alphabet.c:67 (ut_clock_alphabet) - a test suite completed --- 2 test(s) ran. 2 succeded, 0 failed
Starting main.c:main --- [ut_clock_event]
--- ut_clock_event.c:330 (ut_clock_event) - a test suite started ---
Starting ut_clock_event.c:ut_clock_event --- [clock_event_initDayOfMonth() is correct]
Starting ut_clock_event.c:ut_clock_event --- [clock_event_initDayOfWeek() is correct]
Starting ut_clock_event.c:ut_clock_event --- [clock_event_initDayOfYear() is correct]
Starting ut_clock_event.c:ut_clock_event --- [clock_event_getEventDetails() is correct]
Starting ut_clock_event.c:ut_clock_event --- [clock_event_initList() is correct]
Starting ut_clock_event.c:ut_clock_event --- [clock_event_updateList() is correct]
Starting ut_clock_event.c:ut_clock_event --- [clock_event_findClosestFromList() is correct]
Starting ut_clock_event.c:ut_clock_event --- [clock_event_yearInfoToStr() is correct]
--- ut_clock_event.c:330 (ut_clock_event) - a test suite completed --- 8 test(s) ran. 8 succeded, 0 failed
--- main.c:48 (main) - a test suite completed --- 6 test(s) ran. 6 succeded, 0 failed

project.vim

This you will only need if you are a ViM geek as I am.
$ alias gvimproject
alias gvimproject='gvim --cmd "source project.vim"'
I have the above alias in my environment and whenever I run gvimproject the shell starts an instance of GViM for me and preloads project.vim from the current folder.

This particular project.vim initializes path variable in ViM essentially telling it where the source code is located. So even if you work with the code under emulator folder and want to do gf (goto file) on a filename which refers to the code in lib, the Vim will know what you are trying to do. It also sets the appropriate ctags for you as you open up a *.c or an *.h file. It rebuilds the appropriate ctags file when you work with the code, so that when you press ctrl+] the tags are always fresh. And finally project.vim saves your current session when you leave Vim and automatically loads the last session when you start Vim. So, you don't have to remember which file was the last one that you worked with, what windows were opened, what line you were at, window size and position, etc.