Thursday, December 18, 2014

Redmine plus S3 - Secure projects

Prelude

Many developers have been in a position where one way or another they were using a project management tool such as Microsoft TFS, JIRA or others. Some of them are better than others and many of them are paid or require certain components. Many of them are very hardly customizable and many more are not a good fit for a personal project or when the costs should stay low. The best option of all of the low cost alternatives is one which is free. And wide spread. And which has a big community.

Redmine

Redmine is a flexible project management web application. Written using the Ruby on Rails framework, it is cross-platform and cross-database. Redmine is open source and released under the terms of the GNU General Public License v2 (GPL).
Redmine is easy to use, easy to extend and easy to customize software. With a basic skills of CSS it takes very little time to give it a new appearance if you can not find anything suitable out of the plenty themes available for free on the Internet.

Dump

Every developer can recall at least some cases when he has done something and then a catastrophe has happened - he wrongfully ruined his work himself, the software glitch corrupted the data in the database, hardware glitch bricked the hard drive and the information was gone. All of those are examples of a disaster which can hardly be foreseen, but which always can be prevented. Dumping the information regularly will significantly lower if not absolutely remove all consequences of data corruption. Dumps can be local and remote. While local dumps will help you in a situation when the data is corrupted, the remote dumps are essential if the system goes down completely. The best way to dump data is an automated way.

S3

Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage. Amazon S3 is easy to use, with a simple web services interface to store and retrieve any amount of data from anywhere on the web. With Amazon S3, you pay only for the storage you actually use. There is no minimum fee and no setup cost.
S3 is the best option as of the end of 2014 to have you data preserved remotely at low or now cost at all.

Redmine-Dump

The idea is to dump data every day locally and send a weekly backup to S3. The best option is to create a new user specifically for backup reasons with a very restricted permissions.

S3 IAM User policy

From AWS IAM console create an individual IAM user and attach the following policy. You may want to change bucket_name and Version
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::bucket_name/dump/*"
    }
  ]
}
The Resource should be the place on S3 where you want the backups to go. That same resource path should appear in s3-dump.sh S3_DUMP_DIR variable

credentials.dump.sh

credentials.dump.sh is where you define your AWS credentials for the user you've created a few steps above.

$ cat ${HOME}/.config/aws/credentials.dump.sh
export AWS_ACCESS_KEY_ID="A...Q"
export AWS_SECRET_ACCESS_KEY="E...S"

Other things to override

  • common.sh REDMINE_DUMP_DIR - where you want to have your backups at
  • dump.sh REDMINE_HOME - where your Redmine is located
  • s3-dump.sh AWS_REGION - the AWS region you specified when you created an S3 bucket
  • s3-dump.sh S3_DUMP_DIR - where the backups should go to in S3

crontab

Cron is a daemon to have a job to start at a certain day and time.
The corresponding crontab can look like this
$ crontab -l
# 10:00 AM UTC every day - Dump Redmine
0 10 * * * /home/ec2-user/etc/bin/redmine/redmine-dump/dump.sh >> /home/ec2-user/dump/redmine/redmine-dump.log 2>&1

# 11:30 AM UTC every Sunday - Dump Redmine to S3
30 11 * * 0 /home/ec2-user/etc/bin/redmine/redmine-dump/s3-dump.sh >> /home/ec2-user/dump/redmine/redmine-dump.log 2>&1

Friday, August 1, 2014

Браво! downloader

It turned out I like Russian Rock'n'Roll band 'Браво'. You may check it out here: Браво! official website.
The guys made all of the songs available for a free download. The quality is not so good, but hey, it's free.
I also found myself very lazy on clicking each and every link to download all of the tracks. If you feel the same - here is a Python script for your convenience which I've just developed: download.py.
It will download all of the tracks and put them under appropriate folder structure: album/track###.mp3
If it ever stops / fails to fetch a file, you may rerun the script - it is smart enough to understand what files have been downloaded so far, provided that they were not moved and the folder structure hasn't changed.
Enjoy!

Wednesday, June 11, 2014

Troubleshooting Bing! Rewards Automation

Many people have issues with the script which are relatively easy to understand. Some of the examples include (1, 2, 3, 4, 5). The questions are very similar in nature and in most situations if something like this happens to you, you may save your time explaining me in vague form how "strangely the script works" and you may save my time pointing you to the same steps to troubleshoot the problem. This page is designed to troubleshoot the issue yourself as far as you can.

Steps to understand

  • What script version do you use?
    $ main.py --version
    
    Run
    $ git pull
  • To understand the retries you should start from the project page, where it says:
    *) Bing! Rewards Automation version 3.0 (added events support - notifications and retries) http://sealemar.blogspot.com/2013/10/bing-rewards-automation-version-30.html
    *) Configuration for Bing! Rewards Automation script http://sealemar.blogspot.com/2013/10/configuration-for-bing-rewards.html
  • If Bing! Rewards Automation Script prints no stack trace, but you have a very strong feeling that something exceptional happens, try to comment events section in your config.xml temporarily. That also may help if you see the message like this:
    For account xxx:
    
    And then goes nothing after that.
  • The script supports PC Searches and Mobile searches. To see if this happens for the mobile searches or the PC searches you should check the last corresponding message, it should say Running mobile searches or Running PC searches.
  • If the script emits:
    The System Cannot Find the Path Specified.
    
    Like in this case, that means you can resolve the issue yourself.
  • Note that if you abuse Bing! by doing your searches one after another with no delay, Bing! will ban you.

I don't get it

If after running the above steps you can't figure out what's happening, open an issue on the GitHub project page. The issue must include
  • The message:
    I have tried the troubleshooting guide and here is what I get
  • Bing! Rewards Automation Script version
  • Python version
  • OS type
  • On what type of searches that happens: PC / Mobile / All
  • Stack trace
  • Steps to reproduce