Friday, October 18, 2013

Bing! Rewards Automation version 3.0

Check out the new version of Bing! Rewards Automation script - version 3.0

The main reason behind a new major version was the fact that I'd been marketing Bing! Rewards Automation script as a script which can be easily croned in true OS or scheduled on Windows to fully automate the usage, but I'd never done that myself. And I'd never done that myself because I didn't have a piece of hardware which I could use as a server. But now, as I have Raspberry Pi, which is definitely one of the best $45 investments, I've been automating everything around. Check out how to monitor a server and how to send automatic alerts.

I thought for a while how would I want the script to work and came up with an idea that it should do at least two things: retry if it fails (who knows what Bing! folks are doing, sometimes the right choice is a simple retry) and notify me (preferably by email) on a condition.

Events

This new version 3.0 features both retries and notifications both of which can optionally be conditional. Say, i.e. you definitely want to retry if the script fails requesting Bing!, or you know for sure that some account earns you 16 points a day, now you may set up a rule to retry (a number of times) if it doesn't earn you that much on some reason. And you may want to get notified. How? Your choice. The notification mechanism works through shell hooks. You specify a command which should be run and an condition condition which should be met. And you may put some specifiers in the command which have special meaning. The final command can send you an email or SMS or trigger some other behavior - send a command to your home robot over Bluetooth :)

Retries and notifications are part of events. Here is a complete list of events:

EventDescription
onErrorDefines what the script should do when an error occurs during processing an account. That can be any kind of error which results in generating result/error_*.html file (i.e. authorization, inexpected server response, etc.)
onCompleteDefines how the script should behave when it completes processing an account.
onScriptCompleteA special event which occurs only once when the script finishes executing.
onScriptFailureA special event which occurs only once and if the script fails with exception some time after successfully loading the config.

Example of events:

        <onError>
            <retry interval="5" salt="3.5" count="3" />
            <notify cmd="/home/BingRewards/bin/BingRewards/notify/onError.sh %a %p %r %P %l %i %e" />
        </onError>
        <onComplete>
            <retry if="%p lt 16" interval="5" salt="3.5" count="3" />
            <notify if="%l gt 3000"
                    cmd='/home/BingRewards/bin/BingRewards/notify/onComplete_reached.sh %a %p %r %P %l %i "reached %l Life Credits"' />
            <notify if="%p ne 16"
                    cmd="/home/BingRewards/bin/BingRewards/notify/onComplete.sh %a %p %r %P %l %i" />
            <notify if="%P gt 475"
                    cmd="/home/BingRewards/bin/BingRewards/notify/onComplete.sh %a %p %r %P %l %i" />
        </onComplete>

Here the script will retry up to 3 times on error sleeping anywhere between 5 and 8.5 seconds between retries. If an error occurres after the third retry, the Bing! Rewards Automation notifies calling onError.sh substituting the right values instead of the specifiers (%Letter). Once the script completes processing an account it will retry if a number of points earned for that account is less than 16. Afterwards it will notify if Lifetime Credits is more than 3000. It will notify if it earned not exactly 16 points. It will notify if total rewards points available on the account is greater than 475 - at the time of writing this post it was the price of $5 Amazon.com gift card on Gold account.

Check out Configuration for Bing! Rewards Automation script for the full manual on how to configure Bing! Rewards Automation script.

If you want to set up email alerts - check my post about how to do that.

This is an email I get from the script:

onComplete: account =Live_n...@hotmail.com, points = 0, retries = 3, totalPoints = 201, lifeCredits = 1831, ifStatement = 'p ne 16'
onComplete: account = Live_a...@hotmail.com, points = 0, retries = 3, totalPoints = 210, lifeCredits = 550, ifStatement = 'p ne 16'
onComplete: account = Live_s....@hotmail.com, points = 0, retries = 3, totalPoints = 208, lifeCredits = 1838, ifStatement = 'p ne 16'
onComplete: account = Facebook_n...@gmail.com, points = 0, retries = 3, totalPoints = 355, lifeCredits = 7120, ifStatement = 'p ne 31'

Example of cron job

0   3   *   *   *   /home/BingRewards/bin/main.py > /dev/null 2>&1
Run every morning at 3 am.

Etc.

One more thing was to move between queries sleep interval and between queries sleep salt from bingRewards.py to config.xml for easier configuration.

Project's folder structure was redesigned:

/main.py and config.xml
pkg/all the rest of the code
notify/notification scripts
test/unit tests
results/ -> result/script results and errors

Finally, a new command line option was added:
$ ./main.py --version
Bing! Rewards Automation script: 
Version: 3.0
See 'version.txt' for the list of changes
This code is published under LGPL v3 
There is NO WARRANTY, to the extent permitted by law.

Developed by: Sergey Markelov

Related posts

62 comments:

  1. i get an error:

    2013-10-19 21:12:56.8 - script started
    --------------------------------------------------------------------------------

    Traceback (most recent call last):
    File "./main.py", line 248, in
    config.parseFromFile(configFile)
    File "pkg/config.py", line 438, in parseFromFile
    self.__parse(root)
    File "pkg/config.py", line 413, in __parse
    for node in helpers.getXmlChildNodes(root):
    File "pkg/helpers.py", line 21, in getXmlChildNodes
    childNodes = xmlNode.getChildren()
    AttributeError: _ElementInterface instance has no attribute 'getChildren'

    ReplyDelete
    Replies
    1. Sorry, i meant to add a link to the config paste in the previous post;

      http://pastie.org/8415496

      Delete
    2. I've just pushed a hotfix. Check out Bing! Rewards Automation script version 3.0.1. That was a bug for Python < 2.7. I missed it previously because I use the latest version of Python 2 :) Anyways, that should work out. Just do

      git pull

      and try again. Tell me how it works out.

      Delete
    3. That seems to have fixed it, thanks for the quick reply.
      Keep up the great work, this is an awesome script. =)

      Delete
    4. Hi Sergey, outstanding application. I've had issues ever since switching to Microsoft's 2-step authentication. Even using a generated app password to use with the script I still am unable to have it successfully search. I just receive a failure notice with no debug or trace information to go on. Thanks

      Delete
    5. It works for many other people which makes me think that the solution to your problem should be relatively simple.

      Can you edit your config.xml and comment out or remove section _onScriptComplete_?
      Alternatively, you may comment out or remove section _events_ completely at which point the script would behave the same way to Bing Rewards Automation v.2.0.

      The reason you don't see a stack trace of an exception is that _onScriptComplete_ ignores everything, but the very last message. So, if something doesn't work, just try to comment the section and run the script again. There are relatively much fewer ways when the script just doesn't print anything at all which should be considered as bugs.

      Post an output here and we'll go from there.

      Delete
    6. Thanks in advance for the support. Here is what I am seeing.

      bingrewards@ubuntu:~/bin$ ./main.py
      2013-11-22 16:49:06.7 - script started
      --------------------------------------------------------------------------------


      For: Live - XxXxXxXxX@hotmail.com

      --------------------------------------------------------------------------------
      Traceback (most recent call last):
      File "./main.py", line 273, in
      EventsProcessor.onScriptFailure(config, e)
      File "./main.py", line 271, in
      __run(config)
      File "./main.py", line 195, in __run
      __processAccount(config, reportItem, account.password)
      File "./main.py", line 163, in __processAccount
      earnRewards(config, reportItem, accountPassword)
      File "./main.py", line 48, in earnRewards
      bingAuth.authenticate(reportItem.accountType, reportItem.accountLogin, password)
      File "./pkg/bingAuth.py", line 273, in authenticate
      authMethod(login, password)
      File "./pkg/bingAuth.py", line 226, in __authenticateLive
      s = page.index("<form ")
      ValueError: substring not found

      Delete
    7. When was the last time you did 'git pull'?
      Make sure that './main.py --version' shows version 3.1.1

      Delete
    8. Just did a fresh pull yesterday.

      bingrewards@ubuntu:~/bin$ ./main.py --version
      Bing! Rewards Automation script:
      Version: 3.1.1 from November 08, 2013
      See 'version.txt' for the list of changes
      This code is published under LGPL v3
      There is NO WARRANTY, to the extent permitted by law.

      Developed by: Sergey Markelov

      Delete
    9. I suggest you two things:

      1. Check that your password for the account in config.xml doesn't contain any characters which XML might not like;

      2. To understand the root cause if the first step doesn't work, you have to temporarily insert this on the line 225 (make sure that you have the same number of spaces as the line 226).

      filename = helpers.dumpErrorPage(page)
      print "THIS IS BAD: " + filename

      now run the script once again, locate THIS IS BAD in the output. Check the file which was generated. If you don't understand it, you may use http://pastebin.com/ and share it with me.

      Delete
    10. in #2 I was referring to pkg/bingAuth.py, line 225

      Delete
    11. So, the password is fine, the microsoft 2-step password generated is only alphas. Since this call is being made to the website URL, it is expecting my normal password + 2-step authentication.

      It appears that either the call to how the script logs us in would need to change or I will have to disable 2-step authentication. I will continue to think about another work around, but right now I don't see away.

      Delete
    12. any hope you can add support for 2-step authentication in the future? Thanks

      Delete
    13. Thanks for a proposal, but I don't think I will add a support any time soon. My suggestion is just create another account designated to earning points if you really have to have 2-step auth on that one. Or better create several accounts and put all of those in config.xml and start earning points.

      My point is to automate a routine. If there is a simpler workaround which has to be done only once, I don't want to automate it, because parsing pages (especially generated by MS's products) is a real PITA.

      Delete
    14. This is very true. I will be taking your suggestion and running with it. Thanks again for a great application

      Delete
  2. This comment has been removed by the author.

    ReplyDelete
  3. It stopped working for me. It logs in correctly, and it runs queries, but I don't get any points from them and they searches do not appear in my search history

    ReplyDelete
    Replies
    1. Yep, I've been witnessing the same for the last week or so. My guess is that Bing! expects some cookie or POST parameter which I don't pass. The thing is that it can be extremely hard to reverse engineer what information you need to pass in order to get an expected result, cause you treat the system as a black box. I will take a look at that if it doesn't cure itself, but I don't know when.

      Delete
    2. Thanks, I'm glad I'm not the only one with this problem. I figured that was the issue, and I have been trying to fix it, but haven't made much progress yet. I'll let you know if I find a solution.

      Delete
    3. That would be wonderful! Let me know how you progress and if you have any questions.

      Delete
    4. Did anyone find a fix for this problem? I am looking into it myself but I don't have much experience with python. Sergey, thanks for the great work!

      Delete
    5. Unfortunately, it really looks as something like an extra cookie or an extra header was introduced. I would love to investigate and fix that, but I don't have time :(

      Delete
    6. Hi Sergey,

      I know you have been busy but have you been able to take a look at this issue?

      Thanks,

      Delete
    7. No, unfortunately I didn't have time to look at that, but I discovered an interesting behavior.

      The thing is that if the script doesn't abuse Bing! (by processing the same account several times) then after some time (a week ?) it starts working again (provided that it hasn't been active for a week). That happens to me twice and now it earns the points again. Probably, the remedy is just to increase delays between two consequent searches (?) I'm gonna try that one out.

      Delete
    8. Thanks for keeping us updated! If there's anything I can do to help, just ask. I appreciate all the time and effort you put into this project for us.

      Delete
    9. During the investigation, I increased intervals in config.xml. It looks like Bing! black lists an IP if it abuses with frequent requests. With the increased intervals it's been working so far.

      Note that if an IP abuses Bing! it will take time for Bing! to remove the IP from the black list. So, if Bing! Rewards Automation doesn't work with the updated config.xml, don't run the script for a week or so and check back later.

      Version 3.1.2 has the updated values. Run git pull.

      Delete
    10. Thanks for the update. I've manually updated my config file and will test it tomorrow when the script is scheduled to run.

      Delete
    11. This comment has been removed by the author.

      Delete
  4. ~~Sorry to bug you again but as of today I am getting the "onScriptFailure" again. Is anyone else having any issues? Looking at my logs, it was working fine as of yesterday.~~

    Seems like /u/supere found the issue here https://github.com/sealemar/BingRewards/issues/13

    ReplyDelete
    Replies
    1. That's right. And the fix is out there. Check version 3.1.3 :)

      Delete
  5. I'm very unfamiliar with python, but I am having an issue. When I run the script it will only run for one account. I don't quite understand what's happening. Even when I run it with the config.xml that it came with it only runs hoho@hotmail.com.
    2014-01-29 12:34:51.4 - script started
    --------------------------------------------------------------------------------


    For: Live - hoho@hotmail.com

    --------------------------------------------------------------------------------

    Full log:
    ---------
    Empty log
    onScriptFailure: -mail

    ReplyDelete
    Replies
    1. Default config.xml is generic and should be treated as a template. You should populate accounts section with your accounts information which you want the script to earn reward points for. Whichever account you use - Facebook or Live - to earn points on Bing! you put login and password of that account under accounts and Bing! Rewards Automation will start earning points for you. Also take a look at http://sealemar.blogspot.com/2013/10/configuration-for-bing-rewards.html to find out more about config.xml

      Delete
    2. This is what I get when I run the script with two account in it. Ignore that it's not doing anything. It ran once and got the points so I know the script works, but it will not go to the next account in the list.

      2014-01-29 14:01:20.0 - script started
      --------------------------------------------------------------------------------


      Live - @gmail.com

      Points before: 51
      Points after: 51
      Points earned: 0
      Lifetime Credits: 51

      --------------------------------------------------------------------------------
      retry #2

      Live - @gmail.com

      Points before: 51
      Points after: 51
      Points earned: 0
      Lifetime Credits: 51

      --------------------------------------------------------------------------------
      retry #3

      Live - @gmail.com

      Points before: 51
      Points after: 51
      Points earned: 0
      Lifetime Credits: 51

      --------------------------------------------------------------------------------
      onComplete: account = Live_@gmail.com, points = 0, retries = 3, totalPoints = 51, lifeCredits = 51, ifStatement = 'p ne 16'

      Delete
    3. Note that I took out the email addy before the @gmail.com
      This is all for the same address

      Delete
    4. yeah, I see. Can you check that you're on the latest version of the script - 3.1.4 as of now. If you are, can you put some placeholder instead of your login part (leave @gmat.com or whatever) and a placeholder for the password and then post it on http://pastebin.com/ and give me a link to take a look at.

      Delete
    5. http://pastebin.com/Vm6SqQMd

      The comments are just from trying to get the code to work. I get the same results with or without them.

      Delete
    6. I will note that I had to change the #!/bin/sh to #!/bin/bash if that makes any difference. I was getting syntax errors before I changed it.

      Delete
    7. Let's move it offline. I opened an issue for you https://github.com/sealemar/BingRewards/issues/15

      Delete
  6. I am trying to make the script run searches on another account. This is what I get though, For: blabla@outlook.com
    The System Cannot Find the Path Specified.

    ReplyDelete
    Replies
    1. For some reason, the script just suddenly works. I do sometimes get a lot of frequent messages that say the system cannot find the path specified. How do I fix it so that I never see those messages again?

      Delete
    2. It happens probably due to the paths not set up correctly in the events.*.cmd section of your config.xml

      http://sealemar.blogspot.com/2013/10/configuration-for-bing-rewards.html

      Your issue is discussed here a little bit:

      https://github.com/sealemar/BingRewards/issues/18

      Delete
    3. So should I replace /home/BingRewards/bin/BingRewards with /mypathtothefiles ?

      Delete
    4. That works. But for some reason, once the script finishes with one account, it opens onComplete.sh. Is that supposed to happen?

      Delete
    5. http://sealemar.blogspot.com/2013/10/configuration-for-bing-rewards.html
      Section (optional) [configuration.events.onComplete]

      Delete
    6. Ahh, thanks! Sorry to ask you all of these questions. I should have noticed the OnScriptComplete section of the script.

      Delete
  7. And I forgot to ask, is there a limit to a number of accounts for 1 config file? I have tried adding more accounts in and I get errors.

    ReplyDelete
    Replies
    1. Let's take personal questions offline. Please, open a question on GitHub - https://github.com/sealemar/BingRewards/issues/new

      Delete
  8. Thanks for the script. Please add support for proxy server, because Bing Rewards is only supported from US IPs. Thanks

    ReplyDelete
  9. For some reason, my script is just doing the special offers to get rewards. It doesn't actually search things up so I only get about 1-2 points for each account I have.

    ReplyDelete
    Replies
    1. it's a known issue: https://github.com/sealemar/BingRewards/issues/17

      Delete
  10. Everybody who had issues with Bing! Rewards Automation Script not earning the points - check out the new version 3.2, it should resolve the issue.

    ReplyDelete
    Replies
    1. I think the default betweenQueriesInterval should be higher because some searches return no results. If I look in the error file, I see that I am required to enter in a captcha because Bing is getting suspicious. I am changing the interval to 15 seconds instead of 10.

      Delete
    2. That should be discussed in GitHub project issues. However in your case you may have too many accounts. Also make sure that other intervals are set up correctly:

      betweenQueriesInterval="10.0"
      betweenQueriesSalt="5.0"
      betweenAccountsInterval="300"
      betweenAccountsSalt="600"

      Finally, make sure that you have the latest version of script, which is 3.3.1 as of now

      Delete
    3. I have the latest version and my config file looks like that. I probably have too many accounts. Changing the interval to 15 did work however.

      Delete
  11. This is what I've got on one of my accounts right after the script completed:

    lifeCredits = 10101

    Binary ;)

    ReplyDelete
  12. Is today's 5th anniversary searches messing up the script? I couldn't seem to get them to search today.

    ReplyDelete
    Replies
    1. https://github.com/sealemar/BingRewards/issues/40

      Delete
  13. As of today the bot does not seem to be working anymore. Searches do not register.

    Also I found a few strange issues. you tell the user to create bingrewards, but the config file points to /home/BingRewards this causes issues. Also the notify lines look for bing/BingRewards/notify, but the git pull does not create that same structure.

    Great bot, just a lot of tweaking to get it working somewhat.

    ReplyDelete
  14. This was working perfectly for me yesterday, but as of today, when I try to run it, I get this:

    Traceback (most recent call last):
    File "./main.py", line 293, in
    config.parseFromFile(configFile)
    File "./pkg/config.py", line 519, in parseFromFile
    tree = et.parse(configFile)
    File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1182, in parse
    tree.parse(source, parser)
    File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 656, in parse
    parser.feed(data)
    File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1653, in feed
    self._raiseerror(v)
    File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1517, in _raiseerror
    raise err
    xml.etree.ElementTree.ParseError: mismatched tag: line 150, column 2


    I notice someone already posted asking about this, and was told it was a bug with Python and that a fix has been released with 3.0.1.

    I am running Python version 2.7.12.

    Any suggestions?

    ReplyDelete
  15. Tomorrow fails with error
    The server couldn't fulfill the request.
    Error code: 400

    ReplyDelete