Wednesday, July 9, 2014

RHM picks for classes to take at the Amazon Summit NY

Right Handed Monkey is going to be at the New York Amazon Summit 7.10.2014!

I'm very excited to spend this time getting to know more about the Amazon platform and meet other startups and entrepreneurs in the area.  Let's do this, people.  Leave a comment if you're interested in meeting up at one of the social events.

My picks for the Amazon Summit 2014:


1. Deploy, Manage, and Scale Your Apps with OpsWorks and Elastic Beanstalk


Why: I picked this class because I want to learn about scaling my current system and what tools are available to make it grow.  Plus I want to make sure when I am programming a site that I do it in a way that is compatible with scaling (ie. Beanstalk.).

2. Scaling on AWS for the First 10 Million Users


Why: This is a given.  Everyone wants to know how to scale up their software so it can support as many users as possible.

3. Updating Security Operations for the Cloud, presented by Trend Micro


Why: Originally I considered 'Managing an Enterprise Class Hybrid Architecture', but the truth is I'm not managing mega-sized legacy infrastructures and won't be anytime soon.  My sites are lean and have simple needs, but may have to scale fast.  Plus my sites get lots of weird hits and spam from places like Russia and China.  Some extra security won't be that bad of an investment.

4. Building Mobile Apps on AWS


Why: Because, you had me at mobile.

5. AWS Summit Social and AWS Activate - Networking & Social Event for Startups


Why: Get to meet at greet the brains behind the organization.  

Hope to see you there!

Monday, June 2, 2014

PHPUnit - Migrate from PEAR Install to PHAR (Windows & Linux)

Migrate PHPUnit from PEAR toPHAR


You have installed PHPUnit with PEAR, but the installation method is migrating and you keep getting the error message:

You have installed PHPUnit via PEAR. This installation method is no longer supported and http://pear.phpunit.de/ will be shut down no later than December, 31 2014.

Please read http://phpunit.de/manual/current/en/installation.html and learn how to use PHPUnit from a PHAR or install it via Composer. 


It lets you know that you haven't installed it the right way, but now how do you correct the problem and remove the annoying error message? 


First remove it (Windows & Linux):

pear uninstall phpunit/PHPUnit



Get and install PHPUnit using the phar, Linux (or Amazon EC2) version:

wget https://phar.phpunit.de/phpunit.phar
chmod +x phpunit.phar
mv phpunit.phar /usr/local/bin/phpunit

#and in my case, the phpunit executable also needed to be placed herecp /usr/local/bin/phpunit /usr/bin/phpunit

Get and Install PHPUnit using the phar, Windows version:

Download the .phar file to somewhere you can run it with php.
  1. Open this address in a browser and save to your disk: https://phar.phpunit.de/phpunit.phar 
  2. Open the command line and go to the directory (cd {savepath}) you have saved the file in. 
  3. Execute tests with: php phpunit.phar testfile.php  
  4. Copy phpunit.phar to overwrite your existing phpunit file (replace c:\php\phpunit with your installed location).  Back up the original phpunit file just in case.
    copy c:\php\phpunit c:\php\phpunit.bakcopy phpunit.phar c:\php\phpunit

NOTE:

If you are getting the error below when running your tests:
Failed opening required 'PHPUnit/Autoload.php'
You may need to remove or comment out the Autoload include line .  This file has already been included in the .phar PHP Archive.

//require_once 'PHPUnit/Autoload.php';



Now, use this testing framework to check your code and help you feel confident when deploying your applications.


There are a great number of resources on the official PHPUnit page that you should check out if you want to learn more about PHPUnit and how to use it to your advantage - PHPUnit Presentations.

Monday, May 19, 2014

Great Code Syntax Highlighter for Blog Posting

How do you get nicely formatted code into your Blog that looks as good as Stack Overflow or at least as good as your IDE?  


You need an online syntax highlighter that converts your source code into pretty HTML code.  I liked the one at http://markup.su/highlighter/ and found the Slush and Poppies theme to be the most attractive.
  1. Paste code into the Online Syntax Highlighter
  2. Paste that code back into your blog as HTML (use HTML view in Blogger)
  3. Add a scrollbar using the "overflow: auto;" css setting to handle longer lines of text

Blogger Notes: If you don't want to see scrollbars, you might want to manually wrap columns at 70 characters to keep it looking nice, because Blogger doesn't mind letting lines extend beyond their section.

public class CTGTablePool {

    /**
    *
    * @param tableClass
    * @param c - Use application context if your process lives longer
    * than the activity
    * @return
    */
    public synchronized static < T > T getTable(Class < T > 
        tableClass, Context c) {
        
        self().checkInit();
        String hashKey = self().dbName + tableClass.getName();
        if (self().tableMap.get(hashKey) == null) {
            Utils.LogD(TablePool.class.getName(), 
                "Creating table connection for: " +
                tableClass.getName() + " in db: " + self().dbName);

            Constructor < T > construct;
            T tableInstance = null;
            try {
                construct = tableClass.getConstructor(Context.class,
                    String.class);
                tableInstance = (T)construct.newInstance(
                    c.getApplicationContext(), self().dbName);
                self().tableMap.put(hashKey, tableInstance);
            } catch (Exception e) {
                throw new RuntimeException(
                    "Could not create the table: " + 
                    tableClass.getName() + " in db: " + 
                    self().dbName + ", because of: " + e.toString());
            }
            return tableInstance;
        } else {
            Utils.LogD(TablePool.class.getName(), 
                "Reusing existing table connection for: " + 
                tableClass.getName() + " in db: " + self().dbName);
            return (T)self().tableMap.get(hashKey);
        }
    }

    /**
    * Reset all the database connections
    */
    public synchronized static void invalidate() {
        Utils.LogD(CTGTablePool.class.getName(), 
            "Invalidating database connections");
        self().tableMap.clear();
        self().tableVersionTable = null;
        TableManager.invalidate();
    }
}



Ok, now that looks ok, but it is a pain to manually wrap the lines to 70 chars.  Help!  Is there a better blogging tool that is more friendly to programmers?

Note: if you don't manually wrap it before pasting the code into the syntax highlighter, you get an odd looking line extending beyond the sample area.

    private void checkInit() {
        Assert.assertNotNull(CTGTablePool.class.getName()+".setDbName() needs to be called first!", self().dbName);
    }


You can modify the <pre> style to add a scroll bar instead which looks a little better.  Although I don't know why Blogger seems to think all I should get is this narrow space.  I don't think I am asking for too much, am I?
 <pre style="background: #f1f1f1; color: black; overflow: auto;"> 

private void checkInit() {
        Assert.assertNotNull(CTGTablePool.class.getName()+".setDbName() needs to be called first!", self().dbName);
    }


Friday, May 16, 2014

Easiest Way to Handle Spammers on Contact or Comment Forms

What is the easiest way to prevent spam in contact or comment forms on a website?


I have a contact form on one of my websites www.lutz-engr.com and was getting some actual people filling out the form for legitimate reasons, but the vast majority of the content was spam from countries such as Russia and from IP addresses reported from many other countries.  See my original post on "How to Handle Spammers".  Sometimes the forms are filled out in groups and I suspect there is either a system that can disguise its IP address or there is a network of spam controllers running on different computers.  Either way I always found that the spammed forms had one thing in common...

Spammers do not run javascript.


At least my spammers weren't using javascript.  So now I could turn this observation into an advantage and simply require users to be running javascript in order to submit a form on the website.  That's not too big a deal - I don't know any typical users that browse the web with javascript disabled.  Have you tried it?  It sucks!


This is what my form looks like in HTML.  Note I am using the PUT HTML verb here instead of the typical POST or GET.  The reason I use PUT is that my PHP page will not respond to the PUT method, so unless it is changed by the javascript to POST, activating the form won't be handled by the webserver.

<form action="" method="PUT" name="contact_form">
    Your Name: &nbsp;</div>
    <input type="text" name="name_field" size="35">*<br>
    Phone: &nbsp;</div>
    <input type="text" name="phone_field" size="35"><br>
    Email: &nbsp;</div>
    <input type="email" name="email_field" size="35">*<br>
    <input type="hidden" value="nojs (unused)" name="timedayjs" 
        id="timedayjs">
    <input type="submit" value="Submit" name="comment_submit_button" 
        id="comment_submit_button" disabled="disabled">
</form>

Change your form so that it is initially disabled in HTML

  • Form Method - Set to method="PUT" to disable, but changed to "POST" for javascript users
  • Form Action - Set to action="" to completely disable the form and when run, javascript puts the correct location here
  • Form Field 'timedayjs' - This is just a form field I used to capture if the user had run the javascript on the page.  It really could be named anything, but I choose this name to  throw anyone off who actually did take the time to look at the HTML code.
  • Submit Button - This I set to disabled="disabled" so a regular user that had javscript disabled would notice they couldn't click on the form.  Of course a spammer could just ignore this directive, but without the correct form method and form action the results still won't get sent to the server.

Enable your form using javascript


Below is the javascript code I use that requires the user to have javascript running which enables the form and to fixes the four configuration items that are disabled in the HTML code.  I am using jQuery so the functions go in the $(document).ready function to be executed after the page is ready.  If you don't use jQuery you could put the functions in a javascript setupForm() function and execute in on the name_field using something like onchange="setupForm()".

<script type="text/javascript">
    $(document).ready (function() {
      $('#timedayjs').val('set by js');
      document.contact_form.method = 'post';
      document.contact_form.action = 'index.php';
      document.getElementById('comment_submit_button').disabled=false;
    });
</script>


All of these actions has resulted in a drastic reduction of spamming through our contact us forms on the website.  Please try this code on your site.  Leave a note or +1 if you feel this was helpful. 

Wednesday, April 23, 2014

Android Networking Example and Tutorial (with easy retries)

This post explains how to use the WorxForUs Network framework to have an Android app that works robustly even in areas that have poor network connectivity. 

Background:

I wrote this library because none of the tutorials I found went clearly into the different types of faults that you can experience when developing web apps.  They worked fine to get you started, but what about if you are now on a mobile network and not all of your packets are getting through?  What happens if you socket times out?  What do I do if I need to handle cookies?  This framework is an attempt to address those issues along with adding a simple way to retry communications that did not go through properly.

Features:

  • Automatic cookie handling
  • Baked in network retries (just specify number of retries to allow before failing)
  • Easy to understand handling of the various types of network errors
  • Handles authenticated and unauthenticated HTTP requests

Accessing the Network with Retries (without authentication):

First, download the WorxForUs framework from the github page (or clone here).
Import the project into your Eclipse or Android Studio.

Check to see if the network is connected (you don't want to try and download something if the user has activated flight mode).

    //NetResult is an object to store the results
    NetResult netResult = null; 
    String url = "http://www.google.com/";
    //Load the values being posted
    List<NameValuePair> params = new ArrayList<NameValuePair>();    params.add(new BasicNameValuePair("q", "bears"));

    String serverResponse ="";
    //If the network is not currently connected, don't try to talk
    if (NetHandler.isNetworkConnected(con)) {
        netResult = NetHandler.handlePostWithRetry(url, params, NetHandler.NETWORK_DEFAULT_RETRY_ATTEMPTS);

        //get the server response as a string
        serverResponse = Utils.removeUTF8BOM(EntityUtils.toString(net_result.net_response_entity, Utils.CHARSET));
        //Notify the HTTP client that all data was read
        netResult.closeNetResult();
    }
What is this NetResult object ?  This object contains all the information you need to decode the response from the webserver. If netResult.net_response_entity is not null, then that is the response from the server.  Send that value to your handling routing.

    NetResult.net_success - Equals true when the server was successfully contacted (this says nothing about if your request was valid though).
    NetResult.net_error - Contains the error message or exception associated with the connection
    NetResult.net_error_type - Contains the name of the type of error that occurred (ie. HttpResponseException, SocketTimeoutException, SocketException, or IOException)
    NetResult.net_response_entity - This is the actual response from the server.  A common use is to run:
String consume_str = Utils.removeUTF8BOM(EntityUtils.toString(net_result.net_response_entity, Utils.CHARSET)); 
or if you want to capture JSON data
NetResult.handleGenericJsonResponseHelper(net_result, this.getClass().getName()); 
Note: the class name is passed for logging purposes only
Now net_result.object will contain your JSON objects parsed for you

After reading your data from the netResult.net_response_entity, you will need to call netResult.closeNetResult().  This function eventually calls HttpEntity.consumeContent() which releases any resources associated with that object.

Accessing the Network with Authentication:

 To call the network with authentication is simple once you have the authentication helper configured for your particular webserver.





// load authentication data
if (!AuthNetHandler.isAuthenticationSet()) {
    // passing the context here allows the system to update the preferences with a validated usernumber (if found)
    AuthNetHandler.setAuthentication(host, new MyAuthenticationHelper(con));
    NetAuthentication.loadUsernamePassword(username, password);
}
// if network is ready, then continue
// check if network was disabled by the user

if (NetHandler.isNetworkConnected(context)) {
    // if user has credentials - upload then download so data is not lost
    if (NetAuthentication.isReadyForLogin()) {
        netResult = AuthNetHandler.handleAuthPostWithRetry(url, params, num_retries);
        //...handle the netResult response here
        netResult.closeNetResult();
    } else {
        Log.e(this.getClass().getName(), "Did not attempt to login, no authentication info");
    }
}


When you have authenticated requests there are a few extra steps before you call  AuthNetHandler.handleAuthPostWithRetry(...).

First you will need to implement a class from NetAuthenticationHelper.  This will tell the framework how you are going to login, what messages are errors,  and basically how you are validating a login.   It may seem like a lot of extra code, but try to stick with it.  Having all your authentication code in one place can be extremely helpful.

public class MyAuthHelper implements NetAuthenticationHelper {
    @Override
    public String getLoginURL(String host) {
        return "https://myweb/remote_login_address.php";
    }


getLoginURL is simple the location where you expect the app to connect to when sending login parameters.  Just put the URL you need in here.

    @Override
    public void markAsLoginFailure(NetResult result) {

        result.object = new String("Login Error");
    }


or, in my case I use JSON objects

    @Override
    public void markAsLoginFailure(NetResult result) {
        try {
            result.object = new JSONObjectWrapper("Jsonstring");
        } catch (JSONExceptionWrapper e) {
            throw new RuntimeException("Could not parse default login failed JSON string.");
        }
    }

Put whatever your webserver responds with in here on a failed user login.  This section forces a result to be simulated as a login failure in the result.object variable.  Let's say you've identified a failed login, this output gets sent through validateLoginResponse(...) where the failed login will be identified.

    @Override
    public void markAsLoginSuccessFromCache(NetResult result) {

        result.object = new String("Login Successful");
    }


Put whatever your webserver could respond with on a successful login.  This section forces a result to be simulated as a login success in the result.object variable.

    @Override
    public String getLoginErrorMessage() { return "Could not login user"; }


Here put what you would like be passed as the error message to netResult.net_error when a user is not able to be logged in.

    @Override
    public int validateLoginResponse(NetResult netResult) {
        //Returns NetAuthentication.NO_ERRORS, NETWORK_ERROR, or SERVER_ERROR
        //Check the login server result from netResult.object
        //to determine if your login was successful or not
        if (netResult.net_success) {
            String response = netResult.object;
            if (response.contains("not logged in indication")) {
                return NetAuthentication.NOT_LOGGED_IN;
            } else if (response.contains("login error indication"))) {
                return NetAuthentication.LOGIN_FAILURE;
            } else if (response.contains("cant parse data indication"))) {
                return NetAuthentication.SERVER_ERROR;
            }
        } else {
            return NetAuthentication.NETWORK_ERROR;
        }
        return NetAuthentication.NO_ERRORS;    
    }

validateLoginResponse(...)performs the bulk of the login checking, it determines if the user wasn't logged in, there was a login error (ie. wrong password), a server error, or network error.  Depending on what you expect from the server, you will send a response of NetAuthentication.NO_ERRORS, NETWORK_ERROR, or SERVER_ERROR.

    @Override
    public int peekForNotLoggedInError(NetResult netResult) {

        //... check the netResult.object for a login failure on a page that wasn't the login page.
    }


Similarly to the previous function, the peekForNotLoggedInError(...) checks for login errors, but on pages that are not the login page.  Consider the example where you have already logged in, but then check a different page to download some other data.  If your user's session is suddenly logged out, you will get an error that could look different than the one you get on the login page.  So that specific logic for unexpected login failure goes in here. 

    @Override
    public NetResult handleUsernameLogin(String host, String username, String password) {
        List<NameValuePair> params = new ArrayList<NameValuePair>();
        params.add(new BasicNameValuePair("uname_field", "username"));
        params.add(new BasicNameValuePair("pword_field", "password"));
       
        NetResult netResult = NetHandler.handlePostWithRetry(this.getLoginURL(host), params , NetHandler.NETWORK_DEFAULT_RETRY_ATTEMPTS);
        //save the result and close network stream
        consume_str = Utils.removeUTF8BOM(EntityUtils.toString(result.net_response_entity, Utils.CHARSET));
        netResult.object = consume_str;
        netResult.closeNetResult();
        return netResult;
    }


The handleUsernameLogin(...) function provides the actual fields and logic needed to send the request to the webserver.  Simply fill in your specific fields for login.

If you have a different request using a token, the handleTokenLogin(...) function can be used for that purpose.

Wow, if you've made it to the end of this tutorial, you are a real trooper and I salute you!

Notes:

HTTP Params are encoded with UTF-8.  If your webserver expects another character set, you will need to change the handlePost(...) routing in NetHandler to use that encoding scheme.





Tuesday, April 22, 2014

Obscured Shared Preferences for Android

What are Shared Preferences for?


Shared Preferences on Android are great for keeping information that you use in an app.  But this information is all stored in plain text.  If a user has a rooted device, then they can simply go in an examine what you have in your preferences under /data/data/com.package/shared_prefs/AppPrefs.xml.

The data is all right there in plain view:
AppPrefs.xml
<?xml version='1.0' encoding='utf-8' standalone='yes' ?>
<map>
<string name="UserName">JoeBlow</string>
<int name="UserNum" value="2" />
<string name="auth_modified_date">2014-04-21 16:15:32</string>
<string name="Password">pass123</string>
</map>


So if you plan on storing passwords or other data that you don't want people to have easy access to, that is obviously no good.  So to help secure the data in your shared Preferences, use the ObscuredSharedPreferences.java code here.

The encrypted data will look like this:
AppPrefs.xml
<?xml version='1.0' encoding='utf-8' standalone='yes' ?>
<map>
<string name="UserName">vbBchG+mE+/QGg+3+YuDW9MEI16hC7</string>
<string name="UserNum">PK8XONc4=</string>
<string name="auth_modified_date">C441dQ1G7L4=</string>
<string name="Password">Fx8kG7rWXqQ=</string>
</map>


How to make your Shared Preferences encrypted


Thanks to emmby at http://stackoverflow.com/questions/785973/what-is-the-most-appropriate-way-to-store-user-settings-in-android-application/6393502#6393502

Simply wrap your own SharedPreferences object in this one, and any data you read/write will be automatically encrypted and decrypted.  This is a drop in replacement for SharedPreferences and includes some additional functionality over the code in the StackOverflow post.

Usage:
    ObscuredSharedPreferences prefs = ObscuredSharedPreferences.getPrefs(this, MY_APP_NAME, Context.MODE_PRIVATE);

    //save data
    prefs.edit().putString("foo","bar").commit();

    //retrieve data
    prefs.getString("foo", null);



Additional Features

  • This code has built-in logic to handle the case where the information was not previously encrypted such as when you upgrade your existing app to this code.
  • The secret key is not hard coded, but is generated at runtime, so that a compromised device only affects that individual device.

The code can be retrieved for download here at the WorxForUs github repository for ObscuredSharedPreferences.


The Code - ObscuredSharedPreferences.java

package com.worxforus.android;

import java.util.Map;
import java.util.Set;

import javax.crypto.Cipher;
import javax.crypto.SecretKey;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.PBEParameterSpec;

import com.worxforus.Base64Support;

import android.content.Context;
import android.content.SharedPreferences;
import android.provider.Settings;
import android.provider.Settings.Secure;
import android.util.Log;


/**
 * Thanks to emmby at http://stackoverflow.com/questions/785973/what-is-the-most-appropriate-way-to-store-user-settings-in-android-application/6393502#6393502
 * This class has the following additions over the original:
 *  additional logic for handling the case for when the preferences were not originally encrypted, but now are.
 *  The secret key is no longer hard coded, but defined at runtime based on the individual device.
 *  The benefit is that if one device is compromised, it now only affects that device.
 *
 * Simply replace your own SharedPreferences object in this one, and any data you read/write will be automatically encrypted and decrypted.
 *
 * Updated usage:
 *    ObscuredSharedPreferences prefs = ObscuredSharedPreferences.getPrefs(this, MY_APP_NAME, Context.MODE_PRIVATE);
 *    //to get data
 *    prefs.getString("foo", null);
 *    //to store data
 *    prefs.edit().putString("foo","bar").commit();
 */
public class ObscuredSharedPreferences implements SharedPreferences {
    protected static final String UTF8 = "UTF-8";
    //this key is defined at runtime based on ANDROID_ID which is supposed to last the life of the device
    private static char[] SEKRIT=null;


    protected SharedPreferences delegate;
    protected Context context;
    private static ObscuredSharedPreferences prefs = null;
  
    //Set to true if a decryption error was detected
    //in the case of float, int, and long we can tell if there was a parse error
    //this does not detect an error in strings or boolean - that requires more sophisticated checks
    public static boolean decryptionErrorFlag = false;
 
    /**
     * Constructor
     * @param context
     * @param delegate - SharedPreferences object from the system
     */
    public ObscuredSharedPreferences(Context context, SharedPreferences delegate) {
        this.delegate = delegate;
        this.context = context;
        SEKRIT = Settings.Secure.ANDROID_ID.toCharArray();
    }
  
    /**
     * Only used to change to a new key during runtime.
     * If you don't want to use the default per-device key for example
     * @param key
     */
    public static void setNewKey(String key) {
        SEKRIT = key.toCharArray();
    }

    /**
     * Accessor to grab the preferences in a singleton.  This stores the reference in a singleton so it can be accessed repeatedly with
     * no performance penalty
     * @param c - the context used to access the preferences.
     * @param appName - domain the shared preferences should be stored under
     * @param contextMode - Typically Context.MODE_PRIVATE
     * @return
     */
    public synchronized static ObscuredSharedPreferences getPrefs(Context c, String appName, int contextMode) {
        if (prefs == null) {
            //make sure to use application context since preferences live outside an Activity
            //use for objects that have global scope like: prefs or starting services
                prefs = new ObscuredSharedPreferences(
                        c.getApplicationContext(), c.getApplicationContext().getSharedPreferences(appName, contextMode) );
        }
        return prefs;
    }
  
    public class Editor implements SharedPreferences.Editor {
        protected SharedPreferences.Editor delegate;

        public Editor() {
            this.delegate = ObscuredSharedPreferences.this.delegate.edit();                  
        }

        @Override
        public Editor putBoolean(String key, boolean value) {
            delegate.putString(key, encrypt(Boolean.toString(value)));
            return this;
        }

        @Override
        public Editor putFloat(String key, float value) {
            delegate.putString(key, encrypt(Float.toString(value)));
            return this;
        }

        @Override
        public Editor putInt(String key, int value) {
            delegate.putString(key, encrypt(Integer.toString(value)));
            return this;
        }

        @Override
        public Editor putLong(String key, long value) {
            delegate.putString(key, encrypt(Long.toString(value)));
            return this;
        }

        @Override
        public Editor putString(String key, String value) {
            delegate.putString(key, encrypt(value));
            return this;
        }

        @Override
        public void apply() {
            //to maintain compatibility with android level 7
            delegate.commit();
        }

        @Override
        public Editor clear() {
            delegate.clear();
            return this;
        }

        @Override
        public boolean commit() {
            return delegate.commit();
        }

        @Override
        public Editor remove(String s) {
            delegate.remove(s);
            return this;
        }

        @Override
        public android.content.SharedPreferences.Editor putStringSet(String key, Set<String> values) {
            throw new RuntimeException("This class does not work with String Sets.");
        }
    }

    public Editor edit() {
        return new Editor();
    }


    @Override
    public Map<String, ?> getAll() {
        throw new UnsupportedOperationException(); // left as an exercise to the reader
    }

    @Override
    public boolean getBoolean(String key, boolean defValue) {
        //if these weren't encrypted, then it won't be a string
        String v;
        try {
            v = delegate.getString(key, null);
        } catch (ClassCastException e) {
            return delegate.getBoolean(key, defValue);
        }
       
        return v!=null ? Boolean.parseBoolean(decrypt(v)) : defValue;
    }

    @Override
    public float getFloat(String key, float defValue) {
        String v;
        try {
            v = delegate.getString(key, null);
        } catch (ClassCastException e) {
            return delegate.getFloat(key, defValue);
        }
        try {
            return Float.parseFloat(decrypt(v));
        } catch (NumberFormatException e) {
            //could not decrypt the number.  Maybe we are using the wrong key?
            decryptionErrorFlag = true;
            Log.e(this.getClass().getName(), "Warning, could not decrypt the value.  Possible incorrect key.  "+e.getMessage());
        }
        return defValue;
    }

    @Override
    public int getInt(String key, int defValue) {
        String v;
        try {
            v = delegate.getString(key, null);
        } catch (ClassCastException e) {
            return delegate.getInt(key, defValue);
        }
        try {
            return Integer.parseInt(decrypt(v));
        } catch (NumberFormatException e) {
            //could not decrypt the number.  Maybe we are using the wrong key?
            decryptionErrorFlag = true;
            Log.e(this.getClass().getName(), "Warning, could not decrypt the value.  Possible incorrect key.  "+e.getMessage());
        }
        return defValue;
    }

    @Override
    public long getLong(String key, long defValue) {
        String v;
        try {
            v = delegate.getString(key, null);
        } catch (ClassCastException e) {
            return delegate.getLong(key, defValue);
        }
        try {
            return Long.parseLong(decrypt(v));
        } catch (NumberFormatException e) {
            //could not decrypt the number.  Maybe we are using the wrong key?
            decryptionErrorFlag = true;
            Log.e(this.getClass().getName(), "Warning, could not decrypt the value.  Possible incorrect key.  "+e.getMessage());
        }
        return defValue;
    }

    @Override
    public String getString(String key, String defValue) {
        final String v = delegate.getString(key, null);
        return v != null ? decrypt(v) : defValue;
    }

    @Override
    public boolean contains(String s) {
        return delegate.contains(s);
    }

    @Override
    public void registerOnSharedPreferenceChangeListener(OnSharedPreferenceChangeListener onSharedPreferenceChangeListener) {
        delegate.registerOnSharedPreferenceChangeListener(onSharedPreferenceChangeListener);
    }

    @Override
    public void unregisterOnSharedPreferenceChangeListener(OnSharedPreferenceChangeListener onSharedPreferenceChangeListener) {
        delegate.unregisterOnSharedPreferenceChangeListener(onSharedPreferenceChangeListener);
    }

    @Override
    public Set<String> getStringSet(String key, Set<String> defValues) {
        throw new RuntimeException("This class does not work with String Sets.");
    }


    protected String encrypt( String value ) {

        try {
            final byte[] bytes = value!=null ? value.getBytes(UTF8) : new byte[0];
            SecretKeyFactory keyFactory = SecretKeyFactory.getInstance("PBEWithMD5AndDES");
            SecretKey key = keyFactory.generateSecret(new PBEKeySpec(SEKRIT));
            Cipher pbeCipher = Cipher.getInstance("PBEWithMD5AndDES");
            pbeCipher.init(Cipher.ENCRYPT_MODE, key, new PBEParameterSpec(Settings.Secure.getString(context.getContentResolver(), Secure.ANDROID_ID).getBytes(UTF8), 20));
            return new String(Base64Support.encode(pbeCipher.doFinal(bytes), Base64Support.NO_WRAP),UTF8);
        } catch( Exception e ) {
            throw new RuntimeException(e);
        }

    }

    protected String decrypt(String value){
        try {
            final byte[] bytes = value!=null ? Base64Support.decode(value,Base64Support.DEFAULT) : new byte[0];
            SecretKeyFactory keyFactory = SecretKeyFactory.getInstance("PBEWithMD5AndDES");
            SecretKey key = keyFactory.generateSecret(new PBEKeySpec(SEKRIT));
            Cipher pbeCipher = Cipher.getInstance("PBEWithMD5AndDES");
            pbeCipher.init(Cipher.DECRYPT_MODE, key, new PBEParameterSpec(Settings.Secure.getString(context.getContentResolver(), Secure.ANDROID_ID).getBytes(UTF8), 20));
            return new String(pbeCipher.doFinal(bytes),UTF8);
        } catch( Exception e) {
            Log.e(this.getClass().getName(), "Warning, could not decrypt the value.  It may be stored in plaintext.  "+e.getMessage());
            return value;
        }
    }
}

Compile it

To compile this you will also need the Base64 library.  This was introduced in Android in API level 8, so earlier devices will not have the package.  Include in your project the WorxForUs Base64Support package here.

Related Work


 Please check out the WorxForUs library of Android solutions.  The intro is here and the framework can be found here. It includes other useful nuggets such as database access helpers for correctly handling multi-threaded environments and helpers for network access for cookies, network retry and other useful features.


Friday, March 21, 2014

Android Database Example and Tutorial (thread safe version)

This post explains how to use the WorxForUs SQLite Database framework to have an Android app that works well in multi-threaded applications and supports table level upgrades (instead of the entire database). 

Background:

I wrote this library because I my app was based on the Android examples and worked on the device I had at the time (a Galaxy S), but suddenly was having problems when working on newer devices.  After much searching, I found out that the new devices were multi-core and was accessing the database in multiple places in the program.  I didn't have that problem with the earlier devices because they were single-threaded and so the database access was naturally serialized.  The Android samples were good, but didn't go far enough to address the problems I was seeing.

Typical Classic Style Database access exceptions:

This framework corrects some of the following issues that are commonly seen once your app starts getting more complicated and calls the database from multiple locations or background threads.

Android 4
java.lang.IllegalStateException: Cannot perform this operation because the connection pool has been closed.
android.database.sqlite.SQLiteDatabaseLockedException: database is locked (code 5)

Android 2.3.3
java.lang.IllegalStateException: database not open
android.database.sqlite.SQLiteException: database is locked
Failed to setLocale() when constructing, closing the database

Accessing a SQLite database in a thread-safe manner:

First, download the WorxForUs framework from the github page (or clone here).
Import the project into your Eclipse or Android Studio.

Create your new project: right-click and select 'New / Android Application Project'.

I'm going to name it: 'WorxforusDbSample' and create the project using defaults for the remaining options. 

Once the project is loaded, you will need to add a reference to the worxforus_library project from Properties / AndroidCreate a new class in com.example.worxforusdbsample called Nugget and enter the following code.

package com.example.worxforusdbsample;

public class Nugget {
    String type="";
    int id =0;
  
    public static final String IRON = "Iron";
    public static final String GOLD = "Gold";
    public static final String DIAMOND = "Diamond";

    public String getType() {
        return type;
    }
    public void setType(String type) {
        this.type = type;
    }
    public void setType(int type) {
        if (type == 1)
            setType(GOLD);
        else if (type == 2)
            setType(DIAMOND);
        else //set remaining to Iron
            setType(IRON);
    }
  
    public int getId() {
        return id;
    }
  
    public void setId(int id) {
        this.id = id;
    }
  
    public String getDescription() {
        return type+" nugget";
    }
}


Creating your object is the easy part, now create the association with the database.  Create a new class called NuggetTable and extend from the abstract class TableInterface<Nugget>.   Extending from TableInterface allows us to us the TableManager which is what serializes access to the database and prevents from multiple threads colliding when accessing the data.  Get the sample app code here.

package com.example.worxforusdbsample;

import java.util.ArrayList;

import android.content.ContentValues;
import android.content.Context;
import android.database.Cursor;
import android.database.SQLException;
import android.database.sqlite.SQLiteDatabase;
import android.database.sqlite.SQLiteOpenHelper;
import android.database.sqlite.SQLiteDatabase.CursorFactory;
import android.util.Log;

//Result is a convenience class to capture errors and pass objects back to the caller
import com.worxforus.Result;
import com.worxforus.db.TableInterface;

public class NuggetTable extends TableInterface<Nugget> {
    public static final String DATABASE_NAME = "sample_db"; //Instead of a text string, this should be a static constant for your app
    public static final String TABLE_NAME = "nugget_table";
    public static final int TABLE_VERSION = 1;
    // 1 - Initial version

    static int i = 0; // counter for field index
    public static final String NUGGET_ID = "nugget_id"; // int
    public static final int NUGGET_ID_COL = i++;
    public static final String NUGGET_TYPE = "nugget_type"; // String
    public static final int NUGGET_TYPE_COL = i++;

    private static final String DATABASE_CREATE = "CREATE TABLE " + TABLE_NAME + " ( "
            + NUGGET_ID + "     INTEGER PRIMARY KEY AUTOINCREMENT,"
            + NUGGET_TYPE + "   TEXT"
            + ")";

    private SQLiteDatabase db;
    private NuggetDbHelper dbHelper;

    public NuggetTable(Context _context) {
        dbHelper = new NuggetDbHelper(_context, DATABASE_NAME, null, DATABASE_VERSION);
    }

    @Override
    public Result openDb() {
        Result r = new Result();
        try {
            db = dbHelper.getWritableDatabase();
        } catch (SQLException e) {
            Log.e(this.getClass().getName(), r.error);
            throw(new RuntimeException(e));
        }
        return r;
    }

    @Override
    public void closeDb() {
        if (db != null)
            db.close();
    }

    @Override
    public void createTable() {
        dbHelper.onCreate(db);
    }

    @Override
    public void dropTable() {
        db.execSQL("DROP TABLE IF EXISTS "+TABLE_NAME);
        invalidateTable();
    }

    public void wipeTable() {
        synchronized (TABLE_NAME) {
            db.delete(TABLE_NAME, null, null);
        }
    }
   
    @Override
    public void updateTable(int last_version) {
        dbHelper.onUpgrade(db, last_version, TABLE_VERSION);
    }

    @Override
    public String getTableName() {
        return TABLE_NAME;
    }

    @Override
    public int getTableCodeVersion() {
        return TABLE_VERSION;
    }

    /**
     * For ease of use, not efficiency, I combined insert and update as a single statement.  Note that if the item exists,
     * that two operations are performed, a delete and insert.
     */
    @Override
    public Result insertOrUpdate(Nugget t) {
        synchronized (TABLE_NAME) {
            Result r = new Result();
            try {
                ContentValues cv = getContentValues(t);
                r.last_insert_id = (int) db.replace(TABLE_NAME, null, cv);
            } catch( Exception e ) {
                Log.e(this.getClass().getName(), e.getMessage());
                r.error = e.getMessage();
                r.success = false;
            }
            return r;
        }
    }
   
    public Result insert(Nugget t) {
        synchronized (TABLE_NAME) {
            Result r = new Result();
            try {
                ContentValues vals = new ContentValues();
                if (t.getId() > 0)
                    vals.put(NUGGET_ID, t.getId());
                vals.put(NUGGET_TYPE, t.getType());
                r.last_insert_id = (int) db.insert(TABLE_NAME, null, vals);
            } catch( Exception e ) {
                Log.e(this.getClass().getName(), e.getMessage());
                r.error = e.getMessage();
                r.success = false;
            }
            return r;
        }
    }

    @Override
    public Result insertOrUpdateArrayList(ArrayList<Nugget> t) {
        return null; //not implemented in this sample
    }

    public Result insertArrayList(ArrayList<Nugget> list) {
        Result r = new Result();
        db.beginTransaction();
        for (Nugget item : list) {
            try {
                insert(item);
            } catch(SQLException e ) {
                Log.e(this.getClass().getName(), e.getMessage());
                r.error = e.getMessage();
                r.success = false;
            }
        }
        db.setTransactionSuccessful();
        db.endTransaction();
        return r;
    }
   
    @Override
    public ArrayList<Nugget> getUploadItems() {
        return null; //not implemented in this sample
    }

    public ArrayList<Nugget> getAllEntries() {
        ArrayList<Nugget> al = new ArrayList<Nugget>();
        Cursor list = getAllEntriesCursor();
        if (list.moveToFirst()){
            do {
                al.add(getFromCursor(list));
            } while(list.moveToNext());
        }
        list.close();
        return al;
    }
   
    protected Cursor getAllEntriesCursor() {
        return db.query(TABLE_NAME, null, null, null, null, null, NUGGET_ID);
    }
   
    // ================------------> helpers <-----------==============\\

    /** returns a ContentValues object for database insertion
     * @return
     */
    public ContentValues getContentValues(Nugget item) {
        ContentValues vals = new ContentValues();
        //prepare info for db insert/update
        vals.put(NUGGET_ID, item.getId());
        vals.put(NUGGET_TYPE, item.getType());
        return vals;
    }
   
    /**
     * Get the data for the item currently pointed at by the database
     * @param record
     * @return
     */
    public Nugget getFromCursor(Cursor record) {
        Nugget c= new Nugget();
        c.setId(record.getInt(NUGGET_ID_COL));
        c.setType(record.getString(NUGGET_TYPE_COL));
        return c;
    }
   
    // ================------------> db helper class <-----------==============\\
    private static class NuggetDbHelper extends SQLiteOpenHelper {
        public NuggetDbHelper(Context context, String name,
                CursorFactory factory, int version) {
            super(context, name, factory, version);
        }

        @Override
        public void onCreate(SQLiteDatabase db) {
            db.execSQL(DATABASE_CREATE);
        }

        @Override
        public void onUpgrade(SQLiteDatabase db, int oldVersion,int newVersion){

            // called when the version of the existing db is less than the current
            Log.w(this.getClass().getName(), "Upgrading table from " + oldVersion + " to " + newVersion);        }
    }
   
}


Code for the main activity

To connect to the database, first establish a NuggetTable object in your app or activity onCreate method.  You may want to store this connection in a singleton for easy access by any other activity (and also to reduce memory usage and connection time).

public class NuggetDbActivity extends ActionBarActivity {
    NuggetTable nuggetTable;
    static final int NUM_ITEMS_TO_CREATE = 5;
    static final int NUM_THREADS_TO_RUN = 10;
   
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        nuggetTable = new NuggetTable(this);
        //Your activities setup code...
    }


Once the NuggetTable object is ready, then you use the TableManager to serialize access to the database.

    public void addRandomDataWorxForUs(NuggetTable table) {
        TableManager.acquireConnection(this, NuggetTable.DATABASE_NAME, table);
        Nugget nugget = new Nugget();

        Result r = table.insert(nugget);
        TableManager.releaseConnection(nuggetTable);
    }


The TableManager.acquireConnection does a number of things, it checks to see if your table has already been created or if you have marked it for upgrade (ie. increased the TABLE_VERSION and if so, will run your onUpgrade code).  Otherwise it just creates a new table and locks its use by only the current thread.

In addition to the table object we want to lock for our use, a context is passed that is needed to initialize the database connection.  The database name is also passed to the acquire method so that the method knows which database to use for storing the table meta information such as the current version of the table and sync information.

Finally when all the data operations are completed, you will need to release the connection held by the TableManager to allow other methods to access the database.  If you forget to release the connection, you will quickly realize it because your app will hang the next time you try to access the database.

That's all there is to it for simple database access that works in a thread safe manner.  Ok, it was a lot of code, but you get the point!

Upgrading Tables

Let's say you have released an app to the Google Play store and now you realize you need to add an index to speed up database access or maybe you need too add another field to store more data.

Add a new field to the existing database, modify your NuggetDbHelper.onUpgrade code:
@Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
    // called when the version of the existing db is less than the current
    Log.w("SampleW4UsDb", "Upgrading table from "+oldVersion+" to "+newVersion);

    if (oldVersion < 2) { //this means the current version is 1 or less
        //EXAMPLE: add field and change the index
        db.execSQL("ALTER TABLE "+TABLE_NAME+" ADD COLUMN "+NEW_COLUMN+" "+NEW_COLUMN_TYPE);
        db.execSQL("DROP INDEX IF EXISTS "+INDEX_1_NAME); //remove old index
        db.execSQL(INDEX_1); //add a new index
        Log.d(
"SampleW4UsDb", "Adding new field and new index to "    + DATABASE_TABLE + " Table");
    }

}


And change the line for TABLE_VERSION to 2
    public static final int TABLE_VERSION = 2;
 

Now the next time TableManager.acquireConnection is run on this table, it will see that the existing table is version 1 (if it hasn't been updated yet) and proceed to run the update code which will change it to version 2.  In this case it will modify the table to include the new field and also add a new index into the table automatically.  You never have to worry about checking that a table was already created or checking the version, the framework handles all of that for you.

If you've found this post helpful, please take a moment to add a comment, +1, or a funny iguana picture.  Thanks!

Related Work

The source code to this example application can be found on github.
https://github.com/RightHandedMonkey/WorxForUsDb_Samples

The Checklists ToGo app uses this framework as a basis for database and network access.  For a complete example, check out the Checklists ToGo API which uses the more complicated features of the framework.
https://github.com/RightHandedMonkey/CTG_API