A/B Testing on in App Purchase

In the last few months I have been asked a lot of questions about A/B testing.  The questions have come from all different perspectives.  The questions go something like this:

Have you ever done and A/B test?

Are A/B tests complicated?

Do you think that A/B testing is worth it?

So what I thought I would do is create a little public experiment with one of my apps.  These are the results.

 

The setup

I have an app called 1RepMax.  It is a weightlifting app that is used when you are doing a percentage of max lifting program.  The app has been around since 2010 and is fairly popular.  It has some competition in the category, but I believe it is one of the top apps for this purpose.

This past year, I added three consumable in-app purchases as a way for users to appreciate the work I have done over the years by giving me a tip.  I have received some tips, but nothing significant.  So I thought I would look at using an A/B test to see if it improved my numbers a little.

Currently the in-app purchases look like this:

 

 

The New Look

Very recently, Marco Arment, the owner of the Overcast podcasting app changed his business model to go from an in-app purchase feature unlock to a patron model.  The patron model is much like what I am doing in my app, but I am calling it a tip.  It quickly struck me that calling this patron supported was a much nicer way to ask for money.  So I thought I might switch to this language.  So why not turn this into an A/B test?

After some quick research, I decided to follow Marco’s lead and layout my current three options just as he did.  Using the patron terminology and the 3 multi-month tiers to describe what users are giving towards.  The new screen looks like this:

One other thing I did before submitting the app to the Appstore was I changed the plus icon that I used to launch the in-app purchase screen to a variation of the app icon itself.  I did this to grab the attention of long time users of the app.  Like I mentioned, this app has been around since 2010 so I thought it might make sense to make this change.

 

I am using Optimizley, a tool that I am most familiar with for a/b testing, as the testing tool for this experiment.

The Results

Upon initial release I have loaded this test as a 50/50 split between the A and B segments.   I saw a twenty percent increase in the number of in-app purchases made.  The distribution between the three tiers is roughly the same.  With an improvement like that, I will be moving everyone over to the patron language very soon.

The Conclusion

Looking back at some of those initial questions, I have some answers.  The first one, have I done any A/B testing, even though I have done tests in the past, here is a specific example.  Are A/B tests complicated?  There are varying levels of complexity to tests, so it is hard to have a yes or no answer to this question.  However, if the test is set up correctly and has some clear goals in mind, I think they are relatively simple to implement.  Lastly, do I think it is worth it?  In this case it certainly was, the changes in language directly added to the money the app is generating.  However, many times, this is not the case.  I would recommend using A/B testing with a strong dose of intuition.

WatchKit – Day One

WatchKit was made avaliable today by Apple.  I have had just a quick read but this is a key to categorizing the different types of interactions.

  • Your Watch app contains your app’s full user interface. The user launches your app from the home screen and uses the app to view or manipulate data.

  • glance is an optional read-only interface that you use to display the most timely and relevant information from your app. Not all apps need a glance, but having one gives the user a convenient way to access your app’s data.

  • Custom notification interfaces let you alter the default interface displayed for recently received local and remote notifications and add custom graphics, content, and formatting. Custom notification interfaces are optional.

We know that Watch Apps require an iPhone to run third party apps.  That is an interesting wrinkle.  What does that mean for fitness apps?  At least non-Apple fitness apps.  I hope to know more in the coming days.  I will put together some kind of proof of concept app to learn a bit more.

Also, we know that based on a recent press release from Apple that third party developers will have to wait until later in 2015 until they can ship full watch apps.  This means our apps are going to fall into the category of either glances or notifications.  It is not exactly what I was hoping for, but interesting none the less.

I will post something more after I get a little more time with WatchKit.

 

refs.

https://developer.apple.com/watchkit/

http://www.apple.com/pr/library/2014/09/09Apple-Unveils-Apple-Watch-Apples-Most-Personal-Device-Ever.html

Note to self: In-App Purchases

Every time I go to add an in-app purchase to one of my apps, I forget the pattern that was set up for my Product Id.  I like to use the reverse domain and the product name tagged on the end like Apple suggests.

 

A product identifier is a string used to uniquely identify every product you wish to sell from your application. The App Store uses it to retrieve information about a product. It is a string identifier that can only contain alphanumeric (A-Z,a-z,0-9), underscore (_), and period (.) characters. You can use any sequence of these characters for your identifier. However, we recommend that you use the reverse domain name style (for example, com.companyname.application.productid) when creating your identifier

 

Hopefully by writing this post it will help me remember that I have this set up this way.  Burned a couple of hours trying to figure it out and it was right there all along.

I found a bug: Your App can not be loaded because it is not a valid application.

This week I ran across an interesting problem. I was taking an older version of one of my company’s apps and making some minor feature adjustments and redeploying. I spent about 5 hours updating the code, debugging and testing. I then spent about 16 hours trying to get the changes to deploy on my device.

What was strange is, this had never happened before with this app and I had other apps that I made changes to where there was no problem.

Here are the scenarios:

Created a debug build and deployed to my iPad and iPhone directly from Xcode.

Specs:

Macbook Pro 2011 - OSx Yosemite 10.10
iPhone 5 - iOS 8.1
iPad 3 - iOS 8.0.2

This build worked as expected. Nothing really to report. I was working in this method for the entire development cycle.

Created an enterprise build and deployed via the classic Testflight service.

When I tried this method, a few unusual things happened.

1. The app loaded as expected, but the icon was in a disabled state where you could not launch the app and it was semi-transparent.

2. I also received the error message: “Unable to Download App – ‘Appname’ could not be installed at this time.”

download error

No matter how many times I selected retry I received the same message. I figured that I set up the certificates incorrectly. But after revoking the certificate and recreating everything I was convinced that this was not the problem. I also used iTunes and tried to drag and drop the app right on the device with the same results.

Next up was scouring stack overflow, I found a few helpful tips, but I did not really find anything that solved the problem.

At this point, I started looking at some of the core files that are used to load the app. I started
comparing some of the core app files to the app that I had that was working. I quickly identified that that info.plist was different between the two apps.

I renamed the non-working app plist and copied the working plist over. I did and enterprise build and it worked just as expected. At this point, I took the working plst and added all of the custom settings from the backup and everything still worked as expected.

I can’t say for sure but it looks like there is some difference in the legacy plist that does not agree with the new way of doing things.

Problem solved, I don’t plan on breaking this down any further, but I am going to submit a Radar to Apple, they may want more information, if so, I will update this post.

Configuration Web Services

There are several excellent open source projects out there that allow developers the ability to add excellent utility to their apps.  There are apps to help your app store rating, force users to update their app, provide users with release notes and tons of other cool things.  One of the things we have found really useful with Crushmedia.net apps are controlling all of those great utilities from a webservice config call.

 

We have started working on our own open source project to create an SDK that wraps up all of those utilities and supports them with a web service.  We have a demo of the web application that gives you control over the web services and we are following that with an open source version of the SDK.

We still need to do a little clean up to let the SDK out in the wild, but so far it is working really well for us.  Can’t wait to share.

Automated build of Icon/Loading Files

I wrote a post a while back about using Image magick and a bash script to quickly to create the various size app icon images you need for your mobile app.  This is just an updated version of that post.

Automated build of Icon/Loading Files  (originally posted 3/24/2012)

Because of the number of different icons needed for mobile apps I needed a quick method to input a master image and then convert the multiple images for Android and iOS.  Ideally I would like something that runs from the command line so I don’t have to open any software apps that will bog down my system.  As much as I looked, I could not find a quick, cost effective (read that as free), software independent, OS independent, method for resizing icon/loading files.

 

I realize that there are Automaters on my Mac, there are batch scripts in Photoshop and Fireworks, even a really cool Mac app called iConify(iOS only) but none of those methods meet all my requirements.  I decided to use ImageMagick as my image conversion tool.  Let me give you a few reasons for my choice.

  1. ImageMagick is open source
  2. Runs on UNIX, Mac OS X, Windows, iOS
  3. Has command Line processing

After choosing my tool, I set out to make this process as automated as possible.  I am a Mac user, so I wrote a bash script to handle the workload.  The bash script will organize my projects, move them around to my local storage, push to Git Hub, and now to build images for my project.

Here is how it works:

convert -geometry 114×114 -quality 100 appicon.png resources/android/appicon.png

This command executes the ImageMagick command CONVERT which takes a file called appicon.png (I have this sized at 512 x 512), resizes it to 114×114 and outputs it to my android folder with the name appicon.png at 100 percent of the quality of the original image.

I repeat this for each of my file sizes (check out my other post for more of the sizes) needed and we are done.  I just keep adding to this script for other images.

Hope this speeds your image building process.

Here is my full script:


Greg Mccormick is Founder of CRUSHMEDIA.NET, a full service mobile development company located in Manhattan.