This article is an exploration of web service protocols, common data interchange formats, and best practices. If you’re interested in creating performant mobile applications that connect to the web, read on!
Choosing a Protocol and Data Format
Selecting a proper protocol and data interchange format to pass data between your mobile app and the ‘net is one of the most important decisions to make during the development process. The most common and widely used protocols are REST, XML-RPC, and SOAP.
All of these protocols transport data over the HTTP protocol. The XML-RPC and SOAP protocols are XML-based, while REST services can rely on different data formats, the two most common being XML and JSON. SOAP in particular is widely appreciated and adopted by enterprise applications because it strictly enforces data models and it describes public interfaces through WSDL, thus making some development tools (such as Microsoft Visual Studio for .NET) able to automatically set up objects and methods to consume the services just by adding a reference to the service WSDL to the project.
XML however is by-design less efficient than JSON when operating in a bandwidth limited network scenario like mobile development. Mobile data networks (WWANs) are usually less reliable than wired (LAN) or wireless (WLAN) networks because clients on-the-move may easily drop in-and-out of coverage areas. Also, some users do not subscribe to “flat” data plans and are actually charged by network traffic.
So, while developing for mobile you should always opt for simplicity and traffic reduction.
Let’s compare plain XML to JSON with an example response:
XML
<ServiceResponse> <ReturnValue>Hello, device!</ReturnValue> </ServiceResponse>
JSON
{
returnValue: "Hello, device!"
}
As you can see, JSON carries a smaller payload because it just needs to wrap its data with brackets (curly for objects, square for arrays) and quotes, while XML needs a full root node and opening-closing tags for each element or group. Saving on those tags can significantly trim down the response payload, and the more data you are transferring, the larger the payload “padding” will be with XML.
Moreover, SOAP and XML-RPC require more complex XML data structures (like the SOAP “envelope”), adding extra weight to the payload.
For this reason, I recommend choosing JSON for your app’s data interchange format and REST for the transfer protocol unless XML, SOAP or XML-RPC are explicitly required for your needs. Should you need to enforce a data model, you can achieve that with JSON as well by using the JSON Schema or WSDL 2.0.
For references and further reading about the discussed protocols and data formats, check out the following:
- The JSON vs. XML Debate
- JSON Overview
- XML Overview
- XML-RPC Overview
- SOAP Overview
- The JSON Homepage
- More on WSDL
Network Security Considerations
Since most web services operate over public networks, protecting the data that will be exchanged between the services and the app is one of the most critical issues you need to address. Many web services provide remote functionality to authenticated users, so private user data and authentication credentials need to be transferred over the network as well.
Personal data, account data, and credentials should never travel in cleartext form over a public network, so this means your Web Services and apps should implement cryptography. The good news is that most common protocols transport data over HTTP, so you can rely on the well known, trusted, and robust HTTP SSL/TLS protocol (HTTPS) that enjoys widespread support both server-side and client-side.
Enabling SSL on common web servers is fairly easy (there are a lot of useful resources for Apache, IIS, and Nginx). All you need to do is buy an SSL certificate and set it up on your server. In a worst case scenario, you could also issue a self-signed certificate that will bring you free encryption, but I would not recommend doing so. Some CAs issue low-priced certificates, and the benefits you gain from working with a trusted, signed certificate are worth the extra cost (i.e. preventing man-in-the-middle attacks by utilizing the PKI). Using self-signed certificates could also require extra coding or even application-specific “hacks”, since some network libraries will refuse untrusted HTTPS connections (check these articles about Android, #1 and #2, and about iOS).
Enabling SSL on client side is a no-brainer: common programming languages typically have HTTP classes and libraries that provide native HTTP-SSL support and will “automagically” take care of encryption just by using “https://” URLs rather than “http://” URLs to consume web services.
If you opt for a low priced SSL certificate, just make sure that the devices and operating systems you want to develop for have the issuer CA root certificate bundled in out-of-the-box, otherwise they will not trust the certificate.
You should never ask users to patch, alter or hack their systems in order to run your apps, and if you do, do not expect many of them to comply.Web Service Authentication
Authentication for REST Based Services
Token Based Authentication
Web Services transactions are meant to be atomic, interoperable and scalable, so Web Services are usually stateless (check this interesting discussion about the meaning of “stateless” connections).
Many web services provide user-related functionality and access to sensitive information. Naturally, doing so requires authentication. Because such transactions are stateless, you traditionally needed to sign requests with a user-specific key, providing authentication with each remote method call. Even when transferring data over HTTPS, user credentials are at the most risk when they are transmitted over a public network. Encryption is sometimes broken.
This dilemma is where token-based authentication like OAuth comes in handy. Using token-based authentication, you need to send a user’s credentials only once. If the user is authenticated successfully, they will be provided with tokens that can be used to authenticate subsequent remote method calls.
A token’s validity spans over a limited lifetime and can be revoked anytime by its issuer (i.e. server-side). With token-based authentication, you can also share the same user credentials between different apps and services without the linked services/apps ever knowing the real user credentials. Further, this authentication method safely keeps a local cache of the credentials on the user’s device, so that the user can be “remembered” and will not need to authenticate each time he will use the app (typing complex passwords on handhelds can be a real pain and can badly damage app reviews).
OAuth Based Authentication
OAuth is the most common token-based auth framework and it has been adopted by big players like Google, Twitter, Facebook, and so on. By allowing users to reuse their existing accounts, skipping annoying registration forms and keeping control over access to their personal data you can significantly increase your user base.
Some OAuth providers require their own SDK to be imported in your apps to enable authentication, otherwise there are a lot of ready-made OAuth libraries around that you can plug into your app. There are also frameworks like oauth-php available to build custom OAuth providers.
I would suggest signpost as an Android OAuth client library and oauthconsumer for iOS.
All these libraries come with useful examples and are easy to implement. However, building the signpost sources on your machine or importing them into your app can be a bit of a pain, but you don’t actually need to go through that process. Instead, you can just import the JAR binaries into your project and you should be set. On iOS, this would be similar to importing a static library (*.a file) into your project.
Authentication for SOAP Based Services
The most common authentication method for SOAP based services is a SOAP extension called HTTP Basic Authentication (sometimes called Digest Authentication). This is a standard, well-supported procedure that is integrated in all the most common web servers like Apache, Nginx, and IIS. It is based on a username-password pair which has to be sent to the server through HTTP headers.
Normally, you will not have to manually implement basic auth in your applications, as it is widely supported in the most common programming languages as well. The .NET framework relies on the NetworkCredential class to supply basic and digest auth credentials to HTTP requests, PHP supports it through cURL, and Java through Authenticator.
Should you have to implement Basic Auth yourself, you just need to add this header to your requests:
Basic username:password
The “username” and “password” values need to be base64-encoded.
Please notice that HTTP Basic Auth is best used in combination with the HTTPS protocol, as it transfers credentials in plaintext form. Digest Auth is a little safer, because it actually hashes the password value, but HTTPS is recommended anyway to avoid hash bruteforce attacks.
More details and specs on this subject can be found in the IETF RFC 2617 memo.
Native Platform Authentication Libraries
When choosing an authentication method or integrating well-known services (such as social networks), you should also take into account the native libraries which are built into many advanced platforms like iOS and Android. This can save you a lot of time and many lines of code.
Android provides a nice framework to centralize user account managment, the AccountManager class. The official Android Developer guide provides some nice documentation for this class, along with some tips to integrate OAuth 2 or write your own “Custom Account Type”.
With iOS 6, Apple has introduced a new Social Framework to integrate major services like Twitter, Facebook, and Sina Weibo at an OS level. Even if you don’t need to integrate these services into your application, you might find a suitable way to take advantage of the built-in authentication methods via customization.
Optimization Tips
Data Compression
When developing for mobile applications, it’s important to keep your data payloads as low as possible. This section will discuss several strategies for doing so.
Archiving Data
ZIP compression can significantly reduce text and also binary data weight, and is well-supported on most platforms, so make good use of it if you need to transfer large data over mobile networks. Useful libraries to manage ZIP files are available for Android (i.e. Decompress) and iOS (i.e. Ziparchive).
Efficient Media Streaming
Should you need to stream audio/video content to your mobile apps, choose advanced streaming platforms that allow them to scale streams depending on network/device performance, such as Apple’s HTTP Live Streaming (HLS) protocol. Otherwise, favoring responsiveness over media quality is usually a good choice. Play around with different video and audio compressors and settings, optimizing as much as you can. You could also group devices by kind (small-screen handhelds, wide-screen handhelds, and tablets) and deliver different content for each kind.
The Problem with HD
Many mobile devices feature HD displays, but is HD content on small screens really worth the extra bandwidth overhead? Be honest about the relevance of media quality in your apps and try to find the best balance between quality and weight.
Local Caching
Let’s suppose you are to code a reader app for an online newspaper. First of all, you should always let your users operate offline whenever possible. While some apps do always require an active network (i.e. messaging services), many others should use the network only to download data packages and cache them on the device.
This will improve application performance, save user’s money on non-flat mobile data plans, and make the app usable when the network is not available (e.g. during flight).
When downloading content on devices that mount external storage (e.g. memory cards), you should let users choose where to store downloaded content (i.e. internal or external storage) or simply prefer external storage anyway. Entry level devices usually feature rather small internal storage and may become unstable or slow when internal storage is full. If your apps are taking up a lot of storage space, they are likely to be uninstalled to save room.
Chunking Data
Let’s go back to the reader app example. Each article is a single piece, and although articles could be linked together, there is no reason why you should not let users begin reading articles even while additional content is still downloading. So, pack each article’s files (text, pictures, attachments and media) in separate ZIP archives and provide a web service method for the app to retrieve the list of available articles. Have your app download ZIP packages one-by-one and then make them available in the app as soon as each one has been downloaded and while others are being downloaded in the background. This is a great performance and user experience improvement compared to waiting until the entire payload has downloaded! You could also let users choose which packages they want to download or not (allowing them to save space, time, and traffic) and allow them to remove single packages to save storage space without removing the whole app.
The following is a response from the demo server app bundled with this article:
{
apiversion: 1,
status: 0,
packages:
[
{
file: "pack01_01_01.zip",
package: "pack01",
appversion: 1,
revision: 1
},
{
file: "pack01_02_01.zip",
package: "pack01",
appversion: 2,
revision: 1
},
{
file: "pack02_01_01.zip",
package: "pack02",
appversion: 1,
revision: 1
}
]
}
Change Management
Keep in mind that your app and previously released content could evolve over time.
Provide a “revision” flag for each package in the list method, this will come in handy to release updates, fix bugs in the content, and implement autoupdate functionality in your apps. Even if you are not currently planning to implement auto-update, think forward and put the revision flag in your list anyway for future development.
You should also include an “app-version” flag in your packages list. Should you issue a new major release of your app that carries breaking changes in content format, this will let you provide content for both newer and older apps through the same Web Service.
Fault Tolerance and Remote Data Recovery
While developing service-based apps, network and device fault tolerance should also be taken into account. Mobile data connections are usually less stable than cabled ones, apps can be uninstalled and reinstalled and devices can be lost, replaced with newer ones or factory-restored. Users should be given the possibility to restore apps and content in an easy way and platform developers are offering several useful ready-made solutions to these issues.
Monitoring Network Status
Remember to always check network status before calling remote services, or carefully handle network I/O errors and properly inform your users when the desired task cannot be accomplished due to the lack of an active data connection. Should your apps need to always “work online”, you may want to put a watchdog in your apps that constantly monitors network status and fire an event whenever a change occurs. You may also want to inform users of possible extra costs when transferring large amounts of data over 3G or roaming networks rather than Wi-Fi.
On Android devices, this task can be accomplished through ConnectivityManager, and through SCNetworkReachability on iOS (also check the provided sample app).
Restore Content and Settings
Both Android and iOS provide useful APIs to handle remote cloud backup of user app data that you should keep in mind. Check the iCloud API documentation for iOS and the Android Backup Service documentation for Android. Together with built-in backup services, you also get good data security features. However, you must be careful not to overload backup services with redundant or unnecessary backups.
You could also implement custom remote data backup in your web services, but I strongly recommend you to stick to standards and built-in platform APIs. They will usually save you time and they are actively maintained by platform software engineers. OS patches are also more likely to be promptly installed when released.
Restoring In-App Purchases
If you deliver paid content through in-app billing in your app, allowing users to recover their purchases after app reinstalls and device recoveries is mandatory. Fortunately, iOS and Android do have built-in APIs to handle this scenarios as well.
When your in-app-purchase-enabled apps run for the first time (or the first time after a reinstall), you should run a purchase-restore check procedure. Both iOS and Android provide nice official documentation regarding this matter.
When developing for Android, remember that only “managed” items can be restored at a later time, and they are only available on a one-time-per-use purchase. This implies some considerations that apply to iOS development as well.
Let’s suppose you are to develop a role-playing game and want to allow players to buy items like health potions through in-app billing. First of all, since users can buy as many potions as they want, they cannot be “managed” items, so their purchase transactions are not permanently stored. Also, if a user buys 20 potions and uses 10, then uninstalls and reinstalls the game at a later time, restoring purchases through a simple, standard procedure would place 20 potions back into the user’s inventory, 10 of them being an unintended free gift from the developers.
So, you may need to implement your own custom Web Services and app methods to handle transaction storage and recovery in complex scenarios or edge cases.
Designing for the Future
Deadlines and budgets will often cut you off and not let you follow all the best practices explained in this article. But even if you are forced to stick to a smaller subset of features, think forward, spend some time in good design, and pack your methods and classes into libraries that you can reuse and extend later. Leave stubs when you feel there is room for further development. Also try to enforce backwards compatibility when you extend and improve your libraries, so that older applications can be patched as well.
This is an example of a stub:
public void sendData(Object data) {
if (validate(data)) {
client.send(data);
}
}
// Stub
public boolean validate(Object data) {
// TODO - Implement data validation
return true;
}
Conclusion: Where to Go From Here?
If you are a novice developer or have never developed service based applications, you should start from the provided sample application as a good exercise to improve your knowledge and turn it into a real-world application applying all the concepts explained in this article one at a time. Alternatively, start a new application from scratch and integrate it with some existing service provider (Facebook, Twitter, Last.fm, or Dropbox would be a good starting point) following the same schedule.
If you have already developed some service and network based applications, you could review your existing code and improve them according to the principles explained above, keeping track of each improvement and impact on performance and responsiveness.
Don’t forget to check out the linked resources also, as they will take you deeper into the core of each issue, and to download the sample server and client applications provided with the article.