Tag Archives: Android

Android, Linux

Android Systems Engineering: A Quick Look Under the Hood

1 Comment

Overview

Most of us we have become comfortable with the understanding that Android is a very specialized and customized version of Linux. The resulting level of customization does however render the OS sufficiently different that most general purpose Linux Systems Engineering expertise is of limited value. We also need to acknowledge that this simply means we need to get back to the foundations and explore / relearn from the bottom up (or top down – depending on your perspective).

If we take a quick survey of the multitudes of Android devices available, it is clear that there are both similarities and significant differences. Some of these are cosmetic and some are not, and one way to understand which is which,  is by dissecting the construction of the OS – and identifying where these modifications were interjected. From the bottom up we have:

  • The Android Kernel: The Android kernel is the engine at the bottom of the Android software stack that is responsible for everything that happens after the bootloader hands off control.
  • Core Android Services: When you hear about functions named Binder or AshMem, these are two of the core services that are responsible for managing memory, communication and process launching. Like the kernel these services generally are a key part of everything that happens in Android.
  • System Services/APIs: These are functions that provide more specialized functions and programmatic interfaces to the Android platform. This is the layer that applications start to touch.
  • Proprietary Device Drivers: A set of binary loaded drivers that enable the very general interfaces in the OS to talk to specialized hardware devices. These are not open source.
  • Dalvik Virtual Machine (VM): A specialized Java like virtual machine that isolates (or virtualizes) the APIs so that applications can be device independent.
  • AOSP User Interface/Apps: The standard user interface and core applications provided as part of the Android Open Source Project. This is a very small set of applications.
  • Google Applications: A larger set of Google specific applications, which include Gmail, Google Maps, Google Play Market, Google Talk, Google Voice, and Chrome.
  • Device Vendor Theming / Applications: A look and feel overlay on the user interface (also known as ‘skinning’) combined with a set of vendor specific applications. Nexus devices are not “skinned”.
  • Telecom Vendor Applications: A set of applications specific to a telecom vendor that provide some level of function that often duplicate functionality. Nexus devices do have telecom vendor apps.

From an Android Systems Engineering viewpoint, a few of these merit further discussion. Specifically we will expand on the Android Kernel, AOSP, and the proprietary drivers.

Android Kernel

The Android kernel is based on the Linux kernel, but has a number of Android specific patches that provide improved performance and function on a mobile platform. These patches are generally very specific to how Android manages memory, tasks, interrupts and timers to provide better performance and battery life than a more traditional Linux kernel. There has also been an evolving effort to reduce the amount of OS code running as a privileged user (i.e. root), which has been relatively successful, and has driven some kernel architectural changes. This is why Android privilege escalation attacks are so very uncommon.

AOSP: Android Open Source Project

The Android Open Source Project, or AOSP is the open source part of the Android system. The homepage for AOSP is at http://source.android.com/. AOSP includes the Android Kernel, a number of key OS services, the Android API functions and the Dalvik Virtual Machine.

Another way to look at this is that AOSP is the open source foundation from kernel to User Interface – excepting the proprietary apps, drivers, and ‘skinning’. To clarify – skinning is a process that layers a user interface theme on top of the standard Android look and feel. This is generally done by product vendors to create a brand style or appearance. Examples include Motorola MotoBlur, HTC Sense and Samsung TouchWiz.

The most important thing to know about AOSP is that it is open source (distribution, but not development) and it is capable of generating a fully functional Android Operating System without any special or purchased tools.

Android Proprietary Drivers

Although the Android OS is fairly generic for platforms, it is still necessary to have device drivers and a hardware abstraction layer (HAL) that maps a generic API interface to the device. Unlike the PC Linux movement, there is not a significant effort to develop open source drivers for Android hardware devices and for the most part, these are proprietary pieces of code that get installed on the OS image.

If your device is a Nexus device, drivers can be found at https://developers.google.com/android/nexus/drivers. For other Android devices the process is more complicated, and drivers are generally extracted from the factory device firmware.

Beyond Skinning

As the Android marketplace evolves (I hesitate to call it maturing), device vendors are increasingly working to develop product differentiation that would enable them to carve off their own walled gardens in the Android world. Initially this was achieved by changing the look and feel of the User Interface (UI) through skinning – which is a fairly cosmetic process. The upside to skinning is that it is also a relatively safe process with minimal risk of introducing vulnerabilities.

Over the last few years this process has progressed, and Samsung in particular has introduced and applied broad sets of  patches to the AOSP source code that add significant capabilities – such as Multi-Window on the Galaxy devices. These changes are fairly significant and end up touching a many parts from the kernel to the UI, with increasing risk of bugs and vulnerabilities. Of course the real question is whether this has actually happened.

Implications on Stability

My favorite device differentiator is stability, and the following statements make some relatively safe, but somewhat anecdotal assertions about stability / vendor support on smart phones.

  • Apple iPhones are fairly stable but introducing a major new OS version each year that is compatible with 3 generations of hardware may be having some impacts. Just for fun Google “iphone camera crashes” and see what you discover.
  • Google Nexus / Developer devices are more stable because of their minimalism. I have had a Nexus tablet for over a year and a Nexus phone for about 6 months. I have never had to power cycle either of them except for firmware patching / updates. The only app crash I have ever had is Facebook – and it happened once.  Based on discussions with other Nexus owners, this is fairly typical.
  • Google Nexus / Developer devices generally receive OS updates within days of announcement. This is because Google does this directly – and does not go through the device / telecom vendor channels (who have no business interest in maintaining a device already sold and obsoleted out of the sales chain).
  • Non-Nexus Android devices range from slightly flakey to very crashy (and barely functional), require regular reboots, and if they are updated – they can be up to a year behind the current version. Vendors typically lock the bootloader and discourage any third party OS development (AOKP and/or CyanogenMod for example).

I also understand that the plural of anecdote is not fact – but several of these assertions are supported by a recent academic paper that has some interesting findings on code provenance and vulnerabilities. More simply – the paper confirms that some Android phones are significantly more vulnerable / buggy than others, and who is to blame.

The vendors surveyed included Google Nexus, LG, HTC, Samsung and Sony. I will let you read the paper, but there are few tidbits worth sharing.

  • Google Nexus phones were used as the baseline since they have no third party software – and did provide the lowest number of vulnerabilities.
  • Sony customization produced significantly few vulnerabilities than the other three vendors.
  • Of LG, HTC and Samsung devices – between 65% and 85% of identified vulnerabilities came from their 3rd party code and customizations.
  • Samsung Galaxy S2 and S3 take the dubious honor of having the greatest number of vulnerabilities (see table 5 – S3 had 40 vulnerabilities versus the Nexus 4 with 3).

Conclusions I personally drew from this paper include: a) Whatever Sony is doing in their code development QA process – they should keep doing it, b) Whatever Samsung is doing in their code development QA process – they should stop doing it and fix it, and c) This data supports some of my anecdotal assertions above and actually adds detail.

Anecdotes are interesting, and as story tellers we can more easily relate to them versus raw data. However – occasionally data can support the anecdotes and then the combination can become useful.

Bottom Line

For the aspiring Android Systems Engineer, this quick look under the hood is intended to provide you with a few reference points you can related back to your Linux Systems Engineering understanding.

For the truly ambitious, the next step is to download the AOSP source and dig in…

Reference

Android, Chromebook, Linux

Howto: Android Development on Chromebook

18 Comments

Background

One the primary reasons I got the Chromebook was to support a broad range of development options on a Linux platform. The risk is that since it is a Chromebook, it is most definitely not a general purpose Linux environment – and some things may not be practical. Even inside a chroot jail, there are weirdnesses since it inherits the kernel and devices from the platform OS – ChromeOS.

In any case the following is a process I developed to implement an Android application development environment. There are many equally valid solutions, and perhaps better solutions, but this is a working / tested example of how to get from point A to point B.

Approach

Since the base platform I am using is a Chromebook 14 – with a x86 Haswell (64bit), Intel binaries will work fine – making life a quite a bit simpler. Unfortunately, it is not possible to install anything like Eclipse or Debian packages directly to the ChromeOS since the developers have (purposefully) not included most of the traditional shared libraries used in Linux. This minimalist approach to the ChromeOS means that it has a minimal attack surface for malware, but minimal opportunity for us to hack the OS.  As a point of trivia, ChromeOS appears to be based on a Gentoo build model – but it has been scrubbed clean of anything extraneous to the ChromeOS function.

However, since we are lucky enough to have a relatively polished / low pain solution to installing a chroot jail version of Ubuntu – Crouton, our system level approach will be:

  1. Switch the Chromebook to developer mode
  2. Install a Crouton based chroot jail version of Ubuntu on the SD-Card
  3. Install Oracle Java in the chroot jail (along with all of the rest of the pieces)
  4. Install BitTorrent sync to create a shared workspace for Android Studio
  5. Install adb and fastboot and verify operation with an Android device.
  6. Install Android Studio / test with Android device.

Note: All of the instructions below are based on name of my user (joeuser), the name of my SD-Card (chrome-32), and particular versions of the install packages. You will need to modify for your respective names / versions.

Step 1 – Developer Mode

Developer mode is a way to unlock your Chromebook a bit. Of course it is much less secure than the default ChromeOS mode, and setting developer mode means everything on the platform is erased – except the SD-Card . Additionally, switching it back to default mode will also clear everything (again). So any files you want to be persistent should be stored up in the Google drive or on the SD card. After you have mentally prepared yourself for that, take a shot at developer mode – it really is much more interesting than lockdown mode. The details are on my Chromebook Cookbook page. While you are there, spend some time and figure out how to switch between virtual terminals and set the password for ‘chronos’.

Step 2 – Ubuntu on Crouton Chroot

The default Crouton install installs the Crouton tools in /usr/local/bin and the chroot jail in /usr/local/chroots. With a 16GB internal hard drive, it is not practical to risk using most/all of the available local space for a chroot jail – so we should plan to store it off to the SD-Card. There are two basic approaches we can use to accomplish this. The first is to use command line arguments with the Crouton tools to point it at /media/removable/chrome-32 (which happens to be the name of my card). Another option is to symbolically map these directories to corresponding directories on the SD-Card. The first approach means that every time the crouton scripts are used, the command line arguments are needed, and the second approach means it is done one time up front (I recommend the second approach as the less stupid approach).

Open a chrosh (chrome shell – ctl-alt-t), and enter (with corrections for your sd-card name):

shell
cd /media/removable/chrome-32
sudo mkdir bin chroots
cd /usr/opt
sudo ln -s /media/removable/chrome-32/bin/ bin
sudo ln -s /media/removable/chrome-32/chroots/ chroots
ls -al

The last instruction should show the local directories with the symbolic mapping to the sd-card locations. This ensures that both the crouton tools and the chroot jail is installed to the SD-Card enabling a much easier restore if you somehow clear your system (it happens to me at least once a week).

FYI – If that happens, the Crouton install can be restored by recreating the symbolic links above. That’s it.

From here install a crouton chroot jail ubuntu with the following:

sudo sh -e ~/Downloads/crouton -r raring -t unity
sudo startunity

Note that in most cases I recommend Precise due to its stability, but in this case I went with Raring since it has support for ADB and Fastboot in the Ubuntu repository (and Precise does not). From inside the Ubuntu install, open a terminal and enter:

sudo apt-get install ubuntu-standard
sudo apt-get install ubuntu-desktop
sudo apt-get install ia32-libs
sudo apt-get install synaptic

Shutdown the Ubuntu chroot jail. You now have a fairly complete and clean Ubuntu Raring install, and this would be a good point to make a backup. Instructions are on the Crouton Cookbook page. Once again – when the backup is done, move it to Google Drive or the SD-Card for safekeeping.

Step 3 – Install Oracle Java

From the ChromeOS interface (VT1), you can download the  Java 7 JDK for 64 bit Linux – grab the tar.gz package from Oracle (not the RPM). It will download into the Downloads directory, which incidentally is mapped to the Downloads directory inside the Crouton chroot jail.

After the download is complete, switch over the Ubuntu interface on VT3, open a file manager and copy the Java 7 tar.gz package from Downloads to the home directory. Right click and extract. It should create a directory name something like ‘jdk1.7.0_45’ in the home directory. Open an editor and open ‘.bashrc’ and append the following:

PATH=${PATH}:/home/joeuser/jdk1.7.0_45/bin
JAVA_HOME=/home/joeuser/jdk1.7.0_45

This will make it easier for *some* apps to find the JDK. The JDK tar.gz file in the homedir can safely be deleted after this is done.

Step 4 – BitTorrent Sync

On the ChromeOS interface (VT1) download the Linux/64bit install package for BitTorrent Sync from http://www.bittorrent.com/sync/downloads. Open a terminal with <ctrl-alt-T> and enter:

cd /media/removable/chrome-32
sudu mkdir btsync
sudo chmod 777 btsync
cd btsync
mkdir android-studio

This creates a target sync directory that is readable / writable / executable to everybody for syncing. We will use it later.

When the download is complete, copy the tgz file to the user home directory(from the Ubuntu interface) . Extract the files in the home directory. This will create a directory that looks like ‘~/btsync_glibc23_x65’. On the Unity Desktop, click on the gear in the upper right corner and select ‘Startup Applications’. Under Command, browse to the btsync directory and select the ‘btsync’ app. The will configure the app to startup when the chroot is started – similar enough to a service for our purposes. After this is done, the tgz file in the homedir can also be deleted.

Start the app by double clicking from the file manager or reboot the chroot jail to force the app startup (and validate that it is configured correctly). On either the ChromeOS or Ubuntu interface, open a browser with URL ‘localhost:8888/gui’ to confirm that Bit Torrent sync is running. Configure according to directions – using the ‘bysync’ directory (on the SD-Card) we created above as the target. You will also want to create another endpoint to this share on a desktop, server, or other laptop to ensure your data is offloaded from your Chromebook.

Step 5 – ADB and Fastboot

ADB and Fastboot are really a make or break part of effectively using the Chromebook for Android development. Note that in Precise, the adb and fastboot packages need to be retrieved manually from the Debian repository. For details refer to the Ubuntu Cookbook page. In Raring, we can use the easier method shown below.

From the chroot Ubuntu interface on VT3, and open a terminal. Enter the following:

sudo apt-get install android-tools-adb
sudo apt-get install android-tools-fastboot
adb version
fastboot help

The last two lines confirm that both adb and fastboot are operational. The true test is to now plug in an Android test device and enter (this may take a couple of tries):

adb devices

If everything if functional the adb server should start and the attached device will be identified. Note that if the Android device is reasonably current, it will require onscreen approval before it connects to adb.

Step 6 – Android Studio

The last piece in this puzzle is Android Studio. From the ChromeOS interface, download the Linux/64bit install package from http://developer.android.com/sdk/installing/studio.html.

From the chroot jail (VT3) open a file manager and copy the Android Studio tgz file to the user home directory. Right click on the file and extract the files in the home directory. This will create a directory that looks like ‘~/android-studio’ with an ‘bin’ subdirectory. Once again, after this is complete the tgz file in the homedir can be deleted.

In order to make the launch script easier to find, we will put it on the PATH system variable. In the user home directory, edit ‘.bashrc’ and append the following:

PATH=${PATH}:/home/joeuser/android-studio/bin

In some cases, I discovered that the launch script did not seem to be picking up on the .bashrc updates, so it was necessary to define the JDK_HOME more explicitly. Use your favorite editor to open ‘~/andoid-studio/bin/studio.sh’. Right below the ‘#!/bin/sh’ line add the following:

JAVA_HOME=/home/joeuser/jdk1.7.0_45″

Open a terminal and enter ‘studio.sh’ to confirm that Android Studio was found and executed. If / when prompted, select the directory for the Oracle Java JDK. Android Studio should launch.

As a test we are going to create a new project (mostly with the defaults), except for project location. For the location, navigate to ‘media/removable/chrome-32/btsync/android-studio’. This will put the project files in the sync directory, which will then synchronize the project to your others systems on share. This is all part of that concept that everything should be stored in some “cloud” or at least off device.

At the project screen, create a new project and when it comes up on the screen, click the green arrow button at the top of the screen. The connected Android device should show up as an option, select it and go. Alternatively, you can create an emulator and use it. The app should install, run and show ‘Hello World’ on the interface.

Version Note: As part of my testing, I noticed that my initial version of Android Studio had functioning menu drop downs, but after I updated to the current version (0.32) the menus would no longer drop down. I confirmed this with both Precise and Raring on Unity. It may be worth testing Gnome as some point, or it may be fixed in some upcoming update to Android Studio. Overall – it did not prevent me from doing most activities since the control bar was fully functional.

Lastly – go to another node on your BitTorrent Sync share and confirm that the project was created and migrated to that system.

Wrapup / Notes

Overall the Android Studio IDE is fairly functional and well structured. I was able to test on both an external device and the emulator without any issues.  From a practical perspective, I am actually surprised at how easily this came together on a Chromebook. As far as the menu issues, this appears to be an Android Studio / Unity issue which could be resolved by using Eclipse/Android ADT tools or switching to a different window manager.

In Summary – This Chromebook Android Environment provides me with a very slick and portable Android Development environment without a lot of compromises.