Use Android.mk/Application.mk for NDK projects in Android Studio

Android Studio (AS) is straight for Android apps, which in my opinions, is much better than that good old Eclipse. Now Eclipse has solely been for tracing Linux kernel for me; its indexing capability is much better than vim+cscopes after all. Having used AS since version 0.1, now I am trying to develop an App with Android NDK on it.

A first difficulty may come if you are trying to build non-trivial native codes under your JNI folder. For example, I am trying to build 3 native C libraries along with my own one, which obviously cannot be automatically built with the built-in support of gradle in AS. As of now, the automatic NDK build is probably for some simpler project with just a few native source files. Without writing the Android.mk for ndk-build, it's not possible to build them correctly.

An recent and excellent article about the new NDK support in latest Android Studio has been helping me a lot here: http://ph0b.com/new-android-studio-ndk-support/. However I can't get everything working if I want to use Android.mk/Application.mk for NDK projects in Android Studio.

Here is how I did it, as of now, through the latest gradle-experimental plugin: com.android.tools.build:gradle-experimental:0.4.0. I am using this on both AS 1.5.1 stable and 2.0 preview.

Instead of setting jni srcDirs to an non-existing folder as in ph0b's article, which has an side effect of not showing the jni folder at all in the project panel in AS, I'm setting it to the correct one:

android.sources { main { jni { source { //srcDirs = ['src/main/none'] /*non-exiting*/ srcDirs = ['src/main/jni'] } } jniLibs { source { srcDirs = ['src/main/libs'] } } } }

And then skip the NDK module through the gradle script (Module: app) like this:

/* skip the automatic gradle NDK build */ tasks.all { task -> if (task.name.contains('LibmylibSharedLibrary')) task.enabled = false }

provided that your android.ndk setting is something like this:

/* native build settings */ android.ndk { moduleName = "libmylib" }

And now the gradle build will simply skip the automatic NDK build. You should follow ph0b's article to include the ndk-build instead:


/* call regular ndk-build script from app directory see http://ph0b.com/new-android-studio-ndk-support/ for more details */ task ndkBuild(type: Exec) { if (Os.isFamily(Os.FAMILY_WINDOWS)) { commandLine 'ndk-build.cmd', '-C', file('src/main/jni').absolutePath } else { commandLine 'ndk-build', '-C', file('src/main/jni').absolutePath } }

Now everything just works fine!

ps. AS's NDK support and gradle-experimental plugin change a lot since it's under active development (?) for now. This is proven to be working in AS 1.5.1 stable and AS 2.0 preview 4, as of now (Dec. 30, 2015).

Parallel grepping a large code base with progress display (find|pv|xargs -P)

xargs can be used with parallel mode: -P to make best use of current multi-core CPU. When grepping a large code base for some keyword, it is much quicker if we can make full use of the cores. But of course the overall speed not only depends on parallel grepping but also the speed of the storage.

For a 4-core computer, you can easily accelerate your grepping speed by using find with 4 parallel grep, each with 10 files at a time, at current folder:

find . -type f | xargs -P 4 -n 10 grep -Hn "what_to_grep"

Make it an shell function and you're good to go:

function ppg { find . -type f | xargs -P 4 -n 10 grep --color=always -Hn "$1" } $ ~/build_trees/android-current/kernel
$ ppg ehci_resume
./drivers/usb/host/ehci-spear.c:53: ehci_resume(hcd, false);
./drivers/usb/host/ehci-pci.c:365: if (ehci_resume(hcd, hibernated) != 0)
./drivers/usb/host/ehci-msm.c:175: ehci_resume(hcd, false);
....
(of course you can alternatively use some other search tools by indexing the code)

Now what if you want to see the progress? When you consider using parallel grepping, it often means it takes a long time to scan every file for what you want.

A good tool "pv" is already available. However, for pv to make sense to you, you should give it a hint of how large your data set is. In other words, you have to count how many files are there to grep.

To recursively count files under a folder, a typical and straightforward way is to use find with wc -l, with the first part being identical to what is to be fed into xargs in the first example:

find . -type f | wc -l
For instance, the total file count under frameworks/ folder of an android tree is about 30k, which takes about 5s on my notebook:

$ time find ~/trees/android/frameworks -type f | wc -l
30503

real 0m4.950s

Combining everything together, we make it another function:

function ppgv { find . -type f | pv -cN GREP -i 0.5 -l -s `find . -type f | wc -l` |xargs -P 4 -n 10 grep --color=always -Hn "$1" }

When searching something that doesn't exist on a kernel folder, the one with pv takes a little bit longer:

$ time ppg doesnt-exist1

real 0m1.168s

user 0m0.140s

sys 0m0.380s

$ time ppgv doesnt-exist2

     GREP: 45.3k 0:00:01 [  39k/s] [================>] 100%            

real 0m1.339s

user 0m0.500s

sys 0m0.524s

While we can benefit by the progress bar and also ETA, the extra cost doesn't seem to be too much.

Mixing variables in single/double quotes

Say you have a task to find files under a directory, and then execute another command with the result. This is an easy case. You find-exec to achieve it:

find <path> -name 'what_to_find' -exec <what_ever_command_to_execute> \; 

Oh, use {} in the  <what_ever_command_to_execute> if you intend to use the just-found file.

However, if what you want to execute is a complicated bash command which deals with the files just found, it is advised to use $0 instead of {} to avoid any weird characters that may happen to be part of the filename.  What weird characters? Filenames with double quotes, dollar signs, escape combos.. etc. So now we have this form:

find <path> -name 'what_to_find' -exec bash -c <your complicated commands with $0 > {} \;

If your command includes bash functions, then it is not possible to use the previous form. Use bash -c instead.

Then problem comes. If this 'find' command is to be embedded in a bash script, chances are you will have variables which define paths to find. What's worse, variables to be passed to the command to be executed by find. See below example:

find $WHERE_TO_FIND -name $WHAT_TO_FIND -exec bash -c <your complicated commands with $0 and $ANOTHER_VAR> {} \;

Now, $WHERE_TO_FIND, $WHAT_TO_FIND, and $ANOTHER_VAR are to be expanded before running the find command. But $0 is to be passed to 'bash -c' which should not be expanded before running find!

The key to the solution would be mixing single quotes with double quotes. Just remember one thing: variable expansion won't happen inside single quotes. Single quote won't expand variables, while double quotes will. Mixing single quotes with double quotes are easy: just stick them altogether:

$ echo "this is inside double"' while this is inside single'
this is inside double while this is inside single

By default shell will expand variables. So you can add variables too:

$ COMMA=,

$ echo $COMMA

,

$ echo "this is inside double"$COMMA' while this is inside single. $COMMA will not expand.'

this is inside double, while this is inside single. $COMMA will not expand.

This entire string will be treated as one word by shell, and then passed as the first parameter to 'echo' command.

And you can insert unmatched single quote in a double quote, or vice versa:

$ QUOTES="I'm"
$ echo $QUOTES
I'm

$ QUOTES=' a double quote"'
$ echo $QUOTES
 a double quote"

Mixing them together will be like this:

$ QUOTES="I'm"' a double quote"'
$ echo $QUOTES
I'm a double quote"

Now we have everything we need.

Here comes a practical example: use adb shell to copy all *.gcda files under /sys/kernel/debug to a specific folder, using 'cat' instead of tar or cp. This is used to copy GCOV kernel coverage data from an Android device.

It comes with another hierarchy: adb shell 'command to run on android device'. You have to make sure the command as a whole is passed as a single word to adb shell. Final working command would be:

adb shell "busybox find $GCDA -name '*.gcda' -exec sh -c '"'cat < $0 > '$TEMPDIR'/$0'"' {} \;"

Let's see how those mixing single- and double- quotes work step by step:

1. we feed in $0 (the result file path by find) to stdin of cat and save the stdout to the same path under $TEMPDIR on the android device. We don't want to resolve $0 now (because it will be this script's filename if we do it now), so the $0 part should be quoted in single quotes. (see red part)
   adb shell "busybox find $GCDA -name '*.gcda' -exec sh -c '"'cat < $0 > '$TEMPDIR'/$0'"' {} \;"
2. Now make sure commands to be passed to 'sh -c' is one word: (see blue single quotes)
   adb shell "busybox find $GCDA -name '*.gcda' -exec sh -c '"'cat < $0 > '$TEMPDIR'/$0'"' {} \;"
3. Protect everything else with double quotes: (see 2 underlined sections)
   adb shell "busybox find $GCDA -name '*.gcda' -exec sh -c '"'cat < $0 > '$TEMPDIR'/$0'"' {} \;"
4. Mixing everything into one word, so that everything after adb shell will be passed to shell in the android device. The first underline is part 1 by double quotes, part 2 is the red section with single quotes, and then a variable without any quotes as part 3, then another single quote in red as part 4. Finally the second underline section as part 5 quoted in double quotes.
   adb shell "part 1"'part 2'$part_3'part 4'"part 5"

This works as expected:

1. adb shell sends the actual command to run on android device, 
2. find in busybox is to be used to search for all *.gcda files under $GCDA folder
3. and passes each result path to sh
4. sh then runs cat, which reads the just-found file represented as $0 and saves each bit into the same path under $TEMPDIR on the android devices

Annotating date on a photo based on EXIF using imagemagick

To annotate date on a digital photo is a breeze. Most cameras (including probably ALL DSLRs) already remove the function to print the date on a photo probably because it is ugly and useless. However, you may need it one day.

How to do it is extremely easy by extracting date from EXIF and print it on a photo using command line:

$ convert your_photo.jpg -font Ariel 96 -fill white -annotate +100+100  %[exif:DateTimeOriginal] output.jpg

This will print the date on the northwestern corner. To print it on a more common place like the good old times, use:

$ convert your_photo.jpg -gravity SouthEast -font Ariel -pointsize 96 -fill white -annotate +100+100  %[exif:DateTimeOriginal] output.jpg

You may want to list available fonts on your system by

$ convert -list font

You may also want to see available EXIF info which you want to print:

$ identify -format %[exif:*] your_photo.jpg 

As usual, you can resize the photo by the way:

$ convert your_photo.jpg -gravity SouthEast -font Ariel -pointsize 96 -fill white -annotate +100+100  %[exif:DateTimeOriginal] -resize 1024x1024 -quality 90 output.jpg

Example photo below:

A more complete example with a bash script is available through this site if you want batch processing. The LCD font is pretty nice :P

Brief steps to convert HTC one to google edition

(This is only meant for taking notes for personal use. Please google for more information.)

This gives a rough idea about all those necessary steps to convert HTC one to google edition.

1. unlock bootloader (go to htcdev.com, will void warranty)
2. install cwm recovery (fastboot flash recovery), this is to root your phone (still HTC software)
3. root it (use CWM to install SuperSu zip)
4. make it S-OFF (security-off) (run http://rumrunner.us/, for HBOOT version 1.55)
5. change to super CID (fastboot oem writecid 11111111)
6. install google edition (follow http://forum.xda-developers.com/showthread.php?t=2358781)

Now it's pure google. Still a nice phone indeed. All I need is actually a rooted one so that I can run Ubuntu on my phone. I converted it because I do prefer google edition which is cleaner. 

cmus player on-screen display in python

I'm actively using the C* music player (cmus) as my main music player on my working machine. It's a powerful console music player which suits best especially I mainly work in consoles. However it would be convenient if there is on-screen display (OSD) on my Gnome desktop when I am working on another window/console.

Here is a version implemented in python using the notify-send program:

status_display.sh

#!/bin/sh ~/.cmus/status_notify.py "$@" > ~/.cmus/statusinfo&
status_notify.py

#!/usr/bin/env python """ ============================================ CMUS status update program using notify-send ============================================ Example status update as program's parameter list status playing file /data/music/Zeb-Jesterized/12-Zeb-Spy From Cairo.mp3 artist Zeb album Jesterized tracknumber 12 title Spy From Cairo date 2000 duration 322 Usage: Create a shell script at ~/.cmus/status_display_program.sh and 'chmod +x' it: #!/bin/sh program1 "$@" & (replace program1 with path to this script) In CMUS: :set status_display_program=/home/user/.cmus/status_display_program.sh Reference: http://cmus.sourceforge.net/wiki/doku.php?id=status_display_programs """ import sys, os from subprocess import call if __name__ == '__main__': # basic checking if len(sys.argv) % 2 != 1 or len(sys.argv) < 2 \ or sys.argv[1] != 'status': raise Exception('please set this program as CMUS status update program') # make it a dict s = sys.argv[1:] status = dict(zip(s[::2], s[1::2])) osd_info = "[%s] %s (%s-%s)" % \ ( status['status'], os.path.basename(status['file']), status['artist'], status['title'] ) # --hint parameter is for gnome-shell's buggy notification.. anyway call(['notify-send', '--hint=int:transient:1', osd_info])

Unbalanced IRQ enable/disable in pm.c during system suspend/resume

When doing Android system power stress test (entering low power state and exiting it repetitively), I found that sometimes RTC alarm won't wake the device up.

The wake-up mechanism is easy. It's from Android framework:

// using alarm in Android framework // creating a repeated alarm every 30 sec AlarmManager am = (AlarmManager)getSystemService(ALARM_SERVICE); am.setRepeating(AlarmManager.ELAPSED_REALTIME_WAKEUP, SystemClock.elapsedRealtime() + 30*1000, 30 * 1000, pendingIntent); // setRepeating(int type, long triggerAtMillis, // long intervalMillis, PendingIntent operation)

The issue is considered serious. Because whenever the alarm fails to trigger, it won't work anymore until the next power cycle (reboot). I used the scope and confirmed the hardware didn't pull up the interrupt line. Software did set the alarm but there is no interrupt happening. And it won't work since the problem first happened. It's dead until the device is rebooted.

On the device we were using, all interrupts are logically combined on a single output line 'INT'. It's important to note that when an interrupt is masked (bit 1), STATUS register is not updated, and INT is not triggered even if the masked event occurs.

A detailed register dump and trace was performed and we found out the mask is not correctly configured after some time, that's why the RTC alarm's interrupt failed to deliver. The next question became: who was modifying the MASK register so that it failed?

After more tests, we came closer. This issue occurred only if there was some suspend failure (caused by other unrelated device) and system resumed immediately. For some reason the irq_enable part of the RTC driver was not invoked in this case, which was supposed to set the correct MASK value for RTC interrupt to work.

Root cause is then found. The RTC's IRQ is configured to IRQF_EARLY_RESUME (resume IRQ early during syscore instead of at device resume time), and there is unbalanced IRQ enable/disable in system suspend/resume path.

To explain that in more detail, a rough suspend/resume trace is here:

pm_suspend() 
|- enter_state()  
|-- suspend_devices_and_enter() 
|---- dpm_suspend_start() 
|------ dpm_prepare()
|------ dpm_suspend() 
|------- device_suspend()
|---- suspend_enter()
|------ dpm_suspend_end() 
|------ syscore_suspend()  
|=========================  
|------ syscore_resume()  
|------ dpm_resume_start() 
|---- dpm_resume_end() 
|------ dpm_resume() 
|------- device_resume() 
|------ dpm_complete()

And the problem is in suspend_enter(). You have to read counterclockwise (happening in order of time) from the upper-left part of the following diagram:

      [Suspend]           time      [Resume]
==========================  ^ ==============================   
suspend_enter()  (BEGIN)  | | suspend_enter()  (END)
|- dpm_suspend_end()      | | |- dpm_resume_start()
|-- dpm_suspend_noirq()   | | |-- dpm_resume_noirq()
|--- suspend_device_irqs()| | |--- resume_device_irqs()
|    --> {all IRQs}       | | |  --> {Non-IRQF_EARLY_RESUME_IRQs}
|- syscore_suspend()      | | |- syscore_resume()
 (Entering suspend state) | | |-- irq_pm_syscore_ops.resume()
                          | |     or irq_pm_syscore_resume()
                          | |    --> {IRQF_EARLY_RESUME_IRQs}
                          \_/   (Exited suspend state)

Now it is obvious that when some device fails to suspend, syscore_suspend() doesn't get run. That is, syscore_resume() won't run either. So only Non-IRQF_EARLY_RESUME_IRQs will be resumed. So those irq_enable functions defined as IRQF_EARLY_RESUME_IRQ will stay disabled, and possibly forever.

The related code which caused this was first introduced here in this commit:

commit 9bab0b7fbaceec47d32db51cd9e59c82fb071f5a                              
Author: Ian Campbell <ian.campbell@citrix.com>                                
Date:   Mon Oct 3 15:37:00 2011 +0100                                        
                                                                             
    genirq: Add IRQF_RESUME_EARLY and resume such IRQs earlier                
                                                                             
    This adds a mechanism to resume selected IRQs during syscore_resume      
    instead of dpm_resume_noirq.                                              
                                                                             
    Under Xen we need to resume IRQs associated with IPIs early enough        
    that the resched IPI is unmasked and we can therefore schedule            
    ourselves out of the stop_machine where the suspend/resume takes          
    place.                                                                    
                                                                             
    This issue was introduced by 676dc3cf5bc3 "xen: Use IRQF_FORCE_RESUME".

A simple fix will be to resume IRQF_EARLY_RESUME_IRQs in resume_device_irqs() when irq_pm_syscore_resume() doesn't get run. A fix by Laxman has been merged to mainline kernel recently by simply enable all IRQs unconditionally in irq_resume(), which is confirmed that it doesn't cause side effects.

commit ac01810c9d2814238f08a227062e66a35a0e1ea2
Author: Laxman Dewangan <ldewangan@nvidia.com>
Date:   Mon Nov 25 19:39:47 2013 +0530
    irq: Enable all irqs unconditionally in irq_resume
 
    When the system enters suspend, it disables all interrupts in
    suspend_device_irqs(), including the interrupts marked EARLY_RESUME.
 
    On the resume side things are different. The EARLY_RESUME interrupts
    are reenabled in sys_core_ops->resume and the non EARLY_RESUME
    interrupts are reenabled in the normal system resume path.
 
    When suspend_noirq() failed or suspend is aborted for any other
    reason, we might omit the resume side call to sys_core_ops->resume()
    and therefor the interrupts marked EARLY_RESUME are not reenabled and
    stay disabled forever.
 
    To solve this, enable all irqs unconditionally in irq_resume()
    regardless whether interrupts marked EARLY_RESUMEhave been already
    enabled or not.
 
    This might try to reenable already enabled interrupts in the non
    failure case, but the only affected platform is XEN and it has been
    confirmed that it does not cause any side effects.