When doing Android system power stress test (entering low power state and exiting it repetitively), I found that sometimes RTC alarm won't wake the device up.
The wake-up mechanism is easy. It's from Android framework:
// using alarm in Android framework
// creating a repeated alarm every 30 sec
AlarmManager am = (AlarmManager)getSystemService(ALARM_SERVICE);
am.setRepeating(AlarmManager.ELAPSED_REALTIME_WAKEUP,
SystemClock.elapsedRealtime() + 30*1000,
30 * 1000, pendingIntent);
// setRepeating(int type, long triggerAtMillis,
// long intervalMillis, PendingIntent operation)
The issue is considered serious. Because whenever the alarm fails to trigger, it won't work anymore until the next power cycle (reboot). I used the scope and confirmed the hardware didn't pull up the interrupt line. Software did set the alarm but there is no interrupt happening. And it won't work since the problem first happened. It's dead until the device is rebooted.
On the device we were using, all interrupts are logically combined on a single output line 'INT'. It's important to note that when an interrupt is masked (bit 1), STATUS register is not updated, and INT is not triggered even if the masked event occurs.
A detailed register dump and trace was performed and we found out the mask is not correctly configured after some time, that's why the RTC alarm's interrupt failed to deliver. The next question became: who was modifying the MASK register so that it failed?
After more tests, we came closer. This issue occurred only if there was some suspend failure (caused by other unrelated device) and system resumed immediately. For some reason the irq_enable part of the RTC driver was not invoked in this case, which was supposed to set the correct MASK value for RTC interrupt to work.
Root cause is then found. The RTC's IRQ is configured to IRQF_EARLY_RESUME (resume IRQ early during syscore instead of at device resume time), and there is unbalanced IRQ enable/disable in system suspend/resume path.
To explain that in more detail, a rough suspend/resume trace is here:
pm_suspend()
|- enter_state()
|-- suspend_devices_and_enter()
|---- dpm_suspend_start()
|------ dpm_prepare()
|------ dpm_suspend()
|------- device_suspend()
|---- suspend_enter()
|------ dpm_suspend_end()
|------ syscore_suspend()
|=========================
|------ syscore_resume()
|------ dpm_resume_start()
|---- dpm_resume_end()
|------ dpm_resume()
|------- device_resume()
|------ dpm_complete()
And the problem is in suspend_enter(). You have to read counterclockwise (happening in order of time) from the upper-left part of the following diagram:
[Suspend] time [Resume]
========================== ^ ==============================
suspend_enter() (BEGIN) | | suspend_enter() (END)
|- dpm_suspend_end() | | |- dpm_resume_start()
|-- dpm_suspend_noirq() | | |-- dpm_resume_noirq()
|--- suspend_device_irqs()| | |--- resume_device_irqs()
| --> {all IRQs} | | | --> {Non-IRQF_EARLY_RESUME_IRQs}
|- syscore_suspend() | | |- syscore_resume()
(Entering suspend state) | | |-- irq_pm_syscore_ops.resume()
| | or irq_pm_syscore_resume()
| | --> {IRQF_EARLY_RESUME_IRQs}
\_/ (Exited suspend state)
Now it is obvious that when some device fails to suspend, syscore_suspend() doesn't get run. That is, syscore_resume() won't run either. So only Non-IRQF_EARLY_RESUME_IRQs will be resumed. So those irq_enable functions defined as IRQF_EARLY_RESUME_IRQ will stay disabled, and possibly forever.
The related code which caused this was first introduced here in this commit:
commit 9bab0b7fbaceec47d32db51cd9e59c82fb071f5a
Author: Ian Campbell <ian.campbell@citrix.com>
Date: Mon Oct 3 15:37:00 2011 +0100
genirq: Add IRQF_RESUME_EARLY and resume such IRQs earlier
This adds a mechanism to resume selected IRQs during syscore_resume
instead of dpm_resume_noirq.
Under Xen we need to resume IRQs associated with IPIs early enough
that the resched IPI is unmasked and we can therefore schedule
ourselves out of the stop_machine where the suspend/resume takes
place.
This issue was introduced by 676dc3cf5bc3 "xen: Use IRQF_FORCE_RESUME".
A simple fix will be to resume IRQF_EARLY_RESUME_IRQs in resume_device_irqs() when irq_pm_syscore_resume() doesn't get run. A fix by Laxman has been merged to mainline kernel recently by simply enable all IRQs unconditionally in irq_resume(), which is confirmed that it doesn't cause side effects.
commit ac01810c9d2814238f08a227062e66a35a0e1ea2
Author: Laxman Dewangan <ldewangan@nvidia.com>
Date: Mon Nov 25 19:39:47 2013 +0530
irq: Enable all irqs unconditionally in irq_resume
When the system enters suspend, it disables all interrupts in
suspend_device_irqs(), including the interrupts marked EARLY_RESUME.
On the resume side things are different. The EARLY_RESUME interrupts
are reenabled in sys_core_ops->resume and the non EARLY_RESUME
interrupts are reenabled in the normal system resume path.
When suspend_noirq() failed or suspend is aborted for any other
reason, we might omit the resume side call to sys_core_ops->resume()
and therefor the interrupts marked EARLY_RESUME are not reenabled and
stay disabled forever.
To solve this, enable all irqs unconditionally in irq_resume()
regardless whether interrupts marked EARLY_RESUMEhave been already
enabled or not.
This might try to reenable already enabled interrupts in the non
failure case, but the only affected platform is XEN and it has been
confirmed that it does not cause any side effects.