| rinigus | morning! I am debugging suspend on xz2. testing now work any active radios (wifi/bt/cellular/nfc off in settings) | 08:18 |
|---|---|---|
| rinigus | in these conditions, most of the suspend attempts fail, at rate 10 failures per second. overall time in sleep is 10-20% | 08:19 |
| rinigus | rather common snpshot in dmesg is at https://dpaste.de/DeOv#L123 | 08:19 |
| rinigus | as a last active wakeup, I got this night mostly mce_mux and usb_moded_input | 08:20 |
| rinigus | at the same time, phone was without any usb connections | 08:21 |
| rinigus | how can I check what triggered mce_mux specifically? | 08:21 |
| rinigus | when looking at wakeup_source_activate using tracing, I get many hits on mce_mux, usb_moded_input, and even more frequently c440000.qcom,spmi:qcom,pmi8998@2:qpnp,fg | 08:23 |
| rinigus | I suspect that the latter is power supply, but I will ask 'bout it | 08:24 |
| Mister_Magister | rinigus: if i could disable wakelock from that qcom thing there would be nothing to wake my phone up :P | 08:27 |
| rinigus | Mister_Magister: I see. any tips on what's causing mce_mux? do we have some mce-based wakelock debugging available? | 08:28 |
| rinigus | Mister_Magister: btw, is suspend working fine on pro1? | 08:29 |
| Mister_Magister | i guess it's working fine | 08:29 |
| Mister_Magister | rinigus: i have no idea what you are doing so nah not really | 08:29 |
| T42 | Fatrank was added by: Fatrank | 08:29 |
| Mister_Magister | but you could ask spiiroin about mce i believe | 08:30 |
| rinigus | Mister_Magister, re guess: you could check that. but if it is able to stay idle on battery multiple days it should be ok | 08:33 |
| rinigus | spiiroin: would you mind to help with suspend debugging? in particular, how can I distinguish what caused mce_mux to appear? (see my description from todays above) | 08:34 |
| Mister_Magister | rinigus: i can't really tell if battery drain is bad or battery is bad as i have prototype unit… | 08:35 |
| Mister_Magister | but it wasn't really running fast when i forgot about it it lasted 3 days i believe | 08:35 |
| Mister_Magister | rinigus: battery drain was better on 3.0 than 3.1 ;-; | 08:35 |
| rinigus | Mister_Magister: you could easily get stats on suspend using mcetools (see its help, grep suspend). if you don't use the phone, expect to get suspended time as a majority (90% or so). | 08:37 |
| Mister_Magister | rinigus: currently smd_dev irq seems to wake my phone up | 08:38 |
| Mister_Magister | rinigus: for 12k uptime i have 10k suspend | 08:38 |
| Mister_Magister | that didn't change | 08:38 |
| rinigus | Mister_Magister: then you are fine | 08:38 |
| Mister_Magister | but somehow battery drain is 2/3x worse and i'm not the only one who noticed it | 08:38 |
| Mister_Magister | multiple people who use my ports and not only my ports noticed higher battery drain. Even my mom! | 08:39 |
| r0kk3rz | thats fairky subjective though | 08:47 |
| Mister_Magister | that's not subjective at all | 08:47 |
| Mister_Magister | imagine your phone lasting 3 days instead of 3 weeks after you updated it | 08:47 |
| rinigus | I am not going into 3.0 vs 3.1 debate. just my logs over the year don't show any significant difference on onyx when I glance over them. | 08:48 |
| Mister_Magister | and it's not only my port | 08:48 |
| Mister_Magister | well facts are facts | 08:48 |
| r0kk3rz | "facts" | 08:48 |
| rinigus | ... but tips on mce debugging are welcome | 08:48 |
| Mister_Magister | r0kk3rz: yes facts | 08:49 |
| r0kk3rz | if its that bad it should be fairly obvious | 08:49 |
| Mister_Magister | but well it is not | 08:49 |
| r0kk3rz | 3.1 hasnt been out that long | 08:50 |
| Mister_Magister | couple months? | 08:50 |
| r0kk3rz | oh thats right, we're up to 3.2 now | 08:52 |
| T42 | <adampigg> 3.2 seems much better for battery on mido | 09:02 |
| Mister_Magister | will have to update then and check | 09:03 |
| rinigus | spiiroin: I will look into suspend issue with more verbose logs, as set by mcetool to debug level. that should give me some overview. will get back to you when will know more | 11:55 |
| T42 | <nitin03 %lastname%> ril is running https://pastebin.com/t5QU1bkp | 12:50 |
| T42 | <nitin03 %lastname%> logcat returns this https://pastebin.com/vCmVDFwp | 12:50 |
| jusa | elros34: what do you mean by "wrong directory"? dbus packaging has had that binary in /lib/dbus-1/dbus-daemon-launch-helper since initial packaging (in 2013)? | 14:00 |
| jusa | heh ok servicehelper in system.conf indeed does list that as /usr/libexec | 14:02 |
| T42 | <elros34> jusa: looks like something changed in dbus, because when I strace dbus-daemon it expects it in /usr/libexec/ | 14:02 |
| jusa | I'd wager last dbus update has something to do with that.. I'll take a look | 14:02 |
| T42 | <elros34> thanks, this breaks patchmanager in 3.2.0 | 14:03 |
| jusa | aha.. yep, update to 1.13.12 has changed from autotools to cmake for building and at that point --libexecdir=/%{_lib}/dbus-1 was dropped :( | 14:04 |
| Mister_Magister | hi jusa how ya doin | 14:11 |
| spiiroin | rinigus: mce_mux is "multiplexed" wakelock that by nature is "mixed bag of beans" -> some visibility (and expect lots of noise) for ex: systemctl stop mce ; mce -T -lmce-wakelock.c:* | 14:17 |
| rinigus | spiiroin: yeah, as for mce_mux - I figured out that much from reading the code. | 14:18 |
| rinigus | spiiroin: would that mce -T ... command log into terminal? | 14:19 |
| spiiroin | that usb_moded_input might actually be more significant ... if there is lots of udev noise usb-moded needs to wade through -> can block suspend longer than what the intent is | 14:19 |
| rinigus | spiiroin: from reading usb_moded code and your docs. could it be that its reading wrong input to record usb insertion? | 14:20 |
| spiiroin | rinigus: yes, -T -> stderr, with slightly different output format & not subjected to journald line count limitations | 14:21 |
| rinigus | spiiroin: ok, then I will have to redirect it to file to ensure it will not be waking up for typing | 14:22 |
| spiiroin | rinigus: more like: the wakelocking in usb-moded was a quick hack to ensure that connect and/or disconnect actually get handled on - mm maybe l500d - before device suspends | 14:22 |
| spiiroin | so I would not be surprised if it causes problems sometimes in some devices | 14:23 |
| rinigus | spiiroin: so, to proceed - I will gather the logs regarding wakelocks on mce and report back. hopefully, it will tell us where to go from that. | 14:25 |
| spiiroin | rinigus: you might want to put usb-moded to verbose mode | 14:25 |
| rinigus | spiiroin: when using tracing, I had lots of hits from what appears to be fuel gauge as a wakeup_source_activate | 14:26 |
| rinigus | spiiroin: how to put it into verbose? | 14:26 |
| rinigus | (sorry for cross-posting, just wanted to mention) | 14:27 |
| spiiroin | rinigus: edit /lib/systemd/system/usb-moded.service -> add "-Dl" to ExecStart line | 14:27 |
| spiiroin | ... come to think about it. I might have disabled usb-moded wakelock logging ... | 14:28 |
| rinigus | spiiroin: thanks! will do. will be able to get on with it tonight ... | 14:28 |
| rinigus | spiiroin: was it tooo verbose? | 14:29 |
| spiiroin | yes. https://git.merproject.org/mer-core/usb-moded/blob/master/src/usb_moded-common.c#L308 | 14:31 |
| spiiroin | semi botched it in some refactoring too, the define is still in usb-moded.c ... ;-/ | 14:31 |
| spiiroin | rinigus: also: iirc usb moded uses two kinds of wakelocks: one that is explicitly obtained/released, and then ones that use timeout i.e. kernel cancels them automatically | 14:33 |
| spiiroin | the latter ones are used when there are dbus signals to broadcast etc -> guarantee that other ends gets them before device suspends | 14:33 |
| spiiroin | that is the one that might be too long | 14:34 |
| rinigus | spiiroin: so, I will have to recompile usb-moded with wakelock verbose in usb_moded_common | 14:34 |
| rinigus | spiiroin: but in my case its already going for suspend and then cancels it as a new wakelock arrives. as far as I understand the logs | 14:35 |
| spiiroin | that is a bit odd, I'd expect that it could happen occasionally. but constantly = ? | 14:36 |
| spiiroin | should the kernel even attempt to suspend? | 14:36 |
| rinigus | spiiroin: its as if there are no wakelocks active and, while it suspends, some arrive. we are talking about 10 times per second, for hours | 14:37 |
| rinigus | spiiroin: hence my suspicion on whether usb-moded is even reading a correct place to get its data from. note that phone is idle, nothing inserted, nothing communicating | 14:38 |
| rinigus | ... but in this respect, verbose mce may point us into correct direction | 14:40 |
| rinigus | ... or verbose usb-moded | 14:40 |
| rinigus | spiiroin: ... or it could be wlan driver that didn't get suspended with rfkill (when I switch it off). | 14:41 |
| rinigus | (too many options at this moment) | 14:42 |
| rinigus | spiiroin: however, what I do know is that AOSP9 works fine. so, it should be possible for us as well | 14:44 |
| spiiroin | userspace managing to insert wakelocks in mid kernel side suspend attempt... sounds unlikely - could be some kernel hiccup too | 14:44 |
| spiiroin | rinigus: actually, that mce_mux could be related to battery tracking - which also reads udev events similarly to what usb-moded does | 14:45 |
| spiiroin | enable extra verbose udev logging: printf "[BatteryUDevPropertyBlacklist]\nxxx=yyy\n" > /etc/mce/90-xxx.ini | 14:47 |
| spiiroin | then run mce so that they get printed out: mce -Tq -lmodules/battery-udev.c:* | 14:47 |
| spiiroin | rinigus: this was new port? you /might/ need to blacklist some power_supply devices if they produce false positives for mce battery tracking -> https://git.merproject.org/mer-core/mce/blob/master/inifiles/battery-udev-devices.ini | 14:49 |
| rinigus | spiiroin: thanks! will do | 14:49 |
| rinigus | spiiroin: yes, its a new port - sony xz2. | 14:50 |
| spiiroin | ok, and as a clarification: blacklisting = helps with battery state evaluation, does not change the situation regarding udev wakeups / input processing | 14:51 |
| * spiiroin needs to go - back online later | 14:51 | |
| rinigus | spiiroin: thanks! I will deal with the logs later tonigh | 14:52 |
| jusa | elros34: could you check if patchmanager works after changing in /usr/share/dbus-1/system.conf servicehelper to /lib/dbus-1/dbus-daemon-launch-helper ? just to make sure nothing else is needed | 14:59 |
| T42 | <elros34> I made symlink yesterday and it was ok but I will check that conf now | 15:05 |
| T42 | <elros34> jusa: I changed system.conf and rebooted. Now it works | 15:14 |
| jusa | ok thanks | 15:16 |
| deathmist | am I missing something or why aren't there SDK version 2.3.15 (SFOS 3.2) platform SDK chroot tarballs? | 15:46 |
| rinigus | spiiroin : here is mce log that was obtained with verbose udev logging and wakelock logging | 16:26 |
| rinigus | spiiroin: mce log: https://dpaste.de/Rz2j | 16:26 |
| rinigus | spiiroin: journal: https://dpaste.de/nJ0q | 16:26 |
| rinigus | spiiroin: now mce log is 26K lines, pasted just top with the end of the paste continuing for a while | 16:27 |
| rinigus | spiiroin: this phone has some smarter (or 'smarter') charging. it actually allows to suspend while charging, in sfos and aosp | 16:27 |
| rinigus | I do wonder what's increasing this SEQNUM in mce logs... | 16:29 |
| deathmist | oh and also mal: how was the internal discussion regarding nemo-qml-plugin-systemsettings? I (and others that would need such changes) definitely can't just keep using a "fix" like this indefinitely https://git.io/JeVnc :p | 16:38 |
| mal | sorry, haven't had time yet | 16:39 |
| deathmist | alright, guess I' | 16:43 |
| deathmist | ll hold onto this then for a while, do you know anything about the platform SDK chroot for SFOS 3.2 situation? | 16:44 |
| rinigus | spiiroin: the logs were obtained with the stock usb-moded. I haven't recompiled it for verbosity of wakelock. let me know if its needed | 16:49 |
| mal | deathmist: what do you mean? the target is in targets.json | 16:51 |
| mal | deathmist: maybe the latest hasn't been updated | 16:52 |
| mal | deathmist: http://releases.sailfishos.org/sdk/latest/targets.json | 16:52 |
| spiiroin | rinigus: that sequence number is just a increasing event counter. as that is usually reported as the only thing that changes -> the same udev data is sent over and over and over again -> does not sound normal | 17:06 |
| rinigus | spiiroin: maybe that's what it does. question is now whether we can somehow filter it out on android level? | 17:32 |
| rinigus | spiiroin: as far as I understand, we have 1 wakelock created by mce for each of these sequence increases. | 17:32 |
| rinigus | spiiroin: probably one more for usb-moded as well. | 17:33 |
| rinigus | spiiroin: so, if we cannot filter event out, maybe its possible to avoid wakelocks? | 17:33 |
| rinigus | ... I can also look for the code that generates this SEQNUM. maybe there is a way to stop it | 17:35 |
| rinigus | spiiroin: this is what's going on during one suspend attempt - which fails: https://dpaste.de/ApkW | 18:42 |
| T42 | <DylanVanAssche> Is the 3.2.0.12 build target already available on OBS? | 18:43 |
| rinigus | ... on top - dmesg / below udevadm monitor --env. haven't made diffs yet between battery updates | 18:43 |
| rinigus | ... during 0.1 s while suspending we have 13 udev + kernel events that were in udevadm monitor log | 18:50 |
| T42 | <nitin03 %lastname%> any way to debug why wifi not working? | 19:04 |
| T42 | <hacker12455> did you add the auto wlan start service? | 19:04 |
| T42 | <nitin03 %lastname%> yes | 19:05 |
| T42 | <hacker12455> modprobe wlan does not do anything? | 19:05 |
| T42 | <nitin03 %lastname%> no | 19:05 |
| T42 | <hacker12455> check dmesg for firmware loading errors maybe? | 19:05 |
| T42 | <nitin03 %lastname%> https://pastebin.com/5FJaUWnY | 19:08 |
| T42 | <nitin03 %lastname%> from logcat I got this | 19:10 |
| T42 | <nitin03 %lastname%> 07-25 15:53:07.463 4252 4252 I android.hardware.wifi@1.0-service: Wifi Hal is booting up... | 19:10 |
| T42 | <nitin03 %lastname%> 07-25 15:53:07.463 4252 4252 I ServiceManagement: Removing namespace from process name android.hardware.wifi@1.0-service to wifi@1.0-service. | 19:10 |
| *** OhYash1 is now known as ohyash | 19:19 | |
| rinigus | spiiroin: testing now without this wakelock https://git.merproject.org/mer-core/usb-moded/blob/master/src/usb_moded-udev.c#L303 (this and its unlock below commented out) | 19:41 |
| piggz | mal: any builds of qtquick.controls 2 available on sfos? | 20:58 |
| rinigus | spiiroin: after dropping wakelock in usb_moded and mce (modules/battery-udev.c), I am still getting lots of suspend failures. now the last wakelock is `c440000.qcom,spmi:qcom,pmi8998@2:qpnp,fg` | 21:49 |
| rinigus | spiiroin: with kernel tracing, the same wakelock is reported as a main one as well. in trace, there is no mce nor usb-moded wakelocks, as expected after commenting them out | 21:50 |
| rinigus | that wakelock is probably from https://github.com/sailfishos-sony-tama/android_kernel_sony_msm/blob/hybris-sony-aosp-9.0.0_r37_20190620-tama/drivers/power/supply/qcom/qpnp-fg-gen3.c , but I will have to confirm it | 21:52 |
| piggz | mal: i found an error in the QML formatter! | 22:02 |
| piggz | it converts this: | 22:02 |
| piggz | for (i = 0,len = str.length; i < len; i++) { | 22:02 |
| piggz | to: | 22:03 |
| piggz | for (i = 0len = str.length; i < len; i++) { | 22:03 |
| deathmist | @piggz shouldn't "i = 0," be "i = 0;" (notice the comma to semi-colon :p) | 22:05 |
| piggz | deathmist: no, 2 variables are set, not just one | 22:05 |
| deathmist | ah, I see now. I've not done something like that ever before :) | 22:06 |
| deathmist | piggz: any reason you'd want to define "len" inside the for loop initialization part instead of above the for loop? I'm not sure if that is even possible as I've never seen that before anywhere | 22:15 |
| piggz | deathmist: its not something id normally do, im porting an app and thats in the original code | 22:16 |
| deathmist | oh okay. now that I think of it, probable reason for that would be scope | 22:20 |
Generated by irclog2html.py 2.17.1 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!