| + klardotsh (~klardotsh@98.97.114.65) | 00:00 | |
| - chomwitt (QUIT: Remote host closed the connection) (~chomwitt@2a02:587:7a14:af00:9080:176a:ae9d:81cc) | 00:00 | |
| - S0rin (QUIT: Ping timeout: 240 seconds) (~S0rin@user/s0rin) | 00:04 | |
| + S0rin (~S0rin@user/s0rin) | 00:41 | |
| - ec0 (QUIT: Ping timeout: 268 seconds) (~ec0@vps-446f4f39.vps.ovh.ca) | 01:09 | |
| + ec0 (~ec0@vps-446f4f39.vps.ovh.ca) | 01:09 | |
| - cwebber (QUIT: Remote host closed the connection) (~user@user/cwebber) | 01:19 | |
| - GNUmoon2 (QUIT: Remote host closed the connection) (~GNUmoon@gateway/tor-sasl/gnumoon) | 01:20 | |
| + GNUmoon2 (~GNUmoon@gateway/tor-sasl/gnumoon) | 01:21 | |
| - gnou_liber (QUIT: Ping timeout: 240 seconds) (~gnou_libe@223.pool85-50-3.static.orange.es) | 01:36 | |
| + gnou_liber (~gnou_libe@223.pool85-50-3.static.orange.es) | 01:40 | |
| - mtm- (QUIT: Ping timeout: 260 seconds) (~mtm@c-71-228-84-213.hsd1.fl.comcast.net) | 02:03 | |
| - mjw (QUIT: Ping timeout: 268 seconds) (~mjw@gnu.wildebeest.org) | 02:06 | |
| - nsc (QUIT: Ping timeout: 260 seconds) (~nicolas@149-48-142-46.pool.kielnet.net) | 03:21 | |
| + nsc (~nicolas@219-48-142-46.pool.kielnet.net) | 03:23 | |
| + mtm- (~mtm@c-71-228-84-213.hsd1.fl.comcast.net) | 04:09 | |
| - gnou_liber (QUIT: Read error: Connection reset by peer) (~gnou_libe@223.pool85-50-3.static.orange.es) | 04:18 | |
| + gnou_liber (~gnou_libe@223.pool85-50-3.static.orange.es) | 04:28 | |
| chartreuse | minute: The CPU module was installed yes, I didn't even switch the reform on when it smoked itself. Just plugged it in. I didn't have the batteries installed though but didn't think that would be an issue | 04:33 |
|---|---|---|
| - kws (PART: !!unknown attribute: msg!!) (3596bb3a9b@2604:bf00:561:2000::43f) | 05:01 | |
| - vagrantc (QUIT: Quit: leaving) (~vagrant@2600:3c01:e000:21:7:77:0:50) | 05:04 | |
| - gnou_liber (QUIT: Read error: Connection reset by peer) (~gnou_libe@223.pool85-50-3.static.orange.es) | 05:58 | |
| + gnou_liber (~gnou_libe@223.pool85-50-3.static.orange.es) | 06:01 | |
| - Boostisbetter (QUIT: Ping timeout: 260 seconds) (4a410829d7@irc.cheogram.com) | 06:32 | |
| + bgs (~bgs@212-85-160-171.dynamic.telemach.net) | 07:08 | |
| + Boostisbetter (4a410829d7@irc.cheogram.com) | 07:08 | |
| - gnou_liber (QUIT: Ping timeout: 240 seconds) (~gnou_libe@223.pool85-50-3.static.orange.es) | 07:36 | |
| + chomwitt (~chomwitt@2a02:587:7a14:af00:9080:176a:ae9d:81cc) | 07:54 | |
| ex-parrot | chartreuse: I think there are issues with the discharge circuitry sometimes going massively over current when the batteries are disconnected :( | 07:56 |
| - bgs (QUIT: Remote host closed the connection) (~bgs@212-85-160-171.dynamic.telemach.net) | 08:08 | |
| Booster[m] | <Booster[m]> "josch: thanks again for your..." <- josch: unfortunately bounty source does not support gitlab. Any chance we could get an issue on github? Seems this is required to create a bounty.. | 08:38 |
| Booster[m] | minute: in the MNT gitlab for the Reform there is a sleep/wake issue open where you last suspected that the GPU is the issue. Based on the behavior I am seeing when suspend fails to resume, this seems to be the right track. Now whatever regressions occurred between 5.12 and 6.1 seem to be safely within the way the kernel reinitializes the graphics stack on wake. | 08:48 |
| Booster[m] | I would love to drop a bounty on the issue from bounty source but they don't support gitlab it seems. | 08:49 |
| minute | this theory can be tested by rmmod etnaviv on the linux console and then suspending and resuming a few times. | 09:47 |
| Booster[m] | minute: well even though resume performance is much better under 5.12 there are still occasional freezes which occur. 1 failure out of 20 resumes, for example. | 09:58 |
| Booster[m] | So the issue is still there, it is just not triggered as often. | 09:58 |
| Booster[m] | minute: I'm actually on a break now, and one other peculiar thing I noticed, was if you eject the SD card while the system is sleeping, it will wake. | 09:59 |
| Booster[m] | I accidentally ejected the SD card while getting the Reform out of my bag. | 09:59 |
| + mjw (~mjw@gnu.wildebeest.org) | 10:05 | |
| + MajorBiscuit (~MajorBisc@2001:1c00:2492:e200:561f:c4dd:78fa:8040) | 10:09 | |
| - MajorBiscuit (QUIT: Client Quit) (~MajorBisc@2001:1c00:2492:e200:561f:c4dd:78fa:8040) | 10:14 | |
| + MajorBiscuit (~MajorBisc@2001:1c00:2492:e200:561f:c4dd:78fa:8040) | 10:15 | |
| josch | Booster[m]: sure, we could push a copy of the reform-debian-packages git to github and then file an issue there is it's just about this formality | 10:17 |
| - MajorBiscuit (QUIT: Client Quit) (~MajorBisc@2001:1c00:2492:e200:561f:c4dd:78fa:8040) | 10:19 | |
| + MajorBiscuit (~MajorBisc@2001:1c00:2492:e200:561f:c4dd:78fa:8040) | 10:20 | |
| minute | Booster[m]: ejecting sd card is a wakeup source? interesting | 10:24 |
| c-keen[m] | I think that's configurable | 10:25 |
| c-keen[m] | ejecting generates an interrupt | 10:26 |
| Booster[m] | <josch> "Booster: sure, we could push a..." <- That would be great. As soon as it is ready I'll drop the bounty and put some money on it. | 10:29 |
| - chomwitt (QUIT: Ping timeout: 264 seconds) (~chomwitt@2a02:587:7a14:af00:9080:176a:ae9d:81cc) | 10:42 | |
| - mjw (QUIT: Ping timeout: 268 seconds) (~mjw@gnu.wildebeest.org) | 11:26 | |
| * mark_ -> mjw | 11:36 | |
| + chomwitt (~chomwitt@ppp-94-67-206-188.home.otenet.gr) | 11:54 | |
| Booster[m] | <minute> "this theory can be tested by..." <- minute: I'm not familiar with that tool, and so I am not sure I could help with that. Someone with more experience could for sure get it. | 12:24 |
| + klardotsh_ (~klardotsh@98.97.114.65) | 13:09 | |
| - klardotsh (QUIT: Ping timeout: 256 seconds) (~klardotsh@98.97.114.65) | 13:09 | |
| - mtm- (QUIT: Ping timeout: 260 seconds) (~mtm@c-71-228-84-213.hsd1.fl.comcast.net) | 14:02 | |
| minute | Booster[m]: lsmod shows you the list of loaded kernel modules (drivers). rmmod removes (unloads) a driver/module. etnaviv is the GPU driver. thus, by unloading drivers they are out of the equation for suspend testing. | 14:36 |
| minute | so one could potentially narrow down if a certain driver is the culprit by unloading as many drivers as possible on the plain linux console (outside of graphical session, or even over uart) and then see if suspend/wake behavior changes. | 14:37 |
| Boostisbetter | minute: that all makes sense. I think based on on the testing that the initial ticket contained, it was pretty clear that pure console resumes were flawless. Any graphical session introdued the issues. Was a little less pronounced on EXWM versus Sway, but I would chalk that up to the difference between X and Wayland. The driver itself seems to be the issue based on that information, but it is sub | 14:45 |
| Boostisbetter | jective and guess work. | 14:45 |
| Boostisbetter | I'm beginning to think I just need to pick this up myself. But I'll be starting at the far end of the pool. Does anyone have a good book on the linux kernel and driver development? | 14:49 |
| Boostisbetter | I wonder if looking at the deltas in the etnaviv driver between 5.12 and 6.1 would be a shortcut to finding some answers as well? | 14:50 |
| - S0rin (QUIT: Read error: Connection reset by peer) (~S0rin@user/s0rin) | 14:50 | |
| Boostisbetter | I can probably dedicate about a half hour a day towards this. I'm not sure that is going to be enough to bare any fruit. | 14:55 |
| minute | Boostisbetter: i suggest first establishing that you can really 100% reliably resume on console. | 14:57 |
| minute | Boostisbetter: etnaviv is also not established as the source of the problem. you have to be 100% sure about this, otherwise it will be shots in the dark | 14:58 |
| + mark_ (~mjw@gnu.wildebeest.org) | 14:58 | |
| Boostisbetter | no I agree, I just mean that if we were betting this is the direction I would be looking. To me the driver is no correctly reintializing the GPU, but that is all being assumed from a 1000 miles up. So I hear you. | 15:01 |
| minute | Boostisbetter: i don't even remember writing this ticket. so i would ignore it and start fresh. | 15:01 |
| minute | i recently saw a very good talk showing linux kernel debugging methods and tools. let me find it for you | 15:02 |
| Boostisbetter | the other shortcut would be to just pull down the etnaviv driver being used on the Librem 5 6.1 kernel and plug it in to the kernel for the Reform. My guess is that this is the absolute path of least resistence in getting this resolved. | 15:03 |
| minute | why etnaviv? | 15:03 |
| Boostisbetter | minute: thanks! I am keen to get up to speed at this point | 15:03 |
| Boostisbetter | minute: isn't it the gpu driver? | 15:04 |
| - klardotsh_ (QUIT: Remote host closed the connection) (~klardotsh@98.97.114.65) | 15:04 | |
| minute | yes, but why do you think it is the gpu driver without having proof that it is the cause? | 15:04 |
| + klardotsh (~klardotsh@98.97.114.65) | 15:04 | |
| Boostisbetter | minute: because the issue that I always see is a lack of display activity every single time the Reform crashes on resume. | 15:05 |
| Boostisbetter | it isn't scientific, just the direction I want to look. Call it a hunch. | 15:05 |
| minute | Boostisbetter: here is the talk https://www.youtube.com/watch?v=NDXYpR_m1CU | 15:05 |
| minute | Boostisbetter: the display pipeline is a lot more than the gpu. is the rest of the system still alive? | 15:06 |
| Boostisbetter | thank you! I'll get into tonight after work. | 15:06 |
| Boostisbetter | minute, yes, it seems to be, BUT this is only based on power draw as reflected in the LPC. | 15:06 |
| Boostisbetter | Also observation that the activity light for the NVME is flashing, etc. | 15:06 |
| minute | Boostisbetter: that's not good enough. you need to confirm (prove) this with a UART console or ssh session from another computer. | 15:06 |
| + S0rin (~S0rin@user/s0rin) | 15:06 | |
| Boostisbetter | minute, yeah I was never really serious about the issue. | 15:07 |
| minute | Boostisbetter: if the system is indeed alive, one could perhaps inspect the issue while it is happening | 15:07 |
| Boostisbetter | Still if I was a betting man, I would put money on the etnaviv driver in the kernel. | 15:07 |
| minute | Boostisbetter: well, i only care about proof | 15:07 |
| minute | Boostisbetter: my other point is that you have many tools at your disposal to isolate the issue and gather more evidence without needing to dig into kernel code | 15:08 |
| Boostisbetter | minute yeah, as a fellow software engineer, I like to do things the hardware in my spare time. I'm so regulated at work. So I know exactly what you are talking about. | 15:10 |
| Boostisbetter | minute yeah, as a fellow software engineer, I like to do things the hard way in my spare time. I'm so regulated at work. So I know exactly what you are talking about. | 15:10 |
| minute | i see! | 15:10 |
| minute | i will have to work on suspend stuff soon on the imx8mplus (last time i worked on it it was broken), perhaps i can write down my debugging methods when i do it | 15:11 |
| Boostisbetter | minute: would be helpful for sure. | 15:11 |
| Boostisbetter | in josch's case, I don't know what is going on with his Reform. I suspect there is someting else on his mainboard that might be ever so slightly out of spec. | 15:12 |
| Boostisbetter | because he has no luck on resume due to his NVME drive not coming back. | 15:13 |
| minute | yep | 15:14 |
| - mark_ (QUIT: Ping timeout: 248 seconds) (~mjw@gnu.wildebeest.org) | 15:30 | |
| - GNUmoon2 (QUIT: Ping timeout: 240 seconds) (~GNUmoon@gateway/tor-sasl/gnumoon) | 16:07 | |
| + mtm- (~mtm@c-71-228-84-213.hsd1.fl.comcast.net) | 16:08 | |
| + GNUmoon2 (~GNUmoon@gateway/tor-sasl/gnumoon) | 16:10 | |
| minute | one more repair. fuse F2 is blown. this manifests as one battery showing 0V on the battery status. | 16:27 |
| minute | (this flaw is fixed in mb2.5) | 16:27 |
| minute | (the flaw being that power flows around the fuse, through the battery monitor chip) | 16:32 |
| Boostisbetter | minute: good to hear. | 17:31 |
| Boostisbetter | glad you found the issue. | 17:31 |
| + mark_ (~mjw@gnu.wildebeest.org) | 18:16 | |
| - mjw (QUIT: Killed (NickServ (GHOST command used by mark_!~mjw@gnu.wildebeest.org))) (~mjw@2001:1c06:2488:1400:4fd:39a7:74ac:7bae) | 18:16 | |
| * mark_ -> mjw | 18:16 | |
| + mark_ (~mjw@2001:1c06:2488:1400:4fd:39a7:74ac:7bae) | 18:16 | |
| + nottheoilrig (~nottheoil@node-1w7jr9ujcsmbjl4enh4ae1wek.ipv6.telus.net) | 18:24 | |
| minute | tested nvme on rcm4. works. | 18:45 |
| + bgs (~bgs@212-85-160-171.dynamic.telemach.net) | 18:58 | |
| Boostisbetter | minute: woot! | 19:01 |
| + ajr (~ajr@user/ajr) | 19:01 | |
| minute | ethernet also works fine. was just a little soldering issue. | 19:16 |
| - nottheoilrig (QUIT: Quit: Client closed) (~nottheoil@node-1w7jr9ujcsmbjl4enh4ae1wek.ipv6.telus.net) | 19:16 | |
| Boostisbetter | that is great to hear. I would think that the pi stuff would be pretty straight forward to get going, as Raspberry Pi really seems to put a lot of effort into supporting their SBCs. | 19:19 |
| minute | cm4 can't suspend though as far as i know ;) | 19:20 |
| - MajorBiscuit (QUIT: Quit: WeeChat 3.6) (~MajorBisc@2001:1c00:2492:e200:561f:c4dd:78fa:8040) | 19:25 | |
| Boostisbetter | say it ain't so. | 19:42 |
| minute | Boostisbetter: https://github.com/raspberrypi/firmware/issues/1635 | 19:50 |
| Boostisbetter | looks like architecturally it wont be coming to the platform anytime soon if ever. | 19:57 |
| Boostisbetter | Makes me love the i.mx8 a lot more | 19:57 |
| + vagrantc (~vagrant@2600:3c01:e000:21:7:77:0:50) | 20:02 | |
| vkoskiv | Boostisbetter: I also have the issue of my SSD not coming back after suspend | 20:19 |
| vkoskiv | WD Blue drive. | 20:19 |
| vkoskiv | Didn't have that issue on my previous Intel 600p, but that drive had other issues. | 20:20 |
| vkoskiv | Namely it randomly froze during use | 20:20 |
| josch | i didn't try that intel drive -- maybe the failures you had with it were also somehow due to your reform? did you try that drive in another machine? | 20:28 |
| vkoskiv | I used the 600p in other machines before using it in the reform without issues. | 20:28 |
| vkoskiv | Actually now I remember, the drive was quite worn down | 20:29 |
| vkoskiv | So it's possible it just got degraded too much at some point in between | 20:29 |
| Boostisbetter | ok, yeah that is good to know. I mean the drive can be an issue, especially if it doesn't come back up on resume. | 20:53 |
| Boostisbetter | josch: one thing I am noticing about the 5.12 kernel is that there are little graphical glitches that happen revoling around updating GUI elements of open applications. | 20:56 |
| Boostisbetter | these glitches and anomolies don't happen under 6.1 | 20:57 |
| Boostisbetter | but the issues are just temporary and do not cause any serious issue. I have been using 5.12 for suspend for a while now. I'm on my 12 resume so far. | 20:57 |
| minute | now testing BPi CM4 again in rcm4 | 21:22 |
| minute | > SoC: Amlogic Meson G12B (A311D) Revision 29:b (10:2) | 21:22 |
| Boostisbetter | minute: most excellent. You know, I think it would be hard for me to care about the other boards, when a pocket reform was there and could be messed with. | 21:23 |
| Boostisbetter | I'm really looking forward to that thing. I think I'm going to be able to use it a lot. | 21:24 |
| minute | oh nice > Welcome to Armbian 23.02.0-trunk Bullseye! | 21:24 |
| minute | Boostisbetter: i'm sure you'll have fun with it! | 21:24 |
| Boostisbetter | I am hoping that the pocket will get suspend capabilities as well though. | 21:24 |
| minute | oh nice, i have logged in and ethernet is working on the a311d | 21:26 |
| minute | Boostisbetter: yes, it is quite important to me | 21:26 |
| minute | nice, 6 cores | 21:26 |
| minute | oh, pcie works (it detects the ssd) | 21:31 |
| Boostisbetter | mucho nice! | 21:33 |
| minute | oh, and something on hdmi. | 21:38 |
| minute | weston and weston-simple-egl works, so gpu works | 21:44 |
| minute | (panfrost) | 21:44 |
| Boostisbetter | minute: very nice. Glad things seem to be working out on it all. | 22:48 |
| - chomwitt (QUIT: Remote host closed the connection) (~chomwitt@ppp-94-67-206-188.home.otenet.gr) | 22:50 | |
| - bgs (QUIT: Remote host closed the connection) (~bgs@212-85-160-171.dynamic.telemach.net) | 23:38 | |
| minute | the bpi cm4 / amlogic a311d cpu is really snappy. stress tested it a bit with youtube sw rendering https://mastodon.social/@mntmn/110340885480226883 | 23:40 |
| minute | photo of the module: https://mastodon.social/@mntmn/110340887376981595 | 23:41 |
| minute | note the lack of heatsink | 23:41 |
| chartreuse | Impressive that it manages without a heatsink without throttling, compared to say the official CM4 which will start throttling under sustained load without cooling | 23:58 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!