2024-06-21.log

+ chomwitt (~chomwitt@2a02:587:7a02:1a00:1ac0:4dff:fedb:a3f1)00:11
joschvagrantc: i tried building a MNT image using guix but I get: Throw to key `encoding-error' with args `("scm_to_stringn" "cannot convert wide string to output locale" 84 #f #f)'.00:22
joschvagrantc: any idea? https://source.mnt.re/josch/mnt-reform-nonguix/-/jobs/456100:22
- mjw (QUIT: Ping timeout: 268 seconds) (~mjw@gnu.wildebeest.org)00:44
vagrantcjosch: wait, what? :)01:14
vagrantcjosch: you're building a kernel package for debian using guix?01:15
vagrantcjosch: wild hunch would be some issue with the guix locale vs. the system locale ... but i may be mixing that up with some other sort of issue01:18
vagrantcjosch: sometimes due to differing glibc versions (and thus locale data)01:24
vagrantcjosch: you might find people who can actually read those guile tracebacks more proficiently on #guix :)01:35
vagrantcACTION waves01:35
- vagrantc (QUIT: Quit: leaving) (~vagrant@2600:3c01:e000:21:7:77:0:50)01:35
- chomwitt (QUIT: Ping timeout: 240 seconds) (~chomwitt@2a02:587:7a02:1a00:1ac0:4dff:fedb:a3f1)02:44
- jn (QUIT: Ping timeout: 268 seconds) (~quassel@user/jn/x-3390946)03:21
+ jn (~quassel@2001-4dd4-8ba8-0-20d-b9ff-fe49-15fc.ipv6dyn.netcologne.de)03:21
- jn (QUIT: Changing host) (~quassel@2001-4dd4-8ba8-0-20d-b9ff-fe49-15fc.ipv6dyn.netcologne.de)03:21
+ jn (~quassel@user/jn/x-3390946)03:21
- nsc (QUIT: Ping timeout: 246 seconds) (~nicolas@240-99-142-46.pool.kielnet.net)03:25
+ nsc (~nicolas@i5C74DC29.versanet.de)03:27
joschminute: linux 6.9 is in experimental so i rebased our patch stack again. Here is the diff to the stack before: https://paste.debian.net/hidden/b9343dab/06:37
joschas usual, cadence was the most tricky one to rebase06:38
joschminute: i think the Ampere CI runner works well and solves the problem that debian unstable frequently has cross-building broken due to multi-arch:same version skews. But the big intel box had the advantage that cross-building the kernel on it is quite a bit faster. I prepared https://source.mnt.re/reform/reform-debian-packages/-/merge_requests/42 which (if it works) would allow one to select the CI 07:14
joschrunner by variable (defaulting to arm64)07:14
joschminute: could you add the intel box back to the reform-debian-packages project as a CI runner so that i can test this? thanks!07:14
pandorai made an apt update && upgrade and now my pocket reform is not booting anymore :/09:22
pandorai am curious if there is an image that is not based on debian unstable but stable09:22
joschpandora: not yet but soon [tm]09:23
joschpandora: i'm maintaining an unofficial fork of the MNT repos building everything for stable (and backports) on https://reform.debian.net09:23
joschpandora: but i did not yet have time to adept the scripts for the pocket reform release09:23
joschpandora: if you want to help, i'd love to answer all your questions and then it can happen sooner :)09:23
pandoramy background knowledge of the mnt build processes are currently non-existing :D ... the pocket is my first reform since i had to skip the normal reform due to money :D09:26
joschpandora: i'd also love to figure out what broke your pocket reform -- do you happen to be able to access the serial output with a uart adapter to have a look?09:26
pandorai do have a serial adapter but idk where i would connect them09:27
pandorais there a whitepaper on "how to get serial output from a broken mnt"?09:27
joschpandora: https://mntre.com/documentation/pocket-reform-handbook.pdf09:28
joschi think page 65 has the motherboard showing S1 and S2 for uart09:29
pandoraah yeah... i didn't scoll down that far yet :D09:29
joschthe table on page 87 has more details09:29
pandorai will try to get a serial output ... but i have to do some stuff at work first 09:32
joschminute: an apt-upgrade broke boot on the pocket -- maybe you want to have a look in case you manage to do it before pandora?09:33
pandorajust a bit confused by the labeling 09:45
pandorathe pins say gnd, ax2, tx2, gnd, ax1, tx109:46
pandorai guess ax1. means rx1?09:46
pandoraon the board09:46
pandoraits prob. just a bad print and u can mistake A for R09:47
joschminute: and another issue that we will soon have is, that qcacld fails to build with linux 6.9: https://paste.debian.net/1321040/ -- i'd need a patch from you for that09:52
- dozens (QUIT: Ping timeout: 268 seconds) (~dozens@tilde.town)09:53
+ dozens (~dozens@tilde.town)09:53
pandoraok got a serial output09:55
pandorahttps://pastebin.com/GTvKx5Q709:56
joschpandora: i guess it's stopping there?09:57
pandorayes09:57
pandorathats all the output09:57
joschpandora: when it says "Hit any key to stop autoboot:" can you press a key to get into the u-boot console?09:57
joschand then you run:09:57
pandoralet me try to get into uboot09:58
joschsetenv bootargs "${bootargs} loglevel=7"09:58
joschand then boot the thing by running:09:58
joschboot09:58
joschthat should give you a bit more output09:59
pandorait is not accepting any key strokes09:59
joschpandora: you have your system on eMMC?09:59
joschuh oh... :(09:59
pandorayes ... it is kinda like it came out of the box09:59
pandorai myself just installed some python stuff 10:00
pandoraand apt update & upgrade10:00
pandorano customisation so far10:00
joschright, it's unfortunate that you cannot get into uboot (but at least we saw some output)10:00
joschpandora: can you try popping in an sd-card with a fresh system image?10:00
pandorayeah I guess thats kinda the only other option :D10:01
joschpandora: if that boots, you could inspect /var/log/apt/history.log to get a list of packages that got upgraded and maybe that will let us spot potential problems10:01
joschminute: sorry, there is yet another issue on top of the things i mentioned above... the CI on source.mnt.re currently fails with: Uploading artifacts as "archive" to coordinator... 500 Internal Server Error  id=4610 responseStatus=500 Internal Server Error status=500 token=glcbt-6410:02
joschminute: in the past this was because there was not enough space to store the artifacts10:02
pandorado u have a link to a system image by any chance ... i lost the node with the link to the pocket system images :D10:03
joschpandora: sure thing! https://source.mnt.re/reform/reform-system-image/-/jobs/4529/artifacts/browse10:04
joschminute: sorry, yet another issue is that https://mnt.re/system-image still points to artifacts that are 404 -- you have to press the "keep" button in the artifact overview once you adjust the job that you want this to point to10:05
minutepandora: sorry for the troubles! but you can boot a newly flashed system image via microsd, uboot will pick that up automatically10:06
pandoraalready flashing an sd card :D10:07
minutejosch: ok, will fix that today10:07
joschminute: sorry to throw all these todos at you first thing in the morning10:07
joschespecially after all the frustration during the past days...10:07
joschminute: i also wonder why pandora was not able to stop u-boot via uart -- that should work, no?10:08
pandoraoh now the uboot worked10:12
pandorahttps://pastebin.com/gJPM5EzQ10:13
pandorabut not a big difference10:13
joschhuh this is super odd10:14
joschthat's even before the initramfs starts10:14
josch(the last reform-tools upload could've potentially messed up the initramfs)10:14
josch(though i had tested that on the ls1028a)10:14
pandorathat was at least part of the update10:14
pandorabooting form sd rn10:14
joschokay, good10:15
+ andreas-e (~Andreas@2001:861:c4:f2f0::c64)10:16
- nybble (QUIT: Ping timeout: 272 seconds) (~sprang@user/csprng)10:18
pandoraah pasting into firefox doesn't work on the pocket for some weird reason xD10:30
joschpandora: when you have a plain text file you can just do: cat file | curl -F 'arg=<-' https://mister-muffin.de/paste10:31
joschpandora: if you are on Debian you can install pastebinit and then you can do: cat file | pastebinit10:31
+ reform11163 (~user@business-90-187-186-49.pool2.vodafone-ip.de)10:33
reform11163https://mister-muffin.de/p/8SKF.txt10:33
pandorathats great :D10:33
pandora*adding to my notes of helpful commands :D10:34
joschpandora: did you reboot after you installed systemd-cryptsetup?10:35
pandorai rebooted after every apt upgrade10:35
joschokay10:35
joschthe interesting parts of that log are the upgrades of reform-tools as well as console-setup and keyboard-configuration as those packages can mess with the initramfs and thus with the boot process10:36
joschbut then i do not understand why the kernel fails this early without any output10:36
joschpandora: since you are on the sd-card now, could you try upgrading only reform-tools and then reboot to check if the reform-tools upgrade is what kills it?10:37
pandorasure10:37
minutejosch: yes, pandora should have been able to stop uboot via uart.10:38
joschminute: it seems to have worked in the end :)10:38
pandoraah... not enough free space :D10:39
joschpandora: to upgrade reform-tools?10:39
minuteahhh... yeah in the "factory" we have a set of commands to grow the root partition10:39
minuteon the sd card10:39
pandora@josch yeah :D10:39
joschoh!10:39
minutei documented that in the forum, one moment10:39
pandora257 mb available . 286 mb to download10:40
joschpandora: wait, that looks wrong10:40
joschis more than just reform-tools upgrading?10:40
minutehttps://community.mnt.re/t/deleted-the-emmc-partition/2100/210:41
pandora@josch u r right... i forgot to add --only-upgrade option10:44
pandorait is upgrading rn10:44
pandorabut i will put the link into my notes on "helpful links" :D10:44
joschminute: in case the "sleep 1" in that script annoys you, you can replace it with "udevadm settle"10:45
- reform11163 (QUIT: Quit: reform11163) (~user@business-90-187-186-49.pool2.vodafone-ip.de)10:47
+ reform13244 (~user@business-90-187-186-49.pool2.vodafone-ip.de)10:48
reform13244https://mister-muffin.de/p/Tn21.txt10:48
- reform13244 (QUIT: Remote host closed the connection) (~user@business-90-187-186-49.pool2.vodafone-ip.de)10:48
pandorait is booting again10:49
joschpandora: the "apt install" output looks good10:49
joschpandora: does it boot successfully?10:50
pandorait does10:50
joschuff that's a relieve10:51
joschokay, next please try upgrading just the kernel, that would be linux-image-mnt-reform-arm6410:51
pandorakk10:51
joschtheoretically this should also not change anything as the version remained the same at 6.8.12-1+reform20240605T104210Z10:51
joschpandora: if you could send us the "apt install" output again just to be sure that would also be great, thank you!10:52
joschthough you could run into space problems now10:53
minutea new imx8mp video pll dynamic adjust patch by marex just landed in linux, this will probably eliminate some of my patches in the future10:54
joschoh how nice! i'm happy every time i can drop some patches when i rebase everything :)10:55
+ reform5504 (~user@business-90-187-186-49.pool2.vodafone-ip.de)10:55
reform5504https://mister-muffin.de/p/owTA.tx10:55
joschi'm also very impressed how butter smooth the collabora rk3588 patch stack was so far -- the cadence patch is much more problematic than the dozens of rk3588 patches...10:56
pandorahttps://mister-muffin.de/p/owTA.txt10:56
- reform5504 (QUIT: Client Quit) (~user@business-90-187-186-49.pool2.vodafone-ip.de)10:56
pandorayeah that killed it10:57
joschwow10:57
joschpandora: same issue with no output on serial?10:58
pandorahttps://pastebin.com/t5QSqxa010:58
joschpandora: you didn't do the setenv loglevel though10:59
pandoramom10:59
joschbut it's probably going to remain the same... :/10:59
joschminute: this might be a very tricky issue as the kernel sources remained the same (6.8.12-1) and only the dependencies in the build environment changed...11:00
pandorayeah not big changes11:01
pandorahttps://pastebin.com/pA8952hD11:01
pandorabdw... how long can i run the pocket without having the backplate on (since it is acting as the heat sink?)11:02
minutepandora: you can check with "sensors"11:02
minutepandora: it should shut down automatically when critical temp is reached (i think 90 celsius). but only if linux is running! without load or gpu action, it can run without heatsink for a longer while11:03
minutejosch: sooo... the kernel was updated "slightly" and now it is broken? when was that?11:04
pandoraok was just curious because i had to remove the backplate to access the serial port... i might route it to the side at some point. The side slot seems to have a lot of space to make it permanently accessible 11:04
minutepandora: yep, i've done that too11:05
joschminute: the kernel sources are the same -- pandora upgraded from 6.8.12-1+reform20240605T104210Z to 6.8.12-1+reform20240605T104210Z11:05
joschthat's the same version but from different builds in the reform-debian-packages pipeline11:05
minutejosch: bizarre... but what changed?11:05
joschthe 6.8.12-1 bit is the upstream version11:05
joschwhich apparently didn't change, so the sources are the same11:06
- andreas-e (QUIT: Quit: Leaving) (~Andreas@2001:861:c4:f2f0::c64)11:06
joschthe reform20240605T104210Z bit comes from the date of the top git commit in reform-debian-packages11:06
minutemaybe we can sidestep this by upgrading to 6.9 P:11:06
joschthis also remained the same, so it's the same patch stack11:06
minutebut anyway, need to hurry to get to work and examine this11:06
joschminute: in all seriousness, this is something that you might want to try out first before sinking more time into this11:06
joschminute: because you have to spend the time to make 6.9 work anyways so maybe the problems goes away magically?11:07
joschand if it's still there, probably better to fix it in 6.911:07
joschminute: the thing that breaks 6.9 right now is qcalcld11:07
joschhttps://paste.debian.net/1321040/11:08
minutejosch: ok, i can fix that i guess :D11:08
joschmaybe more is broken by my rebase but the CI cannot check because it's out of space (i guess?) and cannot store artifacts11:08
minuteok, i will clean all that up11:08
joschthank you11:09
joschi am maintaining a todo list for you locally, so no worries, nothing will be forgotten ;)11:09
joschpandora: thanks a lot for your very useful logs -- that was a great help!11:10
minutejosch: which .deb could i delete on the repo to prevent upgrades in the meantime?11:10
pandorahehe u r welcome... stuff at work is broken too, so I had some time to spare11:11
joschminute: this one: https://mntre.com/reform-debian-repo/pool/main/l/linux/linux-image-6.8.12-mnt-reform-arm64_6.8.12-1%2Breform20240605T104210Z_arm64.deb11:11
minutei think it's better if apt throws an error temporaily than to have unbootable pockets11:11
minutejosch: thanks!!11:11
joschthank you too :)11:11
joschjust keeping the status quo is already quite a bit of work XD11:12
minuteyeah :011:13
minuteok, deb moved away11:14
minuteand update job pauses11:14
minutepaused11:14
- colinsane (QUIT: Quit: bye) (~colinunin@97-113-64-230.tukw.qwest.net)11:14
+ colinsane (~colinunin@97-113-64-230.tukw.qwest.net)11:15
pandorais there a chance i can boot from the old kernel on my emmc? On my other linux laptop i can just select the old kernel in grub but i am not really familiar with uboot11:29
minutepandora: perhaps you could mount the boot partition from emmc while booted from microsd and just replace the kernel (overwrite it)11:55
minutewe don't have a graphical uboot yet on imx8mp11:55
minutei mean, we kinda could, via hdmi11:55
pandoraI’ll try that tonight or so… gonna be busy until then12:06
joschpandora: this was not really an upgrade. Apt replaced version X of your kernel image with version X.12:21
joschso our /boot/vmlinuz-* remained the same12:21
+ andreas-e (~Andreas@2001:861:c4:f2f0::c64)12:35
- andreas-e (QUIT: Remote host closed the connection) (~Andreas@2001:861:c4:f2f0::c64)12:35
+ andreas-e (~Andreas@2001:861:c4:f2f0::c64)12:36
+ chomwitt (~chomwitt@2a02:587:7a02:1a00:1ac0:4dff:fedb:a3f1)12:36
+ mjw (~mjw@gnu.wildebeest.org)12:46
minutejosch: ok, so i will look into 6.9 now12:49
joschminute: it didn't build on the CI yet but you can test locally with:12:51
joschsh -xc 'BASESUITE=experimental; . ./setup.sh; cd linux; . ./build.sh'12:51
josch(CI fails due to the http error)12:51
joschand i'll probably be away until the evening12:51
joschminute: as usual, please feel free to force push any fixes into the linux6.9 branch12:58
joschthat's this MR: https://source.mnt.re/reform/reform-debian-packages/-/merge_requests/4112:58
minutejosch: ok, i just checked out the linux6.9 branch12:59
minutejosch: iirc just qcacld needs fixup, yes?12:59
minuteah and i need to clean the CI stuff12:59
joschminute: and maybe other fixes are needed -- i only made sure the patches apply but didn't build it yet because i only have a poor weak a311d here :D13:00
minute> /dev/sdb         98G   78G   16G  84% /mnt/gitlab-artifacts13:00
minutehmm13:00
minuteanyway, how did i delete old artifacts... i think using some gitlab cli tool13:01
joschi used the web interface in the past: https://source.mnt.re/reform/reform-debian-packages/-/artifacts13:02
joschthese eat 7.5 G each: https://source.mnt.re/reform/reform-system-image/-/artifacts13:02
minuteok, gonna click some trashcans13:03
minuteah yeah, it takes a while for the actual files to get deleted in the background i guess...13:04
joschaha! i made the same experience13:05
joschgood to hear that it was not just me :)13:05
joschif you have a shell open, can you really verify that /mnt/gitlab-artifacts does not immediately get cleaned up?13:05
minute> /dev/sdb         98G   78G   16G  84% /mnt/gitlab-artifacts13:06
minutenothing :D13:06
joscho013:06
joschwell, then at least i was not imagining things...13:07
minutenon-monolithic webapps were a mistake...13:07
minute   22.4 GiB [##########] /2024_06_18                                                                                                                                         13:07
minute   14.9 GiB [######    ] /2024_06_0513:07
joschoh progress bars!13:07
minutethat's ncdu :D13:07
joschoh :D13:07
joschncdu... like not-cdu -- looks like my new tool of choice ;)13:08
- colinsane (QUIT: Remote host closed the connection) (~colinunin@97-113-64-230.tukw.qwest.net)13:08
minute:D13:11
minutethere's some discussions saying there's a 15 minute interval for cleanups13:11
minuteso lets check again at 13:1513:11
minutewow, my pc is on 6.3.0 :D13:13
+ colinsane (~colinunin@97-113-64-230.tukw.qwest.net)13:13
josch"non-monolithic webapps were a mistake" -- i was sceptical of monolithic webapps but i have been using gitea on my own server for years (after a horrifying experience with gitlab -- too much maintenance cost for the gain) and i'm *very* happy with gitea. I know it now has some more forks and is quite actively developed. Not saying you should switch. Just saying that I changed my opinion over time 13:13
joschregarding monolithic webapps.13:13
minutethe best webapps are single executable + sqlite13:13
josch+113:14
joschyou just know your stuff is in these files and that's it13:14
minuteyeah13:14
minute> sudo apt install linux-image-6.9-amd6413:14
minutejosch: > Total artifacts size 7.46 GiB13:16
minuteok, 15 minute cleanup interval confirmed :313:16
josch\o/13:16
minutei am once again cloning linux git13:18
joschi'm also excited to hear about your tests of 6.9 on rk3588. The patch stack is 13 patches smaller and i was able to remove four patches from other stacks.13:18
minutejosch: oh great :313:19
joschminute: we could also throw this patch stack in if you like: https://lore.kernel.org/linux-phy/20240506124836.3621528-1-heiko@sntech.de/13:25
minutejosch: yesss please13:30
minutelooks like qcacld2 needs only a 2-byte change13:30
minutejosch: you wanted me to reassign a runner for...?13:34
minutejosch: qcacld2 fix merged13:41
minutejosch: > Refusing to run pipeline in the 'reform' namespace with BASESUITE not set to 'unstable'13:43
minute++ echo 'to avoid accidentally tainting the repository at https://mntre.com/reform-debian-repo.'13:43
joschminute: since you disabled the repo updates, just comment out those lines in .gitlab-ci.yml13:45
- colinsane (QUIT: Remote host closed the connection) (~colinunin@97-113-64-230.tukw.qwest.net)13:45
joschminute: lets do the runner re-assignment another time and focus on 6.9 today -- the runner thing can wait and there is a MR to avoid forgetting about this13:46
+ colinsane (~colinunin@97-113-64-230.tukw.qwest.net)13:46
minutejosch: ok, thanks for confirming what i was thinking @ gitlab-ci13:48
minutethis slaps https://mastodon.social/@Invictvs@indieweb.social/11265441286197102713:48
- mjw (QUIT: Ping timeout: 264 seconds) (~mjw@gnu.wildebeest.org)13:50
minutefrom tomorrow i'll have a short vacation (till including wednesday) and i think i wanna sidegrade my pocket to a311d for that. lets see if i have time tonight13:52
joschfwiw the mipi dsi patch applies cleanly on top of 6.913:53
minutejosch: btw it might be interesting to change the imx8mq patchstack to https://patchwork.kernel.org/project/linux-phy/cover/20240306101625.795732-1-alexander.stein@ew.tq-group.com/13:53
minutejosch: nice... i think it has a second part though13:53
josch"playing Bach" nice! :D13:54
joschah there is a v15 now, i can take care of this later13:54
josch <= afk now until tonight13:55
minutejosch: thank you and cya l8r!13:55
- andreas-e (QUIT: Quit: Leaving) (~Andreas@2001:861:c4:f2f0::c64)13:55
* Guest7886 -> mjw13:56
pandora@josch: “this was not really an upgrade. Apt replaced version X of your kernel image with version X” . Hehe ok then I don’t really understand what broke. If I replace x with the same x it shouldn’t brake, should it? :D14:44
minutepandora: actually just copying over the vmlinuz might not work because module signature might differ14:50
pandoraI’ll give it a shot anyway because I have no idea what else to try really14:54
joschpandora: you can extract a working kernel together with fitting modules from the sd-card image that you flashed before. You need to copy vmlinuz to /boot (do not touch your working initrd) and you need to copy the corresponding /usr/lib/modules/ directory16:12
pandoraThx will test that when I get back home16:19
+ mark_ (~mjw@gnu.wildebeest.org)16:31
Twodisbetterminute: hopefully as a little cheer me up for Friday, I just want you to know that MNT is referenced quite a bit on the Framework forums, and I believe that discussion has driven a bit of traffic your way. MNT deserves a lot more limelight for sure. 17:11
minuteTwodisbetter: nice17:27
minutereform next motherboards has first lifesigns, i have linux running on imx8mp over serial (not much use without ports though)17:28
minutejosch: and also the initramfs17:28
minutebecause it has modules17:28
minutejosch: linux6.9 branch has linux6.8.12 :/ https://source.mnt.re/reform/reform-debian-packages/-/jobs/4620/artifacts/browse/repo/pool/main/l/linux/17:38
Twodisbetterminute: is there always something the newer kernel has that you want, or is this just a standard to move to the newer kernel?17:43
pandora@josch: it worked! I copied the vmlinuz and the modules and now it is booting again! thx for the help18:08
minutepandora: nice18:26
+ nybble (~sprang@user/csprng)19:46
jfredminute: woohoo congrats on the reform next progress :D19:50
minutejfred: thaaanx19:50
* mjw -> Guest752120:23
* mark_ -> mjw20:23
joschminute: you have to run the pipeline with BASESUITE=experimental22:17
joschTwodisbetter: it depends on the module. For LS1028A for example, hardware support is nearly all upstreamed. Not so much for other modules though. Most in flux right now is hardware support for rk358822:18
minutejosch: ohh... i thought i had clicked retry22:26
minutejosch: oh i saw you just ran the pipeline!22:26
josch:)22:27
minutejosch: i'm going on a 5-days holiday now (incl the weekend) but taking the a311d module with me, and i have imx8mplus and rk3588, so can make a few tests at home22:27
joschminute: anything on the TODO list that you'd like the community to take care of during your time afk?22:28
minutejosch: just be good while i'm gone!! :D22:28
joschi hope you manage to have a time as worry free and relaxing as it can be during your holidays :)23:17
joschnow i need to remember why past-me explicitly added an override BASESUITE=unstable to the build_linux job... hrm...23:18
joschlets undo this and see what breaks23:21
joschthis is when i wish that past-me was writing more verbose commit messages...23:21
minutejosch: ah, i just wanted to ask why the packages are still 6.8...23:43
joschminute: no worries, 6.9 now fails to build -- much better :)23:45
joschso in the end, no clue why I put BASESUITE=unstable in there... it seems to do the right thing without it and the wrong thing with it23:45

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!