• 0 Posts
  • 72 Comments
Joined 1 year ago
cake
Cake day: August 4th, 2023

help-circle
  • More than a decade ago a user came into #ubuntu-server on Freenode (now libera.chat ) and said that they had accidentally run “rm -rf /* something*” in a root shell.

    Note the errant space that made that a fatal mistake. I don’t remember how far it actually got in deleting files, but all of /bin/ /sbin/ and /usr/ were gone.

    He had 1 active ssh connection, and couldn’t start another one.

    It was a server that was “in production”, was thousands of miles away from him, and which had no possibility for IPMI / remote hands.

    Everyone (but me) in the channel said that he was just SoL and should just give up.

    I stayed up most of the night helping him. I like challenges and I like helping people.

    This was in the sysv-init (maybe upstart) days, and so a decent number of shell scripts were running, and using basic *nix commands.

    We recovered the bash binary by running something along the lines of

    bash_binary_contents="$( </proc/self/exe)"
    printf "%s" > /tmp/bash
    

    (If you can access “lsof” then “sudo lsof | grep deleted” will show you any files that are open, but also “deleted”. You may be surprised at how many there are!)

    But bash needed too many shared libraries to make that practical.

    Somehow we were able to recover curl and chmod, after which I had him download busybox-static. From there we downloaded an Ubuntu LiveCD iso, loop mounted it, loop mounted the squashfs image inside the iso, and copied all of /bin/ , /sbin/ , /etc , and so on from there onto his root FS.

    Then we re-installed missing packages, fixed up /etc/ (a lot of important daemons, including the one that was production critical, kept their configuration files open, and so we were able to use lsof to find the magic symlinks to them in /proc/$pid/fd/ and just cp them back into /etc/.

    We were able to restart openssh-server, log in again, and I don’t remember if we were brave enough to test rebooting.

    But we fucking did it!

    I am certainly getting a lot of details wrong from memory. It’s all somewhere at irclogs.ubuntu.com though. My nick was / is Jordan_U.

    I tried to find it once, and failed.


  • It’s at least gotten a bit better.

    There was a time when Photoshop and other programs used a copy-protection scheme that overwrote parts of grub, causing the user not to be able to boot Linux or Windows.

    They knew about it, and just DGAF. I don’t remember their exact FAQ response, but it was something along the lines of “Photoshop is incompatible with GRUB. Don’t dual boot if you use Photoshop.”

    Grub still has code for BIOS based installs that uses reed-solomon error correction at boot time to allow grub to continue to function even if parts of its core.img were clobbered by shitty copy protection schemes for Windows software.






  • I tried to solve these cross-distro compatibility problems in a generic way with this “standard”, more years ago than I’d like to think about:

    https://www.supergrubdisk.org/wiki/Loopback.cfg

    If someone wants to come up with a bootloader agnostic solution rather than one tied to grub, like an extension to Bootloader spec , https://www.freedesktop.org/wiki/Specifications/BootLoaderSpec/ , I’d be happy to evangelize it and add support to grub for using it.

    I’m not aware of any other bootloader that supports reading a config file that exists within an iso though, and secure boot support may add additional complications.

    Bottom line:

    I feel like we could relatively easily get to a point where every Live iso that actually supports loop booting can just be added, as a file, to your USB drive (from Windows, or your android phone even) and be detected at boot in a nice little menu, no editing of config files needed.

    I don’t have the time or spoons to get the Linux community there alone, but if people are interested in helping I’m more than happy to pick this up again.

    (Note: Please don’t blindly suggest “Just chain load the iso!” Things aren’t that easy, unfortunately).




  • [email protected]

    Sorry again. I wrote this last comment (and this one, TBH) from my phone and “–iso=s” should have been “–iso-8601=s” . I’ve edited my comment and the command should now work (Making a backup of your grub.cfg containing the date, to the second, in the filename. I did that to hopefully avoid you running the same command again after trying some fixes and accidentally clobbering your backup).


  • Ahh, sorry.

    For Fedora it looks like the default /etc/default/grub looks like this:

    GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="rhgb quiet" GRUB_DISABLE_RECOVERY="true" GRUB_ENABLE_BLSCFG=true

    ( Taken from https://discussion.fedoraproject.org/t/how-to-regenerate-etc-default-grub/72677/9 )

    If you’re using LVM / LUKS you may need additional kernel parameters, like resume=… for suspend to disk to work properly.

    Please, before doing anything else, post the output of the following:

    cat /proc/cmdline

    And make a backup of your existing grub.cfg with:

    sudo cp /boot/grub2/grub.cfg /boot/grub2/grub.cfg-backup-$(date --iso-8601=s)

    Also, be sure that you have a LiveUSB on hand. You don’t want to be SOL if we break something and can’t boot again without fixing it first.



  • Jordan_U@lemmy.mltoLinux@lemmy.mlI can't edit /etc/default/grub
    link
    fedilink
    arrow-up
    2
    arrow-down
    2
    ·
    edit-2
    9 months ago

    What version of Ubuntu are you using?

    What is the output of the following command?:

    dpkg -l | grep grub

    If you urgently want your grub menu to default to the first entry that can be done first, but unless that’s needed I’d prefer to get to the root of the problem(s) and get a proper fix.


  • This should get you back to defaults:

    sudo cp /usr/share/grub/default/grub /etc/default/grub && sudo update-grub

    At some point you definitely did accidentally write to /etc/default/grub when you meant to write to /boot/grub/grub.cfg.

    There’s no shame in that; Grub’s configuration process is very confusing and counter-intuitive.

    Everybody who has used Linux long enough has stories of breaking their systems in sillier ways, and this didn’t even really break your system 🙂.


  • Pulseaudio used features of sound cards (most prominently the hardware read pointer) that ALSA/dmix alone never used.

    ALSA/dmix could allow you to get the same power savings as pulseaudio if you set the hardware ring buffer size to, say, 2 seconds.

    And that would be fine of you were just playing some music, but if you were also chatting and wanting to get prompt notification sounds they would always be delayed between 0 and 2 seconds depending on where the hardware read pointer happened to be when the system tried to play a notification sound.

    ALSA/dmix could also allow you to set a tiny buffer size. Then your music would play, and your notification sounds would play promptly too. But if you were just playing music your CPU would never be able to go into the lower power sleep states because it would need to wake up every centisecond to service the tiny ring buffer.

    That would kill your battery life.

    Pulseaudio’s (terribly named) “glitch free” audio feature was the first solution for Linux that allowed you to get power savings and low-ish latency. Your mp3 player filled up the ring buffer once every two seconds, and if a notification came in pulseaudio would look at where the hardware read pointer was, take the contents of the buffer that was just about to be read, and mix the notification sound into it, writing the newly mixed sound data to the buffer just before the sound card read it.

    So, from the user’s perspective nothing interesting seemed to happen, but they get better battery life and things like notifications or game sounds work like they expect them to.

    ALSA drivers would commonly advertise support for accurately and precisely reporting the position of the hardware pointer, but since nothing actually used that info before, many drivers gave incorrect results, which would only cause problems when using pulseaudio.




  • Bash scripts are rarely the best choice for large, complicated, programs or for software that requires complex data structures. (Git is very much in both categories)

    In bash there are so many ways to accidentally shoot yourself in the foot that it’s absurd. That can lead to bizarre seeming behavior, which may break your script, or even lead to a security vulnerability.

    There are things that make it a bit more manageable, like “[[]]” instead of “[]”, but a frustrating number of such things are bash specific and this is written for the subset that is POSIX shell, meaning you don’t even get those token niceties.

    Where you generally “want” to use POSIX sh is for relatively simple scripts where the same file needs to run on Linux with bash, Linux with only BusyBox sh, OSX with zfs (and an outdated version of bash which will never be updated because Apple refuses to ship any GPLv3 software in MacOS).

    This is not that, and so one would expect that:

    1. The developer of this git implementation has poor / foolish judgement.

    2. Shit will be buggy

    3. Shit will be insecure

    4. Shit will be a PITA to try to troubleshoot or fix

    5. And shit will be slow, because bash is slow and this isn’t a job that you can just hand off all of the heavy lifting to grep / sed / awk*, because the data structures don’t lend themselves to that.

    * You could write the entire program in awk, and maybe even end up with something almost as fast as a python implementation done in ⅒ the time, but that would be terrible in other ways.