A Waldorf netinstall script

johnraff · 2012-08-04 05:34:02

Several good people have posted scripts in the past for installing a system from the Debian netinstall core system, though none specifically for #! Waldorf so far. This one, however is somewhat different:
*) It's monstrously clunky and bloated.
*) It requires a download of a tar.gz file.
also
*) It takes hours to install the system.
*) You don't get a live session.
*) It's buggy and you're lucky if it works neutral
(Actually I'm hoping as of 2013/2/20 that most of the bugs have been squashed.)

However, I'm sort of hopeful it might eventually turn out to be useful.
*) It might take some of the heat out of the 700MB limit question.
*) It's intended to be usable by a relatively inexperienced Linux user.
*) It should make an identical system, making forum support easier.
*) It should be quite easy to use it as a platform for a CrunchBang Lite or any other CrunchBang respin you want to make and share with the rest of us.
*) The bloatedness is partly because I've tried to make the installer as fail-safe and stable as possible. A full record of system changes is made, and it's possible to run the script more than once (maybe after tweaking a config file) without breaking anything. It ought to be possible to write an "uninstall" script if there is a demand for one. I figured some more lines of code and extra milliseconds to execute don't matter much in this context.

Basic structure: install script, some lists of apps to install, two directory trees of system files to install (before and after app install) and a couple of other config files.

To customize, you could edit the apps lists and sysfiles2/etc/skel/.config/openbox/{autostart,rc.xml,menu.xml}
You could go on to put new themes in sysfiles2/etc/skel/.themes ...

If anyone would like to play with it and report back with problems and ideas for improvment I'd be more than grateful.

I'm posting the current version of the script itself here, but in order to use it you need the full package with all the other files, which for the moment you can download here:
http://dl.dropbox.com/u/10808732/cb-netinstall.tar.gz
So to use it, in a new netinstall Debian Wheezy core system you'd do:

wget http://dl.dropbox.com/u/10808732/cb-netinstall.tar.gz
tar -xpf cb-netinstall.tar.gz
cd cb-netinstall
./install

There's a bit more info, and hints about the Debian netinstall process, in the README file in that package.

There is now a "How To" thread for users here:
http://crunchbang.org/forums/viewtopic.php?id=25098
I would suggest posting questions about, and problems with, usage on that thread, and keep this thread for discussion of the script itself - how it works, customization, suggestions and bugs.

The whole rhing is now also on Github:
https://github.com/johnraff/cb-netinstall
Pull requests for improvements or fixes are most welcome, but maybe it would be good to discuss them here first. smile

Anyway, here's the the 'install' script:

#!/bin/bash
# This is intended to be a general-purpose script to install
# a desktop environment, possibly CrunchBang,
# on a Debian netinstall base. (See README for more details.)
# It will modify and add some system files, install the necessary
# packages and copy configuration files into your home directory.
# These changes might be reversible, but no guarantees are offered,
# and all the usual caveats apply.
# This install script is intended to be run on a newly installed
# system with no personal settings or data. Attempts will be made to
# backup any such data that might exist, but do not rely on it.

# USAGE
# This script should come in a folder called cb-?-netinstall,
# along with a collection of other necessary files.
# Change directory (cd) into that folder and run
# ./install
# to start the installation process.
# A folder called .cb-netinstall-logs will be added to your home folder.
# This may safely be removed if the installation was successful.
# A folder /backup (or other name) on your root file system will hold
# backups of system files replaced during the install.

################################################

set -o nounset # do not accept unset variables
# If not running in terminal, exit with message
[[ -t 0 && -t 1 && -t 2 ]] || { echo "$0: This script must be run from a terminal" >&2; exit 1; }

[[ -r config ]] || { echo "$0: Cannot find necessary file \"config\"" >&2; exit 1; }
. config
for i in log_dir backup_dir backup_suffix necessary_files replaced_dirs
do
    [[ $i ]] || { echo "$0: Config variable $i has not been set." >&2; exit 1; }
done

logfile=error.log # temporary logfile in installer directory
user=$USER

msg() {
    echo "$1" | tee -a "$logfile"
}

bigmsg() {
    tput bold
    echo "
$1
------------------------------------------------" | tee -a "$logfile"
    tput sgr0
    sleep 3
}

errmsg() {
    tput bold
    echo "######## ERROR ########
$1
------------------------------------------------" | tee -a "$logfile" >&2
    tput sgr0
    sleep 4
}

log() {
    echo "$1" >> "$logfile"
}

warnlog() {
    echo "######## WARNING ########
$1
------------------------------------------------" >> "$logfile"
}

confirm() { # $1 is message, $2 is desired return value: 0 or 1 (passed to 'giveup')
    echo "
$1
(press enter to continue, any other key to exit)
"
    read -srn1
    [[ $REPLY ]] && giveup "terminated by $user, goodbye!" $2
}

giveup() { # $1 is message, $2 is desired return value: 0 or 1
    if [[ $2 = 0 ]]
    then
        bigmsg "$1"
    else
        errmsg "$1"
    fi
    [[ ${replaced_dirs[0]} != "none" ]] && {
        for i in "${replaced_dirs[@]}"
        do
            sudo test -d "$i" || { # if dir has not been replaced by new version, put back the original backup
                if sudo test -d  "${i%/*}"
                then
                    sudo test -d "${backup_dir}/$i" && sudo mv "${backup_dir}/$i" "${i%/*}" > >( tee -a "$logfile" ) 2>&1 || errmsg "Unable to restore missing folder $i ."
                else
                    errmsg "Cannot move $i to ${i%/*} : no such directory."
                fi
            }
        done
    }
    echo "now exiting..."
    exit $2
}

trap 'giveup "Script terminated." 1' 1 2 3 15
[[ $user = root ]] && giveup "This script should be run by a normal user, not root" 1

log "
########################################################################
Starting CrunchBang netinstall script for $user at $(date)"

clear
tput bold
echo "Hi $user, welcome to the CrunchBang Waldorf netinstall script!"
tput sgr0
echo "
$( < greeting )

This script is expected to be run just after completing a netinstall
installation of the Debian Wheezy CORE SYSTEM ONLY.
(See \"Debian Netinstall Hints\" in the README file.)
"
confirm "Would you like to start the install now?" 0

# check for needed files
missing_files=
for i in $necessary_files
do
    [[ -r $i ]] || missing_files+=" $i"
done
[[ $missing_files ]] && giveup "Some necessary files or folders were missing: $missing_files" 1 || msg "Necessary files: OK"

# setup logfile
[[ -d "$log_dir" ]] || { cp -r cb-netinstall-logs "$log_dir" || giveup "failed to copy logfiles directory into $HOME" 1; }
cat "$logfile" >> "$log_dir"/install.log
rm "$logfile" # finished with temporary logfile
logfile="$log_dir"/install.log # this logfile will remain after the install
bigmsg "messages are being saved to $logfile"
cp pkgs-manual pkgs-auto "$log_dir"

# check debian version FIXME Is /etc/debian_version the best way?
grep -q '\(wheezy\|7.0\)' /etc/debian_version && msg "Debian version: OK"|| { warnlog "/etc/debian_version reads: $(cat /etc/debian_version)"
    confirm "You do not appear to have Debian Wheezy installed.
If you think this is incorrect,
you may wish to continue with the installation,
otherwise it would be safer to stop.
Would you like to continue anyway?" 1; }

# is X already installed? FIXME Is this the best way to check if there is a desktop system installed already?
hash startx > /dev/null 2>&1 && { warnlog "Command 'startx' is available. Desktop system already installed?"
    confirm "You seem to have some kind of desktop environment already installed.
If you continue with this CrunchBang installation you may have problems,
or even end up with an unusable system.
Would you like to continue anyway?" 1; }

# can use sudo?
echo "
You will need your password to perform certain system tasks.
Please enter it now and it will be stored for a while.
(You may need to enter it again later.)"
sudo -v || giveup "You do not appear to have permission to use sudo,
which is needed in this script.
Please make the necessary adjustments to your system and try again." 1

bigmsg "Upgrading system before install
(this may take a little while)"

msg "Updating database..."
sudo apt-get --quiet update > >( tee -a "$logfile" ) 2>&1 || giveup "There was a problem with 'apt-get update'.
There may be some information in ${logfile}." 1

msg "Upgrading packages..."
sudo apt-get --quiet --yes dist-upgrade > >( tee -a "$logfile" ) 2>&1 || giveup "There was a problem with 'apt-get dist-upgrade'.
There may be some information in ${logfile}." 1

msg "Removing unnecessary packages..."
sudo apt-get --quiet --yes autoremove > >( tee -a "$logfile" ) 2>&1 || giveup "There was a problem with 'apt-get autoremove'.
There may be some information in ${logfile}." 1

log "Packages originally installed by the Debian netinstaller are recorded in $log_dir/netinstall-core-apps"
[[ -f "$log_dir"/netinstall-core-apps ]] || apt-mark showmanual > "$log_dir"/netinstall-core-apps
log "Packages originally auto-installed by the Debian netinstaller are recorded in $log_dir/netinstall-auto-apps"
[[ -f "$log_dir"/netinstall-auto-apps ]] || apt-mark showauto > "$log_dir"/netinstall-auto-apps

msg "Installing rsync."
sudo apt-get --quiet --yes install rsync > >( tee -a "$logfile" ) 2>&1 || giveup "Failed to install necessary package 'rsync'." 1

############ file copying functions ############
# rsync would do almost the same, but this allows us to keep records and be picky about corner cases.

are_identical() { # two files have same content and same permissions
    sudo cmp -s "$1" "$2" && [[ $(sudo stat -c %a "$1") = $(sudo stat -c %a "$2") ]] && return 0
    return 1
}

install_sysdir() { # recursively copy contents of $1 into $2, make backups in $3 of replaced files and keep records of changes
    [[ $# = 3 ]] || giveup "install_sysdir() needs three arguments" 1
    # use 'sudo test' in case file is only accessible to root
    sudo test -d "$1" || giveup "$1 is not an existing directory" 1
    sudo test -d "$2" || giveup "$2 is not an existing directory" 1
    sudo mkdir -p "$3" > >( tee -a "$logfile" ) 2>&1 || giveup "Unable to make directory $3." 1
    for i in "$1"/*
    do
        sudo test -e $i || break # $1 is empty
        filename="${i##*/}"
        target="${2%/}/${filename}" # avoid double slash if $2 is /
        if sudo test -d $i
        then
            if sudo test -e "${target}"
            then
                sudo test -d "${target}" || giveup "${target} exists, but is not a directory." 1
                sudo rsync --dirs --perms "${target}" "$3" > >( tee -a "$logfile" ) 2>&1 || giveup "There was a problem copying ${target} to $3/${filename}." 1
            else
                msg "adding directory ${target}"
                sudo rsync --dirs --perms "$i" "$2" > >( tee -a "$logfile" ) 2>&1 || giveup "There was a problem copying $i to $2." 1
                echo "\"${target}\"" >> "$log_dir"/sysdirs-added
            fi
            install_sysdir "$i" "${target}" "$3/${filename}"
        else
            if sudo test -e "${target}"
            then
                if are_identical "$i" "${target}"
                then # probably this file was added on a previous run of script
                    msg "$i and ${target} are identical"
                else
                    msg "replacing ${target}"
                    if sudo test -e "$3/${filename}"
                    then
                        msg "A backup copy of ${target} already exists..." # do our best to keep meaningful backups
                        if are_identical "${target}" "$3/${filename}"
                        then
                            msg "but is identical with ${target}"
                        else
                            msg "but an extra backup will be made at $3/${filename}${backup_suffix}"
                            sudo mv "${target}" "$3/${filename}${backup_suffix}" > >( tee -a "$logfile" ) 2>&1 || giveup "There was a problem moving ${target} to $3/${filename}." 1
                        fi
                    else
                        sudo mv "${target}" "$3/${filename}" > >( tee -a "$logfile" ) 2>&1 || giveup "There was a problem moving ${target} to $3/${filename}." 1
                    fi
                    sudo cp --preserve=mode "$i" "$2" > >( tee -a "$logfile" ) 2>&1 || giveup "There was a problem copying $i to $2." 1
                    echo "\"${target}\"" >> "$log_dir"/sysfiles-replaced
                fi
            else
                msg "adding ${target}"
                sudo cp --preserve=mode "$i" "$2" > >( tee -a "$logfile" ) 2>&1 || giveup "There was a problem copying $i to $2." 1
                echo "\"${target}\"" >> "$log_dir"/sysfiles-added
            fi
        fi
    done
}

install_userdir() { # recursively copy contents of $1 into $2, make backups of replaced files in $2 and keep records of changes
    [[ $# = 2 ]] || giveup "install_userdir() needs two arguments" 1
    [[ -d $1 ]] || giveup "$1 is not an existing directory." 1
    [[ -d $2 ]] || giveup "$2 is not an existing directory." 1
    for i in "$1"/*
    do
        [[ -e $i ]] || break # $1 is empty
        filename="${i##*/}"
        if [[ -d $i ]]
        then
            if [[ -e "$2/${filename}" ]]
            then
                [[ -d "$2/${filename}" ]] || giveup "$2/${filename} exists, but is not a directory." 1
            else
                msg "adding directory $2/${filename}"
                rsync --dirs --perms "$i" "$2" > >( tee -a "$logfile" ) 2>&1 || giveup "There was a problem copying $i to $2." 1
                echo "\"$2/${filename}\"" >> "$log_dir"/userdirs-added
            fi
            install_userdir "$i" "$2/${filename}"
        else
            if [[ -e "$2/${filename}" ]]; then
                if are_identical "$i" "$2/${filename}"; then # probably this file was added on a previous run of script
                    msg "$i and $2/${filename} are identical"
                else
                    msg "replacing $2/${filename}"
                    # keep original backup, but try not to fill user's home with meaningless backup files
                    mv --no-clobber "$2/${filename}" "$2/${filename}${backup_suffix}" > >( tee -a "$logfile" ) 2>&1 || giveup "There was a problem renaming $2/${filename} to $2/${filename}${backup_suffix}." 1
                    cp --preserve=mode "$i" "$2" > >( tee -a "$logfile" ) 2>&1 || giveup "There was a problem copying $i to $2." 1
                    echo "\"$2/${filename}\"" >> "$log_dir"/userfiles-replaced
                fi
            else
                msg "adding $2/${filename}"
                cp --preserve=mode "$i" "$2" > >( tee -a "$logfile" ) 2>&1 || giveup "There was a problem copying $i to $2." 1
                echo "\"$2/${filename}\"" >> "$log_dir"/userfiles-added
            fi
        fi
    done
}

########## end file copying functions ##########

bigmsg "Adjusting package management (apt) settings
and adding CrunchBang repositories.
Replaced files will be backed up to ${backup_dir}/etc/apt"

# If extra_repos exists, append to sources.list
# instead of totally replacing it.
# This way you keep the Debian mirror chosen during the netinstall.
# In this case, omit sysfiles1/etc/apt/sources.list .
# If sysfiles1/etc/apt/sources.list is also present, user will be asked
# which to use.
append_repos=false
[[ -s extra_repos ]] && { # extra_repos is non-zero
    if [[ -s sysfiles1/etc/apt/sources.list ]]
    then
        if are_identical /etc/apt/sources.list sysfiles1/etc/apt/sources.list
        then
            msg "/etc/apt/sources.list has already been modified."
        elif [[ $(</etc/apt/sources.list) = *$(<extra_repos)* ]]
        then
            msg "extra_repos already appended to /etc/apt/sources.list"
            mv sysfiles1/etc/apt/sources.list sysfiles1/etc/apt/sources.list~cb-orig2~ # preempt sysfiles1/etc/apt/sources.list from being copied intp /etc again
        else
            echo "
Your /etc/apt/sources.list file entries will be replaced by the
Crunchbang default repositories. If you would prefer to keep the
Debian main mirror you chose during the netinstall, choose the \"append\"
option instead: the needed extra repositories will be appended to your
existing sources.list file. Otherwise choose the \"replace\" option.
( A backup copy of your sources.list file will be kept
 in ${backup_dir}/etc/apt )

If you are unsure, choose \"replace\".

(Press a to append, any other key to replace.)
"
            log "Both extra_repos and sysfiles1/etc/apt/sources.list exist.
User chooses which to use."
            read -rsn1
            if [[ $REPLY = a ]]
            then
                mv sysfiles1/etc/apt/sources.list sysfiles1/etc/apt/sources.list~cb-orig~
                append_repos=true
                log "User chose to append extra_repos to sources.list."
            else
                log "User chose to replace sources.list."
            fi
        fi
    else
        append_repos=true
    fi
}
$append_repos && {
    msg "adding extra repositories to existing sources.list file"
    sed '/deb-src/s/^/#/' /etc/apt/sources.list 2> >( tee -a "$logfile" ) > sysfiles1/etc/apt/sources.list || giveup "There was a problem editing sources.list." 1
    cat extra_repos 2> >( tee -a "$logfile" ) >> sysfiles1/etc/apt/sources.list || giveup "There was a problem appending extra_repos to sources.list." 1
}
[[ -s sysfiles1/etc/apt/sources.list ]] || giveup "There is no sources.list file."

msg "installing apt configuration files"

install_sysdir sysfiles1 / "${backup_dir}"
bigmsg "Apt settings adjusted."

# Install necessary APT keys.
[[ -r apt-keys ]] && {
    . apt-keys
}

bigmsg "Installing packages needed for the system.
This may take some time."
#bigmsg "Please press 'y' if prompted at some point."

msg "updating apt database..."
sudo apt-get --quiet update > >( tee -a "$logfile" ) 2>&1 || {
    warnlog "'apt-get --quiet update' returned an error"
    confirm "There was a problem with 'apt-get update'.
There may be some information in ${logfile}.
Would you like to continue anyway, or exit?" 1
}

msg "upgrading existing packages..."
# FIXME grub may be downgraded, requiring a prompt or apt-get will abort if using --yes.
# Using the (dangerous) --force-yes option for now.
sudo apt-get --yes --force-yes --quiet dist-upgrade > >( tee -a "$logfile" ) 2>&1 || {
    warnlog "'apt-get --quiet dist-upgrade' returned an error"
    confirm "There was a problem with 'apt-get dist-upgrade'.
There may be some information in ${logfile}.
Would you like to continue anyway, or exit?" 1
}

bigmsg "Installing new packages.
There may be around 1000 packages to install,
 so this will probably take a while..."
sudo apt-get --yes --quiet --ignore-missing install $( sed 's/\#.*$//' pkgs-manual ) > >( tee -a "$logfile" ) 2>&1 || {
    warnlog "'apt-get --yes --quiet --ignore-missing install' returned an error"
    confirm "There was a problem installing some packages.
There may be some information in ${logfile}.
You might want to exit the script, comment out packages causing problems
in pkgs-manual, and run the script again.
If the missing packages are unimportant, you might prefer to continue
with the install and fix the issues later.
Would you like to ignore the errors and continue now,
or exit and try to fix the problems?" 1
}

# If there are packages that need to be installed without recommends,
# do it here.
unset pkgs_norecs
pkgs_norecs=($( sed 's/\#.*$//' pkgs-norecs )) && [[ ${#pkgs_norecs[@]} -gt 0 ]] && {
    bigmsg "Installing extra packages, without recommends."
    sudo apt-get --no-install-recommends --yes --quiet --ignore-missing install ${pkgs_norecs[@]} > >( tee -a "$logfile" ) 2>&1 || {
        warnlog "'apt-get --yes --quiet --ignore-missing install' returned an error"
        confirm "There was a problem installing some packages.
    There may be some information in ${logfile}.
    You might want to exit the script, comment out packages causing problems
    in pkgs-norecs, and run the script again.
    If the missing packages are unimportant, you might prefer to continue
    with the install and fix the issues later.
    Would you like to ignore the errors and continue now,
    or exit and try to fix the problems?" 1
    }
}

missing_auto=($( comm -23 <( sed 's/\#.*$//' pkgs-auto | sort ) <( apt-mark showauto | sort ) ))
[[ ${#missing_auto[@]} -gt 0 ]] && {
    msg "Missing auto packages:
${missing_auto[@]}"
    bigmsg "Installing extra auto-installed packages."
    sudo apt-get --no-install-recommends --yes --quiet --ignore-missing install ${missing_auto[@]} > >( tee -a "$logfile" ) 2>&1 || {
        warnlog "'apt-get --no-install-recommends --yes --quiet --ignore-missing install' returned an error"
        confirm "There was a problem installing some packages.
There may be some information in ${logfile}.
You might want to exit the script, comment out packages causing problems
in pkgs-auto, and run the script again.
If the missing packages are unimportant, you might prefer to continue
with the install and fix the issues later.
Would you like to ignore the errors and continue now,
or exit and try to fix the problems?" 1
    }
    sudo apt-mark auto ${missing_auto[@]} > >( tee -a "$logfile" ) 2>&1 || {
        warnlog "'apt-mark auto' returned an error"
        confirm "There was a problem marking some packages as auto.
There may be some information in ${logfile}.
You might want to exit the script, comment out packages causing problems
in pkgs-auto, and run the script again.
Alternatively, you might prefer to continue
with the install and fix the issues later.
Would you like to ignore the errors and continue now,
or exit and try to fix the problems?" 1
    }
    msg "extra auto packages installed"
}

bigmsg "New packages installed"

bigmsg "Copying in new system files.
Replaced files will be backed up to ${backup_dir}"

# move dirs that will be replaced to backup dir
[[ ${replaced_dirs[0]} != "none" ]] && {
    for i in "${replaced_dirs[@]}"
    do
        sudo test -d "$i" || { errmsg "directory $i set in \"replaced_dirs\" in file \"config\" does not exist"; continue; }
        if sudo test -d "${backup_dir}/$i"
        then
            msg "${backup_dir}/$i already exists.
No need to backup $i"
        else
            bigmsg "$i will be replaced: moving it to ${backup_dir}" # we want to completely replace the directory
            sudo mkdir -p "${backup_dir}/${i%/*}" > >( tee -a "$logfile" ) 2>&1 || giveup "Unable to make directory ${backup_dir}/${i%/*}." 1
            sudo mv "$i" "${backup_dir}/${i%/*}" > >( tee -a "$logfile" ) 2>&1 || giveup "Unable to move $i to ${backup_dir}/${i%/*}" 1
        fi
    done
}

shopt -s dotglob # want dotfiles too
install_sysdir sysfiles2 / "${backup_dir}"
shopt -u dotglob
bigmsg "system files installed"

bigmsg "Updating your personal configuration files.
Replaced files wil be backed up with a suffix of $backup_suffix"

shopt -s dotglob # want dotfiles too
install_userdir /etc/skel $HOME
shopt -u dotglob

bigmsg "Files updated."

# final tweaking
bigmsg "Adjusting some system settings."
[[ -r postinstall_commands ]] && {
    . postinstall_commands
}


bigmsg "INSTALL FINISHED!"
confirm "You need to restart your computer to log into your new system.
Would you like to continue to restart now,
or exit the script and restart later?" 0
bigmsg "REBOOTING...
(thank you for your patience so far)"
sudo shutdown -r now
exit

edit 120808 A couple of tweaks to slightly increase the chance of the script completing successfully and make the backups a bit more robust. (The dropbox url will always point to the latest version.)
edit 120819 Now installs the latest version of Waldorf, and the script has had some more tweaking to make it a bit more solid.
edit 120823 Download and install the CrunchBang GPG key to avoid those error messages.
edit 120908 More modular, a bit faster, and hopefully architecture-independent. See this post.
edit 130220 Finally updated to match the #! 130119 release. Yet more cofiguration has been pulled out of the script into the external files. Thanks to corenominal's new packaging system this installer might stay usable even after CrunchBang version upgrades. Customized versions might also stay usable even after the script itself has been upgraded. Some bugs have been fixed. smile A bit more here. (Also edited the explanation above a bit to match the current situation better.)
edit 130301 Added link to how-to thread.
edit 130420 The whole package is now on Github.

Last edited by johnraff (2013-04-19 16:55:29)

sunfizz98 · 2012-08-04 09:12:32

Nifty script. I will have to dig through it and try to understand the functions for myself smile

johnraff · 2012-08-07 16:08:29

Hi sunfizz - you were one of the people I was thinking of who'd already worked on this kind of stuff. Any advice you might have will be appreciated. smile

Those functions - most of them are about sending messages to a logfile and the terminal, with extra bits like ***WARNING*** tacked on or whatever. Just to save me from typing out the same code multi times. That big install_dir function just does the same as rsync or something, but also makes a record of all the files changed, in the hope of being able to undo it all later if necessary.

machinebacon · 2012-08-07 16:20:42

I love the idea. Thank you, John. This will open new opportunities for users to share their spins - simply by providing a link to pastebin or forum code (applist/sysfiles). Big +1! Calls for a future sub-section big_smile

Last edited by machinebacon (2012-08-07 16:21:43)

johnraff · 2012-08-08 04:50:41

Hi mb, thanks for the moral support! smile

It's still kind of messy at the moment, and extra bits of checking and logging keep getting added, but a lot of that is for bugfixing and the final version ought to be a bit simpler, at least from the end user's point of view.

corenominal · 2012-08-08 10:28:56

I have not had chance to actually test it yet, but I really like what I see. Nice work, John, and thank you for sharing. smile

rhowaldt · 2012-08-08 11:35:21

@john: looks very good! i skimmed through the script and i noticed all the logging functions, indeed a script as you describe, 'rough draft' kinda thing. can't try it out right now but i might be getting a spare PC in the near future, which will then be open for experimentation, and i'll be sure to check out the script by then!

el_koraco · 2012-08-08 11:57:44

This is very high quality stuff.

pghjake · 2012-08-09 01:30:08

Just did a new AMD64 wheezy install using the dual-arch netinstall image dated 8/4 and your script. I commented out the linux-image and linux-headers in the apps-to-install file since I had a shiny new 3.2.0-3-amd64 kernel. I had to change the libc6 to i386 rather than i686 as the i686 was not available. There are 3 versions of libboost-iostreams in the apps-to-install, but only 1.49.0 was available. With those changes the script ran through with no issues. Thanks so much!

Last edited by pghjake (2012-08-09 01:31:40)

johnraff · 2012-08-09 06:02:23

@pghjake thanks for catching that!
My apps-to-install file was made by running 'apt-mark showmanual' on a freshly installed Waldorf, so of course it would have been an i686. I should mention somewhere that there's no support for AMD etc at the moment. In fact I'm a bit surprised it worked for you with only a couple of tweaks.

Going over the apps in that list is one little project ahead, but I want to try to make the resulting install as close as possible to the standard Waldorf.

Otherwise, atm the script seems to basically work, but it's still sort of fragile, easily upset by changes in the Debian repos etc, and things can get messed up if you run it twice in a row. Making it a bit more robust is the major aspect I'm thinking about right now.

pghjake · 2012-08-09 12:48:32

You may want to consider adding your "recent files" and "places" scripts to the menu.xml file, since corenominal added them to the latest Waldorf spins. I put them in their own section above "Lock Screen" but you would probably like them near the top of the menu. :-)

johnraff · 2012-08-09 18:17:43

Now both corenominal and Debian have just upgraded I'll have to go back, reinstall Waldorf, update the sysfiles and apps-to-install in the script to match, then test it out on the new Wheezy beta 1 netinstaller.

This could take a few days, especially as I'm just about to go off on a little summer holiday...

Meanwhile the script itself is due to get a couple of tweaks, to make it a bit safer if things get interrupted half-way through eg if the user presses Ctrl+C.

Maybe next week some time... smile

rizzo · 2012-08-09 19:46:53

This is a really nice project johnraff. Keep up the great work big_smile

zacharias · 2012-08-17 00:20:32

gr8 work mate. thoug, is this valid also for amd64 versions? i ask this just out for curiosity, since ill be using a 386 version.
kudos

johnraff · 2012-08-17 04:33:02

@zacharius The sad truth is I have no experience of systems beyond i386/32bit, so the script was built without a thought for anything else. pghjake above was able to tweak the apps-to-install file to get an AMD install, but apps-to-install no longer exists...

@everyone sorry, but the thing is in a very fluid state atm, and as a web download is the easiest way to get the script into my test system the version up on dropbox might be broken at the moment you download it. roll

The install method has switched from apt-get of manual-marked packages to setting the selections pulled from waldorf to get all the auto-marked packages too. I'm hoping the big bugs will be ironed out in a day or two and it will be there for folks to help find all the little bugs. smile At that point I'll have a think about if anything can be done about other architechtures and 64bit systems. (The Debian netinstaller might already be taking care of most of that.)

I'll post again when it's sort of ready to use, but meanwhile be very careful and have a look at the script before running it!

Thanks for all your encouragement!

johnraff · 2012-08-18 17:39:39

I think it's now reasonably safe to try out, if anyone feels like giving it a go. If you hit an issue you can exit the script, tweak something and run it a second (or third...) time with no harm done.

I've just used it to install Waldorf in a virtualbox machine and it seems OK, except that apt-xapian-index eats all the memory (removed it). Hoping that's a Wheezy issue that wil get fixed.

Anyway, eager to hear of any bugs you might find!

zacharias · 2012-08-20 22:45:29

johnraff wrote:

I think it's now reasonably safe to try out, if anyone feels like giving it a go. If you hit an issue you can exit the script, tweak something and run it a second (or third...) time with no harm done.
I've just used it to install Waldorf in a virtualbox machine and it seems OK, except that apt-xapian-index eats all the memory (removed it). Hoping that's a Wheezy issue that wil get fixed.
Anyway, eager to hear of any bugs you might find!

i'll be giving it a go on a virtualbox. i'll feed you back propperly once i run it.
kudos

johnraff · 2012-08-21 04:34:54

Thanks zacharias - any feedback will be appreciated! smile

rich · 2012-08-21 07:46:00

Just saw this, nice work johnraff smile

Have a whole bunch of Wheezy VMs at the moment for various things, am looking forward to testing this on a fresh netinstall.

Incidentally, i've created similar build scripts for custom Debian's, and have also used the same system where the build files were in a downloadable tar.gz.

A nice option is to add a switch where the person has the option to cp from a USB Pendrive as well as wget from a server. Useful for systems with unusual Nic (Non-Free) Firmware and therefore no internet access after the netinstall.

May try and find some free time to adapt this to 64bit as well.

Thanks for sharing.

Last edited by rich (2012-08-21 07:51:30)

zacharias · 2012-08-21 18:53:40

johnraff wrote:

Thanks zacharias - any feedback will be appreciated!

well, mate youre welcome.
i dont like just to give feedback. i also like to tell dev's what in my opinion as a end-user should/could be improved, so here goes.

Suggestions
1) welcoming screen is just ok, filled with warning and the goal behind the script. after this first screen, another one should pop and letting the user choose the machines architecture in order to asjust the update to it's propper version and or HW. this means that the update process/script needs to be slightly changed instead of beeing so straight forward (it's pretty damn good wink ) this would allow a multi-architecture support of this script.

2) i understand the goal behind the script is to get to closest #! experiece as possible to the original, but i think the user coul'd have the choose option in what in fact to install, seperated by topics. meaning, the ideia is to sperate the updates, kinda like: 1st one installs basic system #! feautures like DM WM, thunar, conky scripts, etc. after this, tools like gimp, abiword etc, (that i dont need right away) could be an option. (yes yes, i also think #! waldorf iso is waaaaay 2 big ; ) by deviding it in categories like: Office, Graphic, Multimedia, etc. etc.
this would also allow to decrese significantly the download to the user's and installation time. yeah, i know, it would be kinda like a custom install, but i think i've made my point.

The script itself
ran like a charm. it's not pretty (yet! wink ) but it does what it says. to be noticided that a keyring warning showed up after updating mirror list. i just found a nag, to had to configure again! the keyboard. everything else went just smoothly.
to be noted: i installed with with a mini.iso. prior to all during the installation of wheezy, on must be suggested no to allow root login. why? in my experience, after first reboot, my username didn't show up on the sudoers list, and this might be a nag for someone, specially as "green" as me.

i'll be more than happy to help you with further testing of this script.
kudos

johnraff · 2012-08-22 03:44:57

rich, thanks for your feedback smile

rich wrote:

A nice option is to add a switch where the person has the option to cp from a USB Pendrive as well as wget from a server. Useful for systems with unusual Nic (Non-Free) Firmware and therefore no internet access after the netinstall.

I'm not quite sure what you're suggesting here. wget is used to get the script itself into the netinstall core system. Of course the user is free to download it on another box and import it via a usb stick, but that is before the script itself is even run... When it's running, of course internet access will be needed for apt to install packages. Are you talking about an option to download packages elsewhere and for apt to get them from a usb stick? Another way might be to import the necessary non-free packages before doing the install?

May try and find some free time to adapt this to 64bit as well.

Please! I know nothing about 64bit or non-386 architectures. roll Any help there would be most welcome.

Last edited by johnraff (2012-08-23 04:57:43)

rizzo · 2012-08-22 03:55:11

Hmm, it wouldn't be too hard to come up with a 64bit version. If I get time I'll have a look smile

johnraff · 2012-08-22 04:17:00

zacharias, this kind of feedback is extremely valuable - thank you!

zacharias wrote:

Suggestions
1) welcoming screen is just ok, filled with warning and the goal behind the script.

Maybe a bit less warning and a bit more welcome? Well when that was written it was very much an alpha sort of thing for developers and the brave to experiment with, so I was trying to avoid any blame... Later if/when it all goes stable the interface might be made a bit less off-putting.

after this first screen, another one should pop and letting the user choose the machines architecture in order to asjust the update to it's propper version and or HW.

This would be great! It's definitely a long-term goal. Right now I don't have the knowlege, or any AMD etc machines to test on. I need to study up or get help. If it's just a question of adjusting the list of packages to install it might not be too difficult.

2) i understand the goal behind the script is to get to closest #! experiece as possible to the original, but i think the user coul'd have the choose option in what in fact to install, seperated by topics. meaning, the ideia is to sperate the updates, kinda like: 1st one installs basic system #! feautures like DM WM, thunar, conky scripts, etc. after this, tools like gimp, abiword etc, (that i dont need right away) could be an option. (yes yes, i also think #! waldorf iso is waaaaay 2 big ; ) by deviding it in categories like: Office, Graphic, Multimedia, etc. etc.
this would also allow to decrese significantly the download to the user's and installation time. yeah, i know, it would be kinda like a custom install, but i think i've made my point.

I was thinking about a CrunchBang-Lite installer from the beginning, and tried to make things so it would be easy to customize. The choice between one script with multi-options and a number of customized installer scripts is a tricky one. Right now I'm just trying to make a clone of the regular CrunchBang system, so that if a user comes to the forum for help they won't have to explain how they installed it. When the installer system itself has settled down I'll post a guide to making custom installers, so anyone can share their favourite CrunchBang respin. It should be quite easy, and not require any editing of the script itself. cool

to be noticided that a keyring warning showed up after updating mirror list.

This should be fixed very soon.

i just found a nag, to had to configure again! the keyboard.

Yes, that's because one of the installed packages calls for a reconfigure. It can be got over with a dpkg option to always use the default setting, but there's a possibility something more important might be missed, so maybe it's better just to accept these little prompts. In any case, any non-free software is likely to require you to agree to its conditions. (I had that for my Intel wireless stuff.)

to be noted: i installed with with a mini.iso. prior to all during the installation of wheezy, on must be suggested no to allow root login. why? in my experience, after first reboot, my username didn't show up on the sudoers list, and this might be a nag for someone, specially as "green" as me.

In the netinstall hints (in the README file) I made a point of not entering a root password. The Debian installer says that if you leave it blank you will be given sudo powers instead, which is what happened in my case (on several installs). You can still enable root login later if you want. Anyway, the script tests if the user can sudo, and exits with a message if there's a problem.

Thanks again for your comments. smile

johnraff · 2012-08-22 17:21:55

installing packages in scripts
For anyone who's irritated by all this "selections" stuff, and to anyone who might know a bit more about using apt in scripts than I do...

OK we've got a Waldorf machine with #! Waldorf newly installed the regular way, and a Target machine waiting to be Waldorfed...

To duplicate Waldorf's packages on Target the obvious thing to do would be something like 'apt-get install $list_of_apps' on Target, if you can get the list from Waldorf.
The apt-mark utility (options have changed between squeeze and wheezy) is quite useful.
'apt-mark showmanual' on Waldorf will now give you a list of all packages that were specifically installed on the system so

# on Waldorf
apt-mark showmanual > apps_to_install
# on Target
apt-get $( < apps_to_install )

is the first method I tried.

However, even after installing everything in apps_to_install, along with all the dependencies that apt brought in, there were still some auto-installed packages in Waldorf that weren't installed in Target. Running 'apt-get autoremove' on Waldorf took nothing out, so Waldorf obviously considered those extra packages to be dependencies of the manually-installed ones. Still, apt on Target hadn't seen any need to install them with the same "manual" list.

The only thing I could think of was that the CrunchBang APT settings include NOT regarding recommends as dependencies. Could it be that those extra auto packages had been brought in at some point because they were recommends, and even after setting no-install-recommends they don't get marked as automatically removable?

Some of those extra auto packages are important, so a way of getting them on Target was needed. neutral

To clone a system's packages, around the web and in 'man dpkg' the method you see referred to is to do

# on source box
dpkg --get-selections > selections
# on target box
dpkg --set-selections < selections
apt-get dselect-upgrade

Hardly anyone mentions that --set-selections won't work unless you also have dselect installed, but that's what I found on Wheezy. (Now I've just tried those same commands on Statler with no need for dselect so it looks as if there might have been some changes around here.)
Anyway, another difference I've just discovered with 'dpkg --get-selections' compared with 'apt-mark showmanual' is that dpkg outputs the architecture-specific package name, ie packagename:i386 instead of plain packagename. This makes that list less portable, obviously.

All this feels too messy, so now I'm thinking of going back to plain apt-get with the output of apt-mark. Something like:
'apt-mark showmanual > mark-manual' on Waldorf.
'apt-get install $( < mark-manual ) on the Target.
Comparing the output of 'apt-mark showauto' on the two systems (using eg comm) ought to give a list of packages missing from Target which can be installed with apt-get and marked as auto with apt-mark. Does that seem to make sense?

If it still feels right in a day or two maybe I'll amend the script (again).

edit Cleaned it up a bit. Should avoid latenight/drunk posting. roll

Last edited by johnraff (2012-08-23 04:55:12)

johnraff · 2012-09-03 16:10:07

barebones Crunchbang imitation

This is a very basic install on Debian Wheezy netinstall, just putting in a text editor (leafpad), terminal emulator (urxvt), file manager (thunar), image viewer (viewnior) and web browser (midori), and a bit of background stuff. No display manager, just cli login and auto-startx, 'sudo halt' to shutdown. No pipe-menus, except for places & recent files. ( tongue ) However, it looks just like CrunchBang, even the compositing. On a virtualbox allocated 380MB, it boots up with ~40MB of RAM.

Made it to play with the new package install method and develop the script to be more generic without having to wait 2 hours for all the packages to install each time. Also, it demonstrates how the same script can fairly easily be customized to do almost anything with a Wheexy base, editing the files that come in the package, but leaving the script itself alone. I'll post a Customization Guide in due course...

Have fun!

wget http://dl.dropbox.com/u/10808732/cb-barebones-netinstall.tar.gz
tar -xpf cb-barebones-netinstall.tar.gz
cd cb-barebones-netinstall
./install

...and please report any problems - thank you! smile

Last edited by johnraff (2012-09-23 15:48:37)

CrunchBang Linux

SEARCH

#1 2012-08-04 05:34:02

A Waldorf netinstall script

#2 2012-08-04 09:12:32

Re: A Waldorf netinstall script

#3 2012-08-07 16:08:29

Re: A Waldorf netinstall script

#4 2012-08-07 16:20:42

Re: A Waldorf netinstall script

#5 2012-08-08 04:50:41

Re: A Waldorf netinstall script

#6 2012-08-08 10:28:56

Re: A Waldorf netinstall script

#7 2012-08-08 11:35:21

Re: A Waldorf netinstall script

#8 2012-08-08 11:57:44

Re: A Waldorf netinstall script

#9 2012-08-09 01:30:08

Re: A Waldorf netinstall script

#10 2012-08-09 06:02:23

Re: A Waldorf netinstall script

#11 2012-08-09 12:48:32

Re: A Waldorf netinstall script

#12 2012-08-09 18:17:43

Re: A Waldorf netinstall script

#13 2012-08-09 19:46:53

Re: A Waldorf netinstall script

#14 2012-08-17 00:20:32

Re: A Waldorf netinstall script

#15 2012-08-17 04:33:02

Re: A Waldorf netinstall script

#16 2012-08-18 17:39:39

Re: A Waldorf netinstall script

#17 2012-08-20 22:45:29

Re: A Waldorf netinstall script

#18 2012-08-21 04:34:54

Re: A Waldorf netinstall script

#19 2012-08-21 07:46:00

Re: A Waldorf netinstall script

#20 2012-08-21 18:53:40

Re: A Waldorf netinstall script

#21 2012-08-22 03:44:57

Re: A Waldorf netinstall script

#22 2012-08-22 03:55:11

Re: A Waldorf netinstall script

#23 2012-08-22 04:17:00

Re: A Waldorf netinstall script

#24 2012-08-22 17:21:55

Re: A Waldorf netinstall script

#25 2012-09-03 16:10:07

Re: A Waldorf netinstall script

Board footer