As a Gentoo user, you sometimes emerge packages which you don’t want to be in the world set, you just emerged them because you needed to fix or test something. (Of course, there’s the -1 flag, but you don’t always remember to use it.) With time, these packages become unneccessary, pull in useless dependencies and cause stupid conflicts. Therefore you should remove them from the @world set.

The @world set is stored in a plain text file located at /var/lib/portage/world. You should be aware that manipulating it may cause problems, therefore you’re advised to make a backup before editing. Also, you should do a full update of your system with emerge -uND world. Now you can open the world file in an editor of your choice, as root.

You may delete entries for packages you don’t need anymore, line by line. In order to actually remove those packages and their dependencies from your system, you must run depclean and revdep-rebuild:

emerge -a --depclean
revdep-rebuild -- -q --keep-going

After this, you will have a much leaner system.

Bonus hint: Also check your /etc/make.conf as well as your /etc/portage/package.use for no longer needed USE flags. These also tend to cause conflicts and stuff your system with needless dependencies. (Remember to run emerge -auND world; emerge -a --depclean; revdep-rebuild -- -q --keep-going afterwards.)

I think the following is a common setup these days: You have a webserver and host websites for a couple of friends. Your friends don’t need shell access, but they need FTP.

Now, vsftpd is the FTP server of choice for such a situation: It is light-weight, and as it relies on system features (namely PAM) for user authentication, it can be considered quite secure in regard to user management – as opposed to FTP servers that feature their own authentication layer. Vsftpd can, as most FTP servers, chroot users into their home directories, meaning they cannot access other parts of the server below their home directory.

In order to create a user account which can log in via FTP, but not SSH, you can use the following (as root):

useradd -d /path/to/website/document/root/ -s $(which nologin) username
passwd username

You can also switch the user’s shell later, if the account already exists:

usermod -s $(which nologin) username

The important thing is that the output of which nologin is a path to the nologin tool, which simply tells the authentication layer that it must not open an interactive shell, but issue an error and return a error code instead.

If you experience that a user gets an error from vsftpd if the shell is a normal interactive shell like /bin/bash, but they cannot log in if the shell is something like /usr/sbin/nologin, you must check if the correct path to the nologin tool is included in /etc/shells. If it’s not, you must add it here.

Now your users can enjoy the full comfort of an FTP server, and you don’t have to be afraid that their accounts can be misused to get shell access and snoop around on your server.

The YUI Compressor is a phenomenal tool to minimize (and obfuscate, by the way) CSS and JavaScript code.

Here’s a little bash script that will batch-process all JS and CSS files in your project, while preserving a source copy of that file.

#!/bin/bash
 
JAVA="$(which java)"
BUILDDIR="$(pwd)"
YUIJAR="/path/to/yuicompressor-x.x.x.jar"
 
 
echo "Compressing JS and CSS:"
 
for TYPE in js css
do
	for FILENAME in $(find "$BUILDDIR" -iname "*.$TYPE" | sed -e "s|\.$TYPE$||g")
	do
		echo -n "compressing $FILENAME.$TYPE ... "
 
		mv "$FILENAME.$TYPE" "$FILENAME.src.$TYPE"
		$JAVA -jar $YUIJAR  --charset utf-8 --type "$TYPE" "$FILENAME.src.$TYPE" > "$FILENAME.$TYPE"
 
		if [ "$?" == '0' ]; then
			echo "success."
		else
			cp "$FILENAME.src.$TYPE" "$FILENAME.$TYPE"
			echo "ERROR: failed, using original."
		fi
	done
done
 
echo "done."

Note: You will usually first copy your application source to some sort of build target, and then apply this script. Elsewise, you will have to revert the actions of that script when you continue editing your code.

How can you have concurrent “threads” in a Bash script, and at the same time have a flow control?

My problem was that I wanted to trigger database backups on remote servers via SSH. Not one after another, but at the same time.

Of course, “concurrency” itself is nothing special, you can background tasks and wait for all of them to finish.

# start two processes in the background and wait until the last one finishes
(sleep 4 ; echo 'done 1') & (sleep 2; echo 'done 2') & wait; echo "all done"

But there are a few problems:

• The script can’t do anything until both tasks are finished,
• You can not intervene if a task takes too long or goes crazy otherwise.

But this can be done within a shell script, completely without using wait:

#!/bin/bash
 
 
### config
 
# after $TIMEOUT seconds, the script will definitely abort
TIMEOUT=6
 
# check all $CYCLE seconds (increase, if checks are resource-greedy)
CYCLE=1
 
 
 
### program
 
ts()
{
	echo -n "$(date +%s)"
}
 
thread1()
{
	(sleep 4; echo "thread 1 finished") & PID_1=$!
}
 
thread2()
{
	(sleep 2; echo "thread 2 finished") & PID_2=$!
}
 
 
START="$(ts)"
 
thread1
thread2
 
while [ 1 ];
do
	PID_1_DONE="$(ps -p $PID_1 >/dev/null; echo $?)"
	PID_2_DONE="$(ps -p $PID_2 >/dev/null; echo $?)"
 
	[[ "$PID_1_DONE" != "0" && "$PID_2_DONE" != "0" ]] && \
		echo "everything finished successfully." && \
		exit 0
 
	NOW="$(ts)"
	DIFF="$[ $NOW - $START ]"
 
	echo $DIFF
 
	if [[ $DIFF -gt $TIMEOUT ]]; then
		echo "something went wrong, shutting down."
		kill $PID_1 >/dev/null 2>&1 || kill -9 $PID_1 >/dev/null 2>&1
		kill $PID_2 >/dev/null 2>&1 || kill -9 $PID_2 >/dev/null 2>&1
		exit 1
	fi
 
	sleep $CYCLE
done

What do we see here?

• Both threads are started from a dedicated function in a subshell, and return their process ID to the environment of the script.
• A while loop will run continuously, until either both subshell processes are finished, or until a given timeout is reached.

Of course, instead of a timeout, you can also define other break/exit conditions.

If you put your cash to a bank account, you will want to know how much money you get with a certain amount at a given rate within a given period. Save the following Bash script as /usr/local/bin/loancalc (or something like that), make it executable, and you will be able to calculate the return on your investments with a single one-liner.

#!/bin/bash
 
function calc()
{
	return $(echo "scale=$2; $1" | bc)
}
 
if [ "$1" == "-h" ] || [ "$1" == "--help" ]; then
	echo "$0 calculates the interest loan of a given amount with a given rate for a given duration."
	echo "Usage: $0 AMOUNT RATE DURATION"
	exit 0
fi
 
if [ -z "$3" ]; then
	echo "Please enter the amount, the rate and the duration (in this order)."
	exit 1
fi
 
AMOUNT=$1
ZINS=$2
DURATION=$3
 
echo "Investment: $AMOUNT bucks at a rate of $ZINS% for $DURATION years."
echo "          Loan            Total"
 
for i in $(seq 1 $DURATION); do
	INTERESTLOAN=$(echo "scale=10; ($AMOUNT/100)*$ZINS" | bc)
	AMOUNT=$(echo "scale=10; $AMOUNT+$INTERESTLOAN" | bc)
	printf "Year ${i} %8.2f " $INTERESTLOAN; printf " %14.2f\n" $AMOUNT;
done
 
printf "Final amount: %1.2f bucks.\n" $AMOUNT

For example:

you@yourmachine ~ $ loancalc 3000 3.5 5
Investment: 3000 bucks at a rate of 3.5% for 5 years.
          Loan            Total
Year 1   105.00         3105.00
Year 2   108.68         3213.68
Year 3   112.48         3326.15
Year 4   116.42         3442.57
Year 5   120.49         3563.06
Final amount: 3563.06 bucks.

xgettext is a great shell tool to create po files (i.e. Gettext translation templates). It is also able to merge existing translations with new strings. For example, to create a po template from all PHP files in a directory tree and preserve already done translations from the file my_translation-de_DE.po, enter:

mv my_translation-de_DE.po messages.po
find . -type f -iname "*.php" | xgettext --keyword=__ --keyword=_e -j -f -
mv messages.po my_translation-de_DE.po

Now you have a new my_translation-de_DE.po, with all the translated and the newly added strings appended to this file. (The --keywords parameter is very useful to specify your translation function(s), because if you do so, xgettext will not extract all the other strings, too.)

Unfortunately, xgettext is not able to expunge strings that are no longer existing. This behaviour is easy to explain: In many cases, you might be doing an incremental update of your translation template (i.e. you only add the strings for a subset of your files). If xgettext would expunge the translations for which no sting wasn’t found in your code, you would always have to recreate the po template completely. Anyway, a little option to expunge translations that weren’t found in the original, would be nice.

A little trick will at least help to determine the strings that do no longer exist: Open the po file to be merged and delete all #: lines (the regex ^#:.*$ will match them). Then merge the files as usual. In the new file, all messages without a #: marker do no longer exist. You can even modify the above code to do so automatically:

grep -v "^#:" my_translation-de_DE.po > messages.po
find . -type f -iname "*.php" | xgettext --keyword=__ --keyword=_e -j -f -
mv messages.po my_translation-de_DE.po

This way, you will preserve your translations, but your references are recreated, and you can easily see which strings do not exist anymore.

Update, 16.04.08: When I wrote this post, I didn’t know about the msgmerge tool. Using it is much better than the grep foo above. Use it as follows:

echo '' > messages.po # xgettext needs that file, and we need it empty
find . -type f -iname "*.php" | xgettext --keyword=__ --keyword=_e -j -f -
msgmerge -N existing.po messages.po > new.po
mv new.po existing.po
rm messages.po

It looks a bit more complex, but it’s the best generic solution I can provide. Note the -N parameter for msgmerge: It suppresses the fuzzy mode, which yields more confusion and extra work than anything else.