Unix reminder

All those little things, that as a unix newbie, i keep forgetting.
awk | bash | bc | convert | cut | debian apt-get | emacs | exim | gentoo | git | iptables | make | mutt | ps | sabayon | sed | tail | tcsh

Awk

Lines of an apache log file look like: 
127.0.0.1 - - [24/May/2004:05:06:01 +0200] "GET /keepalive.html HTTP/1.0"
In order to sort IP addresses of the log files by frequency type: 
cut -d ' ' -f 1 < /var/log/apache/access.log | awk -f count.a | sort -n
Where count.a is the following script: 
{ counter[$1] += 1 }
END { for (u in counter)
        printf "%d %s\n", counter[u], u
}

Bash

Setting the prompt

To get a prompt like user@host /path/to/current/dir $ which displays current path as the title of the xterm window add to your bashrc: 
PS1='\[\033]0;\w\007\033[32m\]\u@\h \[\033[33m\w\033[0m\] $'
Sequence \[\033]0;\w displays current path (\w) as xterm title.
Sequence \033[32m\]\u@\h displays user@host in green.
Sequence \[\033[33m\w\033[0m\] display path (\w) and return to default color.
Example from gentoo: 
# for root
export PS1='\[\033[01;31m\]\h\[\033[01;34m\] \W \$\[\033[00m\] '
# for other users
export PS1='\[\033[01;32m\]\u@\h\[\033[01;34m\] \w \$\[\033[00m\]'
PROMPT_COMMAND='echo -ne "\033]0;${USER}@${HOSTNAME%%.*}:${PWD/$HOME/~}\007"'

Redirections

Redirect standard out and standard error separately:
$ cmd > stdout-redirect 2> stderr-redirect
Redirect standard error and out together:
$ cmd > stdout-redirect 2>&1
Merge standard error with standard out and pipe:
$ cmd 2>&1 | cmd2

'If' tests

Here is how to do something if a file my_file.txt contains pattern pattern: 
if [[ $(grep pattern my_file.txt | wc -l) -gt 0 ]]
then echo "Stuff found"
else echo "Stuff not found"
fi

'For' loops

Here is a complicated way to do ls: 
for i in `ls *~`; do echo $i; done
Which is equivalent to 
for i in `ls *~`
do
  echo $i
done

'While' loops

Here is how to display numbers from 0 to 10: 
i=0; while [ $i -le 10 ] ; do echo $i ; i=$((i+1)) ; done
Here is another construct to increment i: 
i=0; while [ $i -le 10 ] ; do echo $i; let "i += 1"; done
Do do infinite loop use the empty command ':' which always returns true: 
while : ; do sleep 2 ; printf '.' ; done

'Case' tests

    case $n in 
    0 )
      echo n is 0
      ;;
    1 )
      echo n is 1
      ;;
    * )
      echo $n is not 0 or 1
      ;;
    esac

Random number

Here is how to set variable n to a random value between 0 and 10: 
RANGE=10
n=$RANDOM
let "n %= $RANGE"

Functions

Here is how to do functions: 
function my_fun {
  w=$1
  h=$2
  echo this is the body of function my_fun
  echo with arguments w=$w and h=$h  
}
my_fun 123 345
my_fun 90 30
interesting link

bc

Here is how to define functions in bc: 
define fact (x) {
  if (x <= 1) return (1);
  return (fact(x-1) * x);
}
define comb(n,p) {
  return ((fact(n)/fact(p))/fact(n-p));
}
If bc is invoked with the -l option, a math library is preloaded. The loaded functions are: s (sinus), c (cosinus), l (natural logarithm), a (arctangent), e (exponentiation). For instance: 
l(2)
.69314718055994530941
4*a(1)
3.14159265358979323844

Convert

To put side by side two images : 
convert img1.jpg img2.jpg +append out.jpg
To put one image above the other: 
convert img1.jpg img2.jpg -append out.jpg
To change colors use the -recolor option with the 3x3=9 parameters of the matrix 
# to green :
convert in.jpg -recolor "0 0 0 0 1 0 0 0 0" out.jpg # out.jpg is in.jpg with green channel only
# to blue
convert in.jpg -recolor "1 0 0 0 0 0 0 0 0" out.jpg 
# to red
convert in.jpg -recolor "0 0 0 0 0 0 0 0 1" out.jpg
# with the identity matrix nothing changes
convert in.jpg -recolor "1 0 0 0 1 0 0 0 1" out.jpg #out.jpg is in.jpg
# lighten the image :
convert in.jpg  -recolor "1 .7 .7  .7  1 .7   .7 .7 1" out.jpg

Cut

Fields

Lines of an apache log file look like: 
127.0.0.1 - - [24/May/2004:05:06:01 +0200] "GET /keepalive.html HTTP/1.0"
In order to list IP addresses of the log file: 
cut -d ' ' -f 1 < /var/log/apache/access.log
Option -d ' ' specifies that space is the delimiter. Option -f 1 means 'return first field'.

Characters

To get only columns between column i and j type cut -ci-j. For instance on get only process number type: 
ps | tail +2 |  cut -c0-5
The tail +2 is only here to remove first line of the output of the ps command.

debian apt-get

To get the new key for apt-get type as root: 
wget http://ftp-master.debian.org/ziyi_key_2006.asc -O - | apt-key add -

Emacs

Here is how to re-open a scratch buffer: 
C-x b *scratch* RET
Here are my default Emacs settings: 
;; shortcuts i like
(global-set-key [f1] 'font-lock-fontify-buffer)  ;; put colors
(global-set-key [f3] 'call-last-kbd-macro)       ;; recall last macro
(global-set-key [f9] 'compile)                   ;; run makefile

;; do not make any sound
(setq visible-bell 1)

;; open .h file in C++ mode
(setq auto-mode-alist (cons '("\\.h$" . c++-mode) auto-mode-alist))

exim

Configuration for exim : at the top of the config file: 
primary_hostname = derepas.com
domainlist local_domains = @:derepas.com
domainlist relay_to_domains =
hostlist   relay_from_hosts = 127.0.0.1
To use the smtp server of my isp, add at the top of the router section: 
 send_to_gateway: 
        driver = manualroute 
        domains = !+local_domains 
        transport = remote_smtp 
        route_list = * smtp.server.of.my.isp.com

Gentoo

Emerge is Gentoo command line tool to access the package system. To update the entire system: 
emerge --sync
emerge -av --newuse --update --deep world
To mask package dev-db/mysql-community: 
echo 'dev-db/mysql-community' >> /etc/portage/package.mask
To check contents of package nstx: 
cat /var/db/pkg/net-misc/nstx-1.1_beta6-r2/CONTENTS
qfile in the portage-utils package finds the package to which a file belongs: 
$ qfile /etc/fonts/fonts.conf
media-libs/fontconfig (/etc/fonts/fonts.conf)
Security : glsa-check is part of app-portage/gentoolkit. Check if your system is affected by GLSAs (Gentoo Linux Security Advisories): 
glsa-check -t all
To apply required fixes 
glsa-check -f $(glsa-check -t all)

Git

Git is an easy to use, distributed, version control system. To create a new repository 
cd (project-directory)
git init
(add some files)
git add .
git commit -m 'Initial commit'
To remove a file 
git rm path/to/file
To perform a "checkout" and submit a patch: 
git clone git://github.com/git/hello-world.git
cd hello-world
(edit files)
git add (files)
git commit -m 'Explain what I changed'
git format-patch origin/master
This generates the 0001-Explain-what-I-changed.patch file. To check out using ssh: 
git clone ssh://derepas.com/~git/snstx
Some other stuff: 
# list config 
git config --list
# change original repository
git config remote.origin.url ssh://mydomain.org/~git/my_repo
# configure user name and email
git config --global user.name "Your Name"
git config --global user.email you@example.com
# list untracked (other) files
git ls-files -o

Iptables

The iptables command is used to manipulated the way ip packets are handled. This is usefull to create a firewall. Here is how to create the firewall. First of all let us define some parameters: 
IPTABLES=/sbin/iptables
#private lan network
INTERNAL_LAN="192.168.0.0/24"
#private lan interface
INTERNAL_LAN_INTERFACE="eth1"
#private lan interface address for the firewall
INTERNAL_LAN_INTERFACE_ADDR="192.168.0.1"
#external lan interface
EXTERNAL_LAN_INTERFACE="eth0"
# external lan interface addr
EXTERNAL_LAN_INTERFACE_ADDR="xxx.xxx.xxx.xxx"
First of all we have to erase all possible rules: 
# flush all
$IPTABLES -F
$IPTABLES -t nat -F
And specify that by default all IP packets are dropped: 
# set default policy
$IPTABLES -P INPUT DROP
$IPTABLES -P OUTPUT DROP
$IPTABLES -P FORWARD DROP
We then decide to enable loop back as well as any ping request: 
# enable loop back
$IPTABLES -A INPUT -i lo -j ACCEPT
$IPTABLES -A OUTPUT -o lo -j ACCEPT

# enable ping requests
$IPTABLES -A INPUT -p icmp -j ACCEPT
$IPTABLES -A OUTPUT -p icmp -j ACCEPT
In order to have the privates machines on the network "appear" with the internet address of the firewall we have to enable masquerading: 
$IPTABLES -t nat -A POSTROUTING -o $EXTERNAL_LAN_INTERFACE \
          -s $INTERNAL_LAN \
          -j MASQUERADE
Ip forwaring should be enabled: 
echo 1 > /proc/sys/net/ipv4/ip_forward
We can then define the following bash function which enables local machines to access some port of machines on the internet: 
allow_lan_to_access() {
  # outbound traffic rules
  $IPTABLES -A OUTPUT -o $EXTERNAL_LAN_INTERFACE -p tcp \
          -s $EXTERNAL_LAN_INTERFACE_ADDR \
          -d 0/0 --destination-port $1 -j ACCEPT
  # inbound traffic rules
  $IPTABLES -A INPUT -i $EXTERNAL_LAN_INTERFACE -p tcp \
          -s 0/0 --source-port $1 \
          -d $EXTERNAL_LAN_INTERFACE_ADDR \
          -j ACCEPT
}

allow_lan_to_access_udp() {
  # outbound traffic rules
  $IPTABLES -A OUTPUT -o $EXTERNAL_LAN_INTERFACE -p udp \
          -s $EXTERNAL_LAN_INTERFACE_ADDR \
          -d 0/0 --destination-port $1 -j ACCEPT
  # inbound traffic rules
  $IPTABLES -A INPUT -i $EXTERNAL_LAN_INTERFACE -p udp \
          -s 0/0 --source-port $1 \
          -d $EXTERNAL_LAN_INTERFACE_ADDR \
          -j ACCEPT
}
These functions can be used the following way, for instance to allow local machines to surf on web pages, port 80 of any machine should be possible: 
allow_lan_to_access 80
If you want to perform DNS requests on DNS located on the internet, you need to allow UDP connections on port 53: 
allow_lan_to_access_udp 53
If some servers are running on the computer which is the firewall you then need to open some ports for the outside world. This is acheived via the following function: 
allow_outside_access_on_firewall()
{
  $IPTABLES -A INPUT -i $EXTERNAL_LAN_INTERFACE \
          -p tcp \
          -s 0/0 \
          -d $EXTERNAL_LAN_INTERFACE_ADDR --destination-port $1 \
          -j ACCEPT
  $IPTABLES -A OUTPUT -o $EXTERNAL_LAN_INTERFACE \
          -p tcp \
          -s $EXTERNAL_LAN_INTERFACE_ADDR --source-port $1 \
          -d  0/0  \
          -j ACCEPT
}
For instance if you have a mail server, you should allow access to port 25,this would be 80 for a web server, 22 for an ssh server, 20 for an sftp server: 
allow_outside_access_on_firewall 20
allow_outside_access_on_firewall 22
allow_outside_access_on_firewall 25
allow_outside_access_on_firewall 80
If you have a server running not on the firewall but on a computer in you local area, you need to have a port forward: 
port_forward() {
  $IPTABLES -t nat -A PREROUTING -d $EXTERNAL_LAN_INTERFACE_ADDR -p tcp \
    --dport $1 \
    -j DNAT \
    --to-dest $2
}

port_forward_udp() {
  $IPTABLES -t nat -A PREROUTING -d $EXTERNAL_LAN_INTERFACE_ADDR -p udp \
    --dport $1 \
    -j DNAT \
    --to-dest $2
}
For instance if you have a web server running on port 80 of 192.168.0.2, which can be accessed by port 8080 on the firewall you can perform the port forwarding using the instruction: 
port_forward 8080 192.168.0.2:80

Make

Make is used to automate compilation commands. Here is a typical makefile to compile all *.cc files in current directory into an executable named myexe: 
# definitions
CXX:=g++
CXXFLAGS:=-Wall -g
SOURCES := $(wildcard *.cc)      # all *.cc files in the directory
OBJ := $(SOURCES:.cc=.o)         # remplace all .cc by .o
EXE_NAME := myexe

# targets
$(EXE_NAME) : $(OBJ)
        $(CXX) -o $@ $(OBJ)
%.o : %.cc .deps
        $(CXX) $(CXXFLAGS) -c $<
.deps : $(SOURCES)
        $(CXX) $(CXXFLAGS) -MM $(SOURCES) > $@
-include .deps

# suppress generated files
clean :
        rm -f *.o $(EXE_NAME) *~ .dep
To have a "clean" target which calls make file in subdirectory other than .git 
clean :
        for i in `/bin/ls -F | grep '/' | grep -v .git` ; do \
                cd "$$i"; make clean; cd ..; \
        done

mutt

Here is my .muttrc file with the interface to spamassassin: 
 
set realname = "Fabrice Derepas"  
set from = "fabrice@derepas.com"  
# Set DEFAULT status color 
color status    blue      white 
 
#color any message with a spam score header in bright red
color index brightred default '~h "X-Spam_score_int:.*[0-9]+"' 
 
unset confirmappend

# Delete is re-bound to move messages to Trash. They will be filed as ham when pressing 'R'
macro index d "s=Trash\n" "move message to trash"
macro pager d "s=Trash\n" "move message to trash"
# send messages to spam folder. They will be procesesd as spam when pressin 'R'
macro index X "s=junkmail\n" "file as Spam"
macro pager X "s=junkmail\n" "file as Spam"
# process spam and ham
macro index R "!/home/fabrice/bin/processSpam"
The script /home/fabrice/bin/processSpam processes the Trash and junkmail folder: 
#!/bin/sh
TRASH=/home/fabrice/Mail/Trash
SPAMBOX=/home/fabrice/Mail/junkmail
[ -w $TRASH ] || exit 1
[ -w $SPAMBOX ] || exit 1

# process Trash as ham, delete if ok
if /usr/bin/sa-learn --mbox --ham $TRASH
then
  /bin/cat /dev/null > $TRASH
  echo "ham processed"
else
    echo "ham not processed";
fi

# process Spam, delete if ok
if /usr/bin/sa-learn --mbox --spam $SPAMBOX
then 
  /bin/cat /dev/null > $SPAMBOX 
  echo "spam processed" 
else 
    echo "spam not processed"; 
fi
/usr/bin/sa-learn --sync
Mark Stosberg has a nice mutt tutorial.

PS

To list processes with a longer list of arguments: 
ps -A o "pid command"

Sabayon

equo is the command line tool for package management in Sabayon. 
equo update
equo install entropy sulfur
equo cleanup
equo search <pkg-name>
equo remove <pkg-name>
equo upgrade --ask
Query commands 
equo query files <pkg-name>     # lists all files of a package
equo query belongs <file-name>  # search from what package a file belongs
Security related commands: 
equo security list --affected
equo security install --ask

Sed

Syntax is sed -e "<regular expression>". To convert resize all current png file in current directory with a width of 100 pixels simply type : 
for i in `/bin/ls *.png| grep -v small` ; do j=`echo $i | sed -e "s/.png/_small.png/"` ; convert $i -resize 100 $j; echo $j ; done

Tail

To output the last tree lines of a file named my_file.txt: 
tail -n 3 my_file.txt
To output all but the first three lines of a file named my_file.txt: 
tail -n +4 my_file.txt

Tcsh

Prompt

To get a prompt which looks like 12:37 /path/to/current/dir >, which displays current path as the title of the xterm window add to your .tcshrc: 
set prompt="%{\033]0;%~\007%}%{\033[32m%}%P%{\033[0m%} %B%~%b%#"
Sequence %{\033]0;%~\007%} displays current path (%~) as xterm title.
Sequence {\033[32m%}%P%{\033[0m%} display time (%P) in green, and goes back to normal color.
Sequence %B%~%b%# displays current path (%~) in bold and default prompt (%#).

'For' loops

Here is a complicated way of doing 'ls': 
>foreach i ( `ls` )
foreach? echo $i
foreach? end
Back to my home page Last-Updated: ven. nov. 4 13:58:32 2011 (+0100)