Keep It With Mine » bruno

Continue SVN history in a git repository

bruno — Mon, 02 Sep 2013 19:00:26 +0000

The following is a basic command to continue (part of) a Subversion repository as a git repository.

git svn clone -A authors --no-metadata --localtime $URL

The file authors can be initially empty. The process will be interrupted when an unknown author is encountered. The format is:

username = Firstname Lastname

Alternatively, if you want to recover the author information from Unix user accounts:

git filter-branch --env-filter ". ~/filter.sh" HEAD

Where filter.sh is something like:

mail="$(cat /etc/mailname)"
 
user="$GIT_AUTHOR_NAME"
full="$(getent passwd "$user" | cut -d ':' -f 5 | cut -d ',' -f 1)"
if [ -n "$full" ]; then
        GIT_AUTHOR_NAME="$full"
        GIT_AUTHOR_EMAIL="$user@$mail"
fi
 
user="$GIT_COMMITTER_NAME"
full="$(getent passwd "$user" | cut -d ':' -f 5 | cut -d ',' -f 1)"
if [ -n "$full" ]; then
        GIT_COMMITTER_NAME="$full"
        GIT_COMMITTER_EMAIL="$user@$mail"
fi

Abridged git repository history

bruno — Fri, 15 Mar 2013 11:56:27 +0000

It is sometimes useful to have a repository for a project that shows only the main points in history. It is possible to start from git repository with full history and create such an abridged history in a separate branch. You can do this efficiently in git by reusing the original tree objects. This post will do a walkthrough for the Mediawiki repository.

We start with a full repository:

git clone git://github.com/wikimedia/mediawiki-core.git
cd mediawiki-core
git config user.name "Bruno De Fraine"
git config user.email "..."

This is a bash function for the main import logic:

function import_mw {
        tag=$1
        tree=
        adate=
        while read key val; do
                case $key in
                        tree) tree=$val ;;
                        author) adate="${val#*> }" ;;
                        "") break ;;
                esac
        done < <(git cat-file commit $tag 2>/dev/null)
        if [ -z "$tree" -o -z "$adate" ]; then
                echo "Could not read tag commit info" 1>&2
                return 1
        fi
        head=$(git symbolic-ref HEAD)
        if parent=$(git rev-parse --verify $head 2>/dev/null); then
                arg="-p $parent"
        else
                arg=
        fi
        if commit=$(GIT_AUTHOR_NAME="Wikimedia Foundation" \
           GIT_AUTHOR_EMAIL="" \
           GIT_AUTHOR_DATE="$adate" \
           git commit-tree $tree $arg <<<"MediaWiki $tag"); then
                git update-ref $head $commit
                echo git tag $tag $commit >> ~/mw-tags
                echo $commit
        else
                echo "Commit failed" 1>&2
                return 1
        fi
}

We start a completely independent branch for abridged history and create commit objects for relevant trees in the history:

git symbolic-ref HEAD refs/heads/ABR1_17
# Note: head ABR1_17 does not exist yet
import_mw 1.17.0
for tag in 1.17.{1,2,3,4,5}; do import_mw $tag; done

To see the result so far:

git reset --hard
git log --format=fuller

We can fork another branch in our abridged history for a different major version:

# Our commit object for 1.17.0 is 35d4f10
git update-ref refs/heads/ABR1_18 35d4f1055b9254b3b3d915ec782d0c16230516e5
git symbolic-ref HEAD refs/heads/ABR1_18
for tag in 1.18.{0,1,2,3,4,5,6,7}; do import_mw $tag; done
git reset --hard
git hist ABR1_17 ABR1_18

And so on for branches ABR1_19 and ABR1_20.

We can then export the abridged repository history to a separate repository:

git remote add abr ../mediawiki-core-abridged 
git push abr ABR1_17
git push abr ABR1_18
git push abr ABR1_19
git push abr ABR1_20
cd ../mediawiki-core-abridged
source ~/mw-tags
git gc

Handling merge conflicts with git

bruno — Wed, 23 Jan 2013 15:31:09 +0000

Like a similar post for Mercurial, this tip tries to document a list of systematic steps to handle merge conflicts with git.

List the conflict status (first column = ours, second column = theirs):
git status -s
Or more detailed (stage 1 = ancestor, stage 2 = ours, stage 3 = theirs):
git ls-files -u
In case of a git-merge, you can check the history of both heads:
git log --graph --left-right --boundary HEAD...MERGE_HEAD
Or, to list only the commits that contribute to conflicts:
git log --left-right --merge
In case of git-rebase, you can investigate the history and the commit that is being applied:
git log git show $(cat .git/rebase-apply/original-commit)
Resolve the conflict:
1. Manual resolution and mark as resolved:
  vim <file> git diff -c <file> git add <file>
2. Resolution using a tool:
  git mergetool <file>
3. Resolve by selecting local/other version:
  git checkout --ours <file> git checkout --theirs <file> git diff -c <file> git add <file>
(If the resolution went wrong) You can still redo the merge if it is not yet staged:
git merge-index git-merge-one-file <file>
(If you decide the merge was a bad idea, or want to start over) Abort the merge:
git merge --abort
Or the rebase:
git rebase --abort

Bash Arrays

bruno — Thu, 10 Jan 2013 09:23:21 +0000

Arrays are the main advanced data structure in Bash, but the syntax can be hard to remember. The following list summarizes the main operations.

Assigning an entire array:

nodes=(a b c d)
nodes=(foo*)

Copying an entire array:

nodes=("$@")
nodes=("${values[@]}")

Getting individual elements (zero-based indexing) or the element count:

echo ${nodes[i]}
echo ${#nodes[@]}

Iterating an array:

for node in "${nodes[@]}"; do ... done
for ((i=0; i<${#nodes[@]}; i++)); do node=${nodes[i]}; ... done

Appending to an array:

nodes[${#nodes[@]}]=foo

If you need the last form regularly, it can be simplified with the following shorthand:

function push_back { eval "$1[\${#$1[@]}]=\"\$2\""; }
 
push_back nodes foo

Makefile auto dependencies with GCC

bruno — Fri, 12 Oct 2012 11:53:06 +0000

There are many ways of having automatic dependencies in a Makefile. The following, however, is the most clean and complete solution that I know of for modern compilers.

GNU Make has a number of builtin rules and variables for building programs, for example, for C:

COMPILE.c = $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c
LINK.o = $(CC) $(LDFLAGS) $(TARGET_ARCH)
OUTPUT_OPTION = -o $@
 
%.o: %.c
	$(COMPILE.c) $(OUTPUT_OPTION) $<
 
%: %.o
	$(LINK.o) $^ $(LOADLIBES) $(LDLIBS) -o $@

Because of these builtins, a basic Makefile for a C project with gcc can be quite simple:

CC=gcc
LDLIBS=-lm
 
SRC=$(wildcard *.c)
OBJ=$(SRC:.c=.o)
 
.PHONY: clean
 
foo: foo.o bar.o
 
clean:
        rm -f $(OBJ) foo

This will not track dependencies on headers and other include files, so this will not properly rebuild when these files are updated. With not too much modification, we can ask gcc to output a dependency file next to each object file that is produced. We ask make to include these dependency files where available:

CC=gcc
OUTPUT_OPTION=-MMD -MP -o $@
LDLIBS=-lm
 
SRC=$(wildcard *.c)
OBJ=$(SRC:.c=.o)
DEP=$(SRC:.c=.d)
 
.PHONY: clean
 
foo: foo.o bar.o
 
-include $(DEP)
 
clean:
        rm -f $(OBJ) $(DEP) foo

The options will generate dependency files alongside compilation (-MMD instead of -MM), and with phony targets (-MP) for dependencies. The rationale for doing this is described in Autodependencies with GNU make, although this article dates from a time when apparently only -MM was available. The options -MMD -MP are a modern way of implementing what is proposed in the article.

Besides GCC, also Clang 3.0 and ICC 12.1 seem to support these options.

A Makefile for LaTeX documents

bruno — Thu, 04 Oct 2012 07:53:52 +0000

This is a Makefile that I’ve perfected over the years for rendering typical LaTeX articles or presentations (e.g. for scientific papers). It handles almost all LaTeX documents, except for large documents that use the \include feature (this is for another tip). It assumes your document has a bibtex bibliography.

The Makefile works out of the box, you only have to configure the jobname (=name of the main .tex-file). Most common prerequisites are automatically detected:

LATEXFLAGS?=-interaction=nonstopmode -file-line-error
PDFLATEX=pdflatex
BIBTEX=bibtex
 
JOB=paper
TEXS=$(wildcard *.tex) $(wildcard *.sty) $(wildcard *.cls)
PICS=$(wildcard *.png) $(filter-out $(JOB).pdf,$(wildcard *.pdf)) $(wildcard *.jpg)
BIBS=$(wildcard *.bib) $(wildcard *.bst)
 
.DEFAULT: all
.PHONY: all clean
 
all: $(JOB).pdf
 
$(JOB).aux: | $(TEXS) $(PICS)
        $(PDFLATEX) $(LATEXFLAGS) $(JOB)
 
$(JOB).bbl: $(JOB).aux $(BIBS)
        $(BIBTEX) $(JOB)
 
$(JOB).pdf: $(TEXS) $(PICS) $(JOB).aux $(JOB).bbl
        @cp -p $(JOB).aux $(JOB).aux.bak
        $(PDFLATEX) $(LATEXFLAGS) $(JOB)
        @if cmp -s $(JOB).aux $(JOB).aux.bak; \
        then touch -r $(JOB).aux.bak $(JOB).aux; \
        else NEWS="$$NEWS -W $(JOB).aux"; fi; rm $(JOB).aux.bak; \
        if [ -n "$$NEWS" ]; then $(MAKE) $$NEWS $@; fi
 
clean:
        rm -f $(JOB).aux $(JOB).log $(JOB).blg $(JOB).bbl $(JOB).out $(JOB).pdf

LaTeX documents have a circular dependency: a latex run produces a .aux-file, which is read as the input for a following run. You have to keep running latex until this process reaches a fixpoint. This is handled as follows:

An order-only prerequisite is used to produce a .aux-file if none exists.
A backup copy of the .aux-file is preserved.
latex is ran to produce a new .aux-file.
If the .aux-file differs from the backup copy, make is restarted to go back to step #1.

The commands take special care to ensure that make considers the .aux-file as new if (and only if) it has changed.

H.264/AAC encoding using FFmpeg

bruno — Tue, 11 Sep 2012 09:49:12 +0000

The x264 encoding guide gives a nice explanation of the different options for H.264 encoding. Normally:

ffmpeg -i in.avi -strict experimental -crf 23 -preset medium out.m4v

Where you can deviate from the above defaults:

-crf [18-28]: perceived output quality (lower crf is better quality)
-preset [ultrafast,superfast,veryfast,faster,fast,medium,slow,slower,veryslow]: compression effort (slower preset is better compression)

Joining H.264/AAC video files

bruno — Tue, 11 Sep 2012 09:19:37 +0000

It is possible to join H.264/AAC video files without re-enconding, if you convert them to an MPEG transport stream (TS). This format supports concatenation. Of course, the video files need to have the same height and width, aspect ratio,…

Using ffmpeg:

ffmpeg -i 01.m4v -c copy -bsf:v h264_mp4toannexb 01.ts
ffmpeg -i 02.m4v -c copy -bsf:v h264_mp4toannexb 02.ts
ffmpeg -i concat:"01.ts|02.ts" -c copy -bsf:a aac_adtstoasc join.m4v
rm 01.ts 02.ts

The bitstream filters take care of container requirements.

Decoding UTF-8 streams

bruno — Thu, 06 Sep 2012 15:10:43 +0000

UTF-8 is one of the most commonly used methods to encode unicode characters into byte values. It has some interesting properties, for example, characters from the ASCII codeset retain their encoding. This is an implementation of UTF-8 decoding in OCaml.

The interface of the decoder is:

val utf8_decode : char Stream.t -> int Stream.t

The decoding works lazily on streams, i.e. the resulting stream will decode characters as they are requested. In case of illegal input, the code will raise stream errors.

type byte = Single of int | Cont of int | Leading of int * int | Invalid
 
(** [classify chr] returns the {!byte} corresponding to [chr] *)
let classify chr =
    let c = int_of_char chr in
    (* Classify byte according to leftmost 0 bit *)
    if c land 0x80 = 0 then Single c
    else if c land 0x40 = 0 then Cont (c land 0x3f)
    else if c land 0x20 = 0 then Leading (1, c land 0x1f)
    else if c land 0x10 = 0 then Leading (2, c land 0x0f)
    else if c land 0x08 = 0 then Leading (3, c land 0x07)
    else if c land 0x04 = 0 then Leading (4, c land 0x03)
    else if c land 0x02 = 0 then Leading (5, c land 0x01)
    else Invalid
 
(** [follow strm n c] returns the code point based on [c] plus [n] continuation
    bytes taken from [strm] *)
let rec follow strm n c =
    if n = 0 then c
    else
        (match classify (Stream.next strm) with
        | Cont cc -> follow strm (n-1) ((c lsl 6) lor (cc land 0x3f))
        | _ -> raise (Stream.Error "Continuation byte expected"))
 
(** [utf8_decode strm] returns a code point stream that will lazily decode
    the byte stream [strm] *)
let rec utf8_decode strm =
    Stream.slazy (fun () ->
        match Stream.peek strm with
        | Some chr ->
            Stream.junk strm;
            (match classify chr with
            | Single c -> Stream.icons c (utf8_decode strm)
            | Cont _ -> raise (Stream.Error "Unexpected continuation byte")
            | Leading (n, c) ->
                let c = follow strm n c in
                Stream.icons c (utf8_decode strm)
            | Invalid -> raise (Stream.Error "Invalid byte"))
        | None -> Stream.sempty)

Posted at the occassion of 20 years of UTF-8.

Using wget to mirror entire website folder

bruno — Fri, 31 Aug 2012 07:08:34 +0000

GNU wget can create a very usable local or offline mirror of an entire folder of a website using the following combination of options.

From the output directory, use this invocation:

URL=http://www.emsl.pnl.gov/docs/global/index.shtml
wget -r -l inf -k -N -np -p -E $URL

You can play around with the options:

-r: recursive
-l inf: follow link infinitely
-k: convert links after downloading
-N: time-stamping
-np: no parent
-p: retrieve prerequisites of pages
-E: page extensions according to MIME type

The files will be stored under hostname/path/to/file. If you know up front that the origin of the files is restricted to a particular host or even subpath, you can additionally use:

-nH: no subdir for host
–cut-dirs=N: strip N path components