Keep It With Mine » Snippets

Abridged git repository history

bruno — Fri, 15 Mar 2013 11:56:27 +0000

It is sometimes useful to have a repository for a project that shows only the main points in history. It is possible to start from git repository with full history and create such an abridged history in a separate branch. You can do this efficiently in git by reusing the original tree objects. This post will do a walkthrough for the Mediawiki repository.

We start with a full repository:

git clone git://github.com/wikimedia/mediawiki-core.git
cd mediawiki-core
git config user.name "Bruno De Fraine"
git config user.email "..."

This is a bash function for the main import logic:

function import_mw {
        tag=$1
        tree=
        adate=
        while read key val; do
                case $key in
                        tree) tree=$val ;;
                        author) adate="${val#*> }" ;;
                        "") break ;;
                esac
        done < <(git cat-file commit $tag 2>/dev/null)
        if [ -z "$tree" -o -z "$adate" ]; then
                echo "Could not read tag commit info" 1>&2
                return 1
        fi
        head=$(git symbolic-ref HEAD)
        if parent=$(git rev-parse --verify $head 2>/dev/null); then
                arg="-p $parent"
        else
                arg=
        fi
        if commit=$(GIT_AUTHOR_NAME="Wikimedia Foundation" \
           GIT_AUTHOR_EMAIL="" \
           GIT_AUTHOR_DATE="$adate" \
           git commit-tree $tree $arg <<<"MediaWiki $tag"); then
                git update-ref $head $commit
                echo git tag $tag $commit >> ~/mw-tags
                echo $commit
        else
                echo "Commit failed" 1>&2
                return 1
        fi
}

We start a completely independent branch for abridged history and create commit objects for relevant trees in the history:

git symbolic-ref HEAD refs/heads/ABR1_17
# Note: head ABR1_17 does not exist yet
import_mw 1.17.0
for tag in 1.17.{1,2,3,4,5}; do import_mw $tag; done

To see the result so far:

git reset --hard
git log --format=fuller

We can fork another branch in our abridged history for a different major version:

# Our commit object for 1.17.0 is 35d4f10
git update-ref refs/heads/ABR1_18 35d4f1055b9254b3b3d915ec782d0c16230516e5
git symbolic-ref HEAD refs/heads/ABR1_18
for tag in 1.18.{0,1,2,3,4,5,6,7}; do import_mw $tag; done
git reset --hard
git hist ABR1_17 ABR1_18

And so on for branches ABR1_19 and ABR1_20.

We can then export the abridged repository history to a separate repository:

git remote add abr ../mediawiki-core-abridged 
git push abr ABR1_17
git push abr ABR1_18
git push abr ABR1_19
git push abr ABR1_20
cd ../mediawiki-core-abridged
source ~/mw-tags
git gc

A Makefile for LaTeX documents

bruno — Thu, 04 Oct 2012 07:53:52 +0000

This is a Makefile that I’ve perfected over the years for rendering typical LaTeX articles or presentations (e.g. for scientific papers). It handles almost all LaTeX documents, except for large documents that use the \include feature (this is for another tip). It assumes your document has a bibtex bibliography.

The Makefile works out of the box, you only have to configure the jobname (=name of the main .tex-file). Most common prerequisites are automatically detected:

LATEXFLAGS?=-interaction=nonstopmode -file-line-error
PDFLATEX=pdflatex
BIBTEX=bibtex
 
JOB=paper
TEXS=$(wildcard *.tex) $(wildcard *.sty) $(wildcard *.cls)
PICS=$(wildcard *.png) $(filter-out $(JOB).pdf,$(wildcard *.pdf)) $(wildcard *.jpg)
BIBS=$(wildcard *.bib) $(wildcard *.bst)
 
.DEFAULT: all
.PHONY: all clean
 
all: $(JOB).pdf
 
$(JOB).aux: | $(TEXS) $(PICS)
        $(PDFLATEX) $(LATEXFLAGS) $(JOB)
 
$(JOB).bbl: $(JOB).aux $(BIBS)
        $(BIBTEX) $(JOB)
 
$(JOB).pdf: $(TEXS) $(PICS) $(JOB).aux $(JOB).bbl
        @cp -p $(JOB).aux $(JOB).aux.bak
        $(PDFLATEX) $(LATEXFLAGS) $(JOB)
        @if cmp -s $(JOB).aux $(JOB).aux.bak; \
        then touch -r $(JOB).aux.bak $(JOB).aux; \
        else NEWS="$$NEWS -W $(JOB).aux"; fi; rm $(JOB).aux.bak; \
        if [ -n "$$NEWS" ]; then $(MAKE) $$NEWS $@; fi
 
clean:
        rm -f $(JOB).aux $(JOB).log $(JOB).blg $(JOB).bbl $(JOB).out $(JOB).pdf

LaTeX documents have a circular dependency: a latex run produces a .aux-file, which is read as the input for a following run. You have to keep running latex until this process reaches a fixpoint. This is handled as follows:

An order-only prerequisite is used to produce a .aux-file if none exists.
A backup copy of the .aux-file is preserved.
latex is ran to produce a new .aux-file.
If the .aux-file differs from the backup copy, make is restarted to go back to step #1.

The commands take special care to ensure that make considers the .aux-file as new if (and only if) it has changed.

Decoding UTF-8 streams

bruno — Thu, 06 Sep 2012 15:10:43 +0000

UTF-8 is one of the most commonly used methods to encode unicode characters into byte values. It has some interesting properties, for example, characters from the ASCII codeset retain their encoding. This is an implementation of UTF-8 decoding in OCaml.

The interface of the decoder is:

val utf8_decode : char Stream.t -> int Stream.t

The decoding works lazily on streams, i.e. the resulting stream will decode characters as they are requested. In case of illegal input, the code will raise stream errors.

type byte = Single of int | Cont of int | Leading of int * int | Invalid
 
(** [classify chr] returns the {!byte} corresponding to [chr] *)
let classify chr =
    let c = int_of_char chr in
    (* Classify byte according to leftmost 0 bit *)
    if c land 0x80 = 0 then Single c
    else if c land 0x40 = 0 then Cont (c land 0x3f)
    else if c land 0x20 = 0 then Leading (1, c land 0x1f)
    else if c land 0x10 = 0 then Leading (2, c land 0x0f)
    else if c land 0x08 = 0 then Leading (3, c land 0x07)
    else if c land 0x04 = 0 then Leading (4, c land 0x03)
    else if c land 0x02 = 0 then Leading (5, c land 0x01)
    else Invalid
 
(** [follow strm n c] returns the code point based on [c] plus [n] continuation
    bytes taken from [strm] *)
let rec follow strm n c =
    if n = 0 then c
    else
        (match classify (Stream.next strm) with
        | Cont cc -> follow strm (n-1) ((c lsl 6) lor (cc land 0x3f))
        | _ -> raise (Stream.Error "Continuation byte expected"))
 
(** [utf8_decode strm] returns a code point stream that will lazily decode
    the byte stream [strm] *)
let rec utf8_decode strm =
    Stream.slazy (fun () ->
        match Stream.peek strm with
        | Some chr ->
            Stream.junk strm;
            (match classify chr with
            | Single c -> Stream.icons c (utf8_decode strm)
            | Cont _ -> raise (Stream.Error "Unexpected continuation byte")
            | Leading (n, c) ->
                let c = follow strm n c in
                Stream.icons c (utf8_decode strm)
            | Invalid -> raise (Stream.Error "Invalid byte"))
        | None -> Stream.sempty)

Posted at the occassion of 20 years of UTF-8.