Tuesday, October 15, 2013

Vim: Distinguish a location-list from a quickfix-list

Strange that there is no easy way (that I know of) to differentiate a location-list from a quickfix-list. Today I noticed that the buffer-list (:ls) displays a different name for each type, and thus:

fun! GetBufferListOutputAsOneString()
    " Return the ':ls' output as one string.  Call it with ':silent'
    " to suppress the normal ls output.
    let buffer_list = ''
    redir =>> buffer_list
    redir END
    return buffer_list

fun! IsLocationListBuffer()
    " Return 1 if the current buffer contains a location list.
    " Return 0 otherwise.
    if &ft != 'qf'
        return 0

    silent let buffer_list = GetBufferListOutputAsOneString()

    let l:quickfix_match = matchlist(buffer_list,
                \ '\n\s*\(\d\+\)[^\n]*Quickfix List')
    if empty(l:quickfix_match)
        " no match and ft==qf: current buffer contains a location list
        return 1
    let quickfix_bufnr = l:quickfix_match[1]
    return quickfix_bufnr == bufnr('%') ? 0 : 1

Saturday, July 6, 2013

Conflict Resolution with Vim Conflict Slides

The Conflict Slides Vim plugin allows you to lock to a conflict range and replace that whole range with individual sections inside the conflict markers.  This has several advantages.  For example, you are independent from how the other buffers are named and because modifications to the buffer are fully controlled, you gain some safety while exploring the conflict.

I recently added display helper functions that allow you to toggle the diff option in selected windows.  That in turn makes it possible to see 2-way diffs at different points of the conflict resolution process.  I am going to show you how such a procedure could look like.

There is more documentation on the Conflict Slides frontpage and the functions are documented in more detail in the plugin file.

The tutorial part focuses on Git somewhat, but as long as you have the same conflict markers, this should work with any VCS.


I would recommend to turn the folding off in the post-lock callback to mitigate jolting when changing content.  You can turn the folding back on with the argument 'on' instead of 'off', and the folding can be toggled with the argument 'toggle'.

fun! g:conflict_slides_post_lock_callback()
    call CS_DiffChangeFolding('off')

If your VCS offers to insert the base content into the markers, you should consider to turn that on. Otherwise the base slide is not available.  In Git you can add the base content with the following command:

git config merge.conflictstyle diff3

Generally, I am using three diff windows with ours, dest, theirs content.  That is the setup provided by vim-fugitive.  You should really check out that plugin, if you haven't done so already.  It is tremendously easy to find the commit for a line in a conflict (among many other things).

Example Mappings

Here are some mappings that extend the default mappings from Conflict Slides (CS) a bit.  You can use this as a template for your own mappings.  The ^[1-3] mappings do essentially turn the diff on in all 3 windows, except in the given window where the diff is toggled.  This allows you to see a two-way diff for any combination of the three windows, as well as return to a three-way diff.

nnoremap <silent> ^l :call CS_LockNextConflict('lock-current')<CR>zz
nnoremap <silent> ^n :call CS_LockNextConflict()<CR>zz
nnoremap <silent> ^N :call CS_LockNextConflict('backward')<CR>zz

nnoremap <silent> ^u :call CS_ReleaseLockedConflict()<Bar>
                     \ call CS_DiffChangeFolding('on')<CR>

nnoremap <silent> ^f :call CS_DiffChangeFolding('toggle')<CR>zz

nnoremap <silent> ^1 :call CS_DiffSwitch3Toggle(1)<CR>
nnoremap <silent> ^2 :call CS_DiffSwitch3Toggle(2)<CR>
nnoremap <silent> ^3 :call CS_DiffSwitch3Toggle(3)<CR>

You can simply define a g:conflictslides_locked_mappings dictionary in your Vim configuration to override the one from CS.  The top section in the following snippet is the same as in the plugin.  Below that are a few mappings that allow you to insert a slide while turning diff on/off in selected windows.  The first two of those mappings are useful immediately after you locked to a conflict to explore the changes in each parent.

let g:conflictslides_locked_mappings = {
        \ 'b' : ":call CS_ModifyConflictContent('base')<CR>",
        \ 'B' : ":call CS_ModifyConflictContent('base', 'append')<CR>",
        \ 'o' : ":call CS_ModifyConflictContent('ours')<CR>",
        \ 'O' : ":call CS_ModifyConflictContent('ours', 'append')<CR>",
        \ 't' : ":call CS_ModifyConflictContent('theirs')<CR>",
        \ 'T' : ":call CS_ModifyConflictContent('theirs', 'append')<CR>",
        \ 'a' : ":call CS_ModifyConflictContent('ours theirs')<CR>",
        \ 'A' : ":call CS_ModifyConflictContent('theirs ours')<CR>",
        \ 'f' : ":call CS_ModifyConflictContent('forward')<CR>",
        \ 'r' : ":call CS_ModifyConflictContent('reverse')<CR>",
        \ 'F' : ":call CS_ModifyConflictContent('forward-nobase')<CR>",
        \ 'R' : ":call CS_ModifyConflictContent('reverse-nobase')<CR>",
        \ 'e' : ":call CS_ReleaseLockedConflict()<CR>",
        \ 'q' : ":call CS_ModifyConflictContent('forward')<Bar>"
        \               . "call CS_ReleaseLockedConflict()<CR>",
        \ '<CR>' : ":call CS_LockNextConflict()<CR>",
        \ '<BS>' : ":call CS_LockNextConflict('restore-conflict')<CR>",
        \ 'V' : ":call CS_SelectCurrentConflictRange(0)<CR>",
        \ 'v' : ":call CS_SelectCurrentConflictRange(500)<CR>",
        \ '<space>b'        : ":call CS_ModifyConflictContent('base')<Bar>"
        \                           . "call CS_DiffSwitch3Off(3)<CR>",
        \ '<space><space>b' : ":call CS_ModifyConflictContent('base')<Bar>"
        \                           . "call CS_DiffSwitch3Off(1)<CR>",
        \ '<space>o'        : ":call CS_ModifyConflictContent('ours')<Bar>"
        \                           . "call CS_DiffSwitch3Off(1)<CR>",
        \ '<space>t'        : ":call CS_ModifyConflictContent('theirs')<Bar>"
        \                           . "call CS_DiffSwitch3Off(3)<CR>",
        \ '<space>a'        : ":call CS_DiffSwitch3AllOn()<CR>",
        \ '<space>1'        : ":call CS_DiffSwitch3Toggle(1)<CR>",
        \ '<space>2'        : ":call CS_DiffSwitch3Toggle(2)<CR>",
        \ '<space>3'        : ":call CS_DiffSwitch3Toggle(3)<CR>",
        \ '<space>f'        : ":call CS_DiffChangeFolding('on')<CR>zz",
        \ '<space>F'        : ":call CS_DiffChangeFolding('off')<CR>zz",
        \ }

There is nothing in there that cannot be done with the global mappings from above.  So, it's just a template in case you want to create more convenient mappings with the mechanism provided by CS.  You may need to adapt the window number in the calls if you use a different setup.

Prepare for Conflict

There is this awesome merge resolution boot camp with solutions and I've picked one merge to demonstrate how to resolve a conflict.  You can follow if you like.  This is the commit:

736a2dd Merge tag 'virtio-next-for-linus' of git://...

First, prepare two tags and a branch.

git tag ExpDest 736a2dd2571ac56b11ed95a7814d838d5311be04
git tag ExpTheirs ExpDest^2
git checkout -b ExpMaster ExpDest^1

Inspect the result with this command:

git log --graph --oneline --decorate --simplify-by-decoration -3 ExpDest

It should look similar to this:

Now, let's try to merge.

git merge --log ExpTheirs

This should result in a conflict with several unmerged files.


If you use Git without Fugitive, you can start the setup described above by adding the following section to your ~/.gitconfig file,

[mergetool "gv3"]
cmd = gvim -f -d  \"$LOCAL\" \"$MERGED\" \"$REMOTE\" \"+2wincmd w\"

and start that mergetool for drivers/char/virtio_console.c like so:

git mergetool -t gv3 drivers/char/virtio_console.c

By pressing ^l you can lock to the one and only conflict in this file.  This will produce a beautiful rainbow of colors.

The initial 3-way diff of the conflict

Now you can use the <space>b and <space><space>b mappings to see two-way diffs between the base slide and ours/theirs respectively.  (The same can be accomplished with b and then ^1 or ^3)

Difference ours/base

Difference base/theirs

Aha!  The conditional branch has been wrapped in a spinlock on our side and a very important call has been replaced by an even more important call on their side.  If you are using Fugitive you can very conveniently open a blame view for their commit and try to find out more about that change.

Fugitive's blame view started from their modification

To resolve it you can take our side with <space>o, and append their side with T.

Spotting the surplus line in the difference dest/theirs

Because this is a two-way diff with theirs, the wrong call can be easily spotted.  Let's unlock with e (for edit) and delete the wrong line.  You can visualize the changes on each side by using the mappings ^1 and ^3.

Resolved difference ours/dest
Resolved difference dest/theirs

Resolve more

If you would like to resolve some more conflicts yourself, search the repository with the following command:

git  log  --merges  --grep Conflicts  master

Monday, May 20, 2013

Powerful insert mode completion configuration for Vim paired with AutoComplPop

Combine results of multiple sources for completion into one menu that pops up automatically. To keep the noise low you can add to an open Popup Menu at different leading word lengths and you will be able to complete matches with the same key (default: Tab) in forward and backward menus (e.g.: <C-P>).

The most interesting part is the localcomplete#localMatches function which allows you to scan an area around the current line (up to the whole buffer) and configure how to sort matches. My favorite sort order starts at the current line and alternates matches from above and below. You can combine local completion with omni completions like RopeOmni for Python.

Local, Rope, and all-buffers completion combined

Now a little disclaimer. This is partially hacking and you might have to wrestle with some quirks. Vim's Popup Menu and completion functions aren't easily accessible. However, I mostly got what I wanted and it is actually a considerable improvement to my workflow. Give it a try.  :D


Generally the scheme in all this is that there are never items preselected in the menu. That is also the most useful way to compare omni completion methods. In addition to what follows, you can still configure completion methods and compare results through the explicit triggers <C-X><C-U> and <C-X><C-O>.
autocmd FileType python setlocal omnifunc=RopeOmni
autocmd FileType python setlocal completefunc=jedi#complete

Beware of the value longest in the completeopt Vim setting. I don't use it, but I've noticed that it conflicts with the expected behavior.

If you don't know much about insert mode completion, reading *ins-completion* in the Vim docs will help you understand some of the terms (and restrictions of the implementation) that follow.

AutoComplPop is used to trigger the Popup Menu. I did not look much at YouCompleteMe. ACP looked like it was what I needed. In particular, you can pretty much configure exactly what will happen for each individual file type. Nevertheless, you will probably be able to use the completion functions with YCM.


The first thing you need is a patched AutoComplPop. Please read the commit messages on top of the fork-point to see what changed. I also updated the documentation in the fork.

The most notable change is that you can configure the keys used to browse the menu. Furthermore the colors and mappings are different for reverse (e.g. <C-P>) and forward menus. This gives you a clue where to look for the most likely completion, and you can always use the same key for completion. It is possible to turn off the mapping switch, and you can configure the same color for both directions.

When you use the switch (the default) you need to include a menu adaption into insert mode completion mappings. I tried to avoid it but when done inside ACP, the previous color was always flickering through.

Here some examples. The number in the call is the direction. One is forward, and two is backward.
" Filename completion
inoremap <C-F> <C-R>=acp#pum_color_and_map_adaptions(1)<CR><C-X><C-F><C-P>

" Spelling corrections
inoremap <C-S> <C-R>=acp#pum_color_and_map_adaptions(1)<CR><C-X><C-S><C-P>

" Omni completion
inoremap <C-O> <C-R>=acp#pum_color_and_map_adaptions(1)<CR><C-X><C-O><C-P>

" User defined completion
inoremap <C-U> <C-R>=acp#pum_color_and_map_adaptions(1)<CR><C-X><C-U><C-P>

" Switch to keyword completion backwards
inoremap <C-L>  <C-R>=acp#pum_color_and_map_adaptions(2)
            \ <CR><C-R>=pumvisible()
            \ ? "\<lt>C-E>\<lt>C-P>\<lt>C-N>"
            \ : "\<lt>C-P>"<CR>

" Close the pum with <Return>.  Otherwise, it would restore the old value after
" manual completion.
inoremap <Return> <C-R>=pumvisible()
            \ ? "\<lt>C-X>\<lt>Return>"
            \ : "\<lt>Return>"<CR>

" Allow to exit the pum with Up/Down
inoremap <Up>  <C-R>=pumvisible() ? "\<lt>C-X>\<lt>Up>" : "\<lt>Up>"<CR>
inoremap <Down>  <C-R>=pumvisible() ? "\<lt>C-X>\<lt>Down>" : "\<lt>Down>"<CR>

A few hints for the ACP configuration.  See the ACP doc for the actual configuration help.
" Remove dictionary lookup from the Vim keyword completion.  It did always
" complete the first match for me.  If you edit files with tags you might
" want to add those.
let g:acp_completeOption = '.,w,b'

" How keyword completion is triggered.  Usually you want variables before
" the current line.  ... Unless you write a file bottom up, that is.
let g:acp_behaviorKeywordCommand = "\<C-P>"

" This adds the local-completion function before all other completions
" (snipmate, keyword, file) in the g:acp_behavior records set up by ACP.
let g:acp_behaviorUserDefinedFunction = 'localcomplete#localMatches'
let g:acp_behaviorUserDefinedMeets = 'acp#meetsForKeyword'

And templates for some new options. All given values are the defaults
" Highlights to use for the pop up menu.
let g:acp_colorForward = 'AutoComplPopColorDefaultForward'
let g:acp_colorReverse = 'AutoComplPopColorDefaultReverse'

" Specify the keys used to select entries.  These create insert mode
" mappings like you know them from the Vim documentation.
let g:acp_nextItemMapping = ['<TAB>', '\<lt>TAB>']
let g:acp_previousItemMapping = ['<S-TAB>', '\<lt>S-TAB>']

" Turn the reverse mapping switch on/off:
let g:acp_reverseMappingInReverseMenu = 1

" Don't preselect the first item.  Use the same key to select and to browse
let g:acp_autoselectFirstCompletion = 0


The localcomplete plugin tries to replicate some built-in Vim completions so that they can be combined and configured.

Of interest are essentially the following two files:


Here you find three completion functions:

This is the function mentioned above. It searches through the configured area of the current buffer only. All of the configuration in the first section of the source file applies to it. Please read through that section. There is no documentation file.

This function searches through all buffers. It respects the case and keyword-char configuration. It's just an unsophisticated replacement for the functionality provided by Vim.

Search the dictionary configured in Vim's 'dictionary' setting for matches. All matches are done case-sensitively, because otherwise all the names in the dictionary would come first. It is a very simple function. The dictionary has to be utf-8 encoded.

All three functions can have individual minimum leading word lengths configured after which they start to produce results. This can be used to keep the number of matches in the menu reasonable. For example, you might want to add local-matches after two characters, but not the masses of dictionary matches at that point. The variables for that are described in the second section of this file.


This is a pretty rough and hardcoded module for demonstration purposes.

It contains a function to combine multiple completion functions into one. ACP can be configured to try several completion functions in a row until one generates results, but it is much more useful to have multiple results combined into one menu. Here is a client function that calls it with the three completion functions from above.
function combinerEXP#completeCombinerTextish(findstart, keyword_base)
    let l:all_completers = [
                \ 'localcomplete#localMatches',
                \ 'localcomplete#allBufferMatches',
                \ 'localcomplete#dictMatches',
                \ ]
    return combinerEXP#completeCombinerABSTRACT(
                \ a:findstart,
                \ a:keyword_base,
                \ l:all_completers
                \ 0)
The zero is an index into l:all_completers to select the function used in the findstart mode. Because of the way completion functions interact with Vim, there can be only one findstart column that is shared by all combined completion functions.

There is a Python combiner that addresses requirements when combining other completion functions with RopeOmni from python-mode. If you are interested in rope completions, you should take a closer look at this file. Here is an example combiner:
function combinerEXP#completeCombinerPython(findstart, keyword_base)
    let l:before_rope = []
    let l:after_rope = [
                \ 'localcomplete#localMatches',
                \ 'localcomplete#allBufferMatches',
                \ ]
    return combinerEXP#ropeCombiner(
                \ a:findstart,
                \ a:keyword_base,
                \ l:before_rope,
                \ l:after_rope,
                \ 0)
It makes sense to prepend local-completion with a very slim search area. The all-buffers completion searches the current buffer first, and so you can fall back to that completion for all matches in the current buffer. More on that below.

Configuring Content

Alright. Now we can connect those two plugins and get a lot of opportunities for configuration. You can specify the completion for each &filetype individually by filling in the g:acp_behavior variable.

For each filetype you can configure multiple entries, that are tried in turn until one generates matches. The following example creates the entry for all files that don't have a filetype specific entry (e.g. an 'xml' key instead of the '*'). This one tries the text combiner first, and falls back to keyword completion in reverse mode.
if ! exists("g:acp_behavior")
      let g:acp_behavior = {
        \ '*' : [
        \     {
        \       'completefunc': 'combinerEXP#completeCombinerTextish',
        \       'command': "\<C-X>\<C-U>",
        \       'meets': 'acp#meetsForKeyword',
        \       'repeat': 1,
        \     },
        \     {
        \       'command': "\<C-P>",
        \       'meets': 'acp#meetsForKeyword',
        \       'repeat': 0,
        \     },
        \ ],

Here are the two completions side by side.

Localcomplete and standard keyword completion side-by-side (the sorting is different)

Now let's configure at what point which entries appear.

It is important to note that ACP has a minimum leading word requirement, as well as the completion functions from localcomplete itself. ACP won't open the menu unless the minimum length is reached. These values are used in the corresponding 'meets' functions. You can see one of those above (acp#meetsForKeyword). You can write your own meets functions. They are called with the line content before the cursor position and are supposed to return non-zero if the requirements for opening the Popup Menu are met.
" count of chars required to start keyword completion
let g:acp_behaviorKeywordLength = 2

If you have multiple completion functions from the localcomplete plugin combined into one, you can configure individual minimal lengths of the word before the cursor. There is a catch, however. The Popup menu has to be updated to display the new content. You can do that after every typed character, but you'll soon notice some lag when combined with expensive omni completions. Therefore it is possible to configure particular word-lengths at which the menu should be updated. You can simply use the minimum length variables for that:
let g:localcomplete#LocalMinPrefixLength = 1
let g:localcomplete#AllBuffersMinPrefixLength = 1
let g:localcomplete#DictMinPrefixLength = 5

let g:acp_refeed_checkpoints = [
            \ g:localcomplete#LocalMinPrefixLength,
            \ g:localcomplete#AllBuffersMinPrefixLength,
            \ g:localcomplete#DictMinPrefixLength]

" Beware. Probably expensive (flickering)
"let g:acp_refeed_after_every_char = 0
In the same way you can transmit the additional keyword configuration to ACP:
let g:localcomplete#AdditionalKeywordChars = '-'
let g:acp_keyword_chars_for_checkpoint =
            \ g:localcomplete#AdditionalKeywordChars

Python Configuration

Here is part of the Python configuration I am playing with right now. A very slim area is searched before RopeOmni has its turn. I think it has a better chance of putting local variables in the right order.
let b:LocalCompleteLinesAboveToSearchCount = 15
let b:LocalCompleteLinesBelowToSearchCount = 10
The combiner for it looks like this:
function! g:localFirstPythonCombiner(findstart, keyword_base)
    let l:before_rope = [
                \ 'localcomplete#localMatches',
                \ ]
    let l:after_rope = [
                \ 'localcomplete#allBufferMatches',
                \ ]
    return combinerEXP#ropeCombiner(
                \ a:findstart,
                \ a:keyword_base,
                \ l:before_rope,
                \ l:after_rope,
                \ 0)
We need to tell ACP when to add entries to the menu. This is really lagging when done after every typed character.
" Minimum leading word lengths
let b:LocalCompleteLocalMinPrefixLength = 3
let b:LocalCompleteAllBuffersMinPrefixLength = 5

" Restart omni completion after these word lengths.
let b:acp_refeed_checkpoints = [
            \ b:LocalCompleteLocalMinPrefixLength,
            \ b:LocalCompleteAllBuffersMinPrefixLength,
            \ ]

" Preemptively override global values
let b:acp_refeed_after_every_char = 0
let b:LocalCompleteAdditionalKeywordChars = ''
let b:acp_keyword_chars_for_checkpoint =
            \ b:LocalCompleteAdditionalKeywordChars
In the RopeOmni version that I used is a bug where keywords override object attributes. So, for example, when you add local-matches after two characters, and an object attribute starts with either of 'is in as', all initially good rope results are removed from the menu. Therefore I would try to avoid using only small checkpoints without a bigger one.

You can add all this in a ftplugin/python.vim file so that it is nicely grouped. Here is the ACP configuration for Python that uses the combiner from above:
let g:acp_behavior['python'] = [
        \     {
        \       'command': "\<C-X>\<C-U>",
        \       'completefunc': 'g:localFirstPythonCombiner',
        \       'meets': 'acp#meetsForPythonOmni',
        \       'repeat': 1,
        \     },
The repeat dictionary key is important for omni-completions that can give you info about, for example, function parameters after typing an opening parenthesis. It tries to restart the completion menu when it closes. If that doesn't work it's probably the fault of the meets function. Take a look at the ones defined in ACP if you want to write your own.

The startup word length for this particular meets function can be set with the following variable. I've added dots and opening parentheses to that function so that a setting of 1 will give you maximal info about objects while not always bothering you.
let g:acp_behaviorPythonOmniLength = 1

Display function parameters and default arguments after an opening parenthesis

RopeOmni needs a syntactically correct file. By default you see that the cursor becomes red when RopeOmni errors are caught in ropeCombiner. Anyway, the other completions can jump in. And with the error silencing done in the combiner it is actually a quiet good and fun completion.

Prefer surrounding lines over Rope's object attributes


Snipmate has the Tab key hardcoded, but you can easily find and change it in after/plugin/snipMate.vim.

This is the standard entry that gets normally added if you don't override entries in g:acp_behavior:
{ 'meets': 'acp#meetsForSnipmate',
\ 'completefunc': 'acp#completeSnipmate',
\ 'onPopupClose': 'acp#onPopupCloseSnipmate',
\ 'repeat': 0, 
\ 'command': "\<C-X>\<C-U>"},
I would recommend to put it first and I think it works best when the length limit is lower than the others:
let g:acp_behaviorSnipmateLength = 1
Only triggers that start with an uppercase character end up in the Popup Menu and you have to explicitly select the completion for it to expand (default: <C-Y>)


Now the sad part. ACP is a strange beast. There is no real access to Vim's PopUp Menu and this is all done by streaming a bunch of key presses to Vim. I don't understand everything that is happening sometimes, but here a few tips:

For the localcomplete functions, always set repeat to 1 (or let it unspecified). If the updates at checkpoints don't work, this might be the cause.  However, the completion is quiet persistent in that case.  If you don't like that, try to increase or decrease the checkpoint by one.
Another alternative in this case is to create mappings that turn the auto-completion on/off:
map <S-F5> :AcpDisable<CR>
map <F5>   :AcpEnable<CR>

If you want to trigger built-in commands, I would recommend to set repeat to 0 explicitly. Otherwise, if you use checkpoints, the completion is undone to the point where the PopUp Menu was opened.

ACP prepares individual records for the following filetypes: ruby, python, perl, xml, html, xhtml, css, javascript, coffee, ls.
This can be a bit surprising if you edit any of such files infrequently. The most failsafe way to fall back to the default behavior is probably to add an autocommand like this:
autocmd FileType ruby call remove(g:acp_behavior, "ruby")

I hope I didn't break too much for you, but I am done experimenting with it. It would be awesome if there could be a more sophisticated access to Vim's PopUp Menu. That would make things a lot easier, I guess.

But don't let all this discourage you. I am actually quiet happy with it. In particular in Python.

Give <C-P> some relief

Happy playing around with the word lengths. There must be some optimal adjustment.

(The snippets from the source code are under LGPLv3)

Sunday, April 21, 2013

Little helpers

Do you sometimes prepare a commit series, look through it again, and find some small changes like doc improvements or name changes that you need to squash into the last commit to that file?  This little snippet creates a fixup commit for each dirty file tracked by Git (it predates the --fixup option somewhat :).


err() {
  echo "Error: $@" >&2

git rev-parse --is-inside-work-tree &>/dev/null || err "Not inside a work tree"
[ -z "$(git diff --staged --name-only)" ] || err "Stage is not empty"

for dirtyfile in $(git diff --name-only) ; do
  commit_message=$(git log --oneline -- "$dirtyfile" | head -1 | sed 's/^\w* //g')
  git commit -m "fixup! ${commit_message}" -- "${dirtyfile}"

These are the sort of tools that are unlikely to be shipped with a VCS, but would fit nicely into a workbench tool-shed.  Of course, there can be much more sophisticated tools as well as whole toolsets describing workflows.

Thursday, August 30, 2012

A Shared Project Workspace

I've been reading a lot about agile procedures lately and I totally agree that colocation is important and a prerequisite. However, a form of online Activities (in the KDE sense) could be used to bring developers interested in an Open Source project closer to that project without boarding a plane.

There could be widgets shared online that would form the project workspace visible to anyone.  Beside displaying project info, widgets could offer to get directly involved with a project.  Remote developers could use those widgets for their own version of a project specific Activity and add other tools like editors.

Ideally, whole Activties could be distributed through the Project Repositories of a workbench and modifications could be layered upon that.

Saturday, October 22, 2011

Get, modify, and build the sources of an RPM package -- two of the three steps are simple


After years of using source based distributions, I finally realized that binary distributions aren't that binary after all. However, the rpmbuild system seems to be optimized for, ... , well, creating RPMs. In the following, I attempt to facilitate inspecting, and building SRPM packages for people who don't work with RPMs on a daily basis, but still want to see and tweak the bits that operate their system.

One thing that struck me as odd, is that by default all SRPM operations happen in one directory, ~/rpmbuild by default on Fedora. I'd really prefer to have all operations in a package specific directory. I'll demonstrate how to accomplish that and how to get to a modified build in three steps.

A little disclaimer: I did not work very much with RPM, but really wanted to easily get to the sources. Some things in here might be wrong or maybe there exist more convenient steps. I'm looking forward to be corrected in the comments.

Oh, and uninspired as I am, I have just prepended the character d to rpm. Maybe you find a word for which that character stands.

The Proceedings -- An Example

Let's say you haven't been working with RPMs for several weeks, but you suspect a bug in the editor Nano and want to take a look at it. You only need to remember and execute a single command:

drpm-get-source nano

This will download the Source-RPM for the currently installed version of Nano and create an SRPM build environment in a version-specific subdirectory of ~/drpm/, for example ~/drpm/nano-2.2.4-1.fc14,

Next, you browse the sources and modify some bits. This is the one step that is not trivial, but as it is the fun part, I leave all the things you do at this point up to you.

Finally, from within the build environment created in the first step, you simply execute the following command to build the result:


These three steps are all I want to do most of the time. If the result is useful enough to enter the production system it can be installed to /usr/local, or a binary RPM can be created. The important point is that it is possible to easily get, modify, and compile the sources in a separate build directory.

Now, let's walk through the parts...

Unpacking The Sources

The script drpm-get-source takes a package name as argument, obtains the associated SRPM and prepares it in a package specific sub-directory of ~/drpm/.

The script uses yumdownloader to obtain the URL of the SRPM. It is possible to obtain the URL by other means and specify it as argument to the option --url. Also, it might be possible that there are missing dependencies to build the SRPM. In that case the script will exit with an error and display the appropriate yum-builddep command, ready to be pasted into a root console. You might want to adapt the displayed message to your distribution.

Here the Python script drpm-get-source:

#!/usr/bin/env python

from __future__ import print_function

import argparse
import collections
import os
import subprocess
import sys
import urllib


# The desfault location for the --destdir option
# The name of the file that marks the root directory of an unpacked srpm.
PKGROOT_MARKER_FILE = ".drpm-root"



# Capture location info of a package
PackageInfo = collections.namedtuple("PackageInfo",
        ['srpm_url', 'srpm_path', 'pkg_topdir'])

def error(msg, exit_status=1):
    Generate a colorful error message and exit with the given exit_status.
    sys.stderr.write("\x1b[1;37;41m%s\x1b[0m\n" % msg)

def execute_shell_command(shell_command):
    Execute shell_command and return the exit code and the combined output of
    stderr and stdout.
    p = subprocess.Popen(shell_command, shell=True, stdout=subprocess.PIPE,
    output = p.stdout.read().decode().strip()
    retval = p.wait()
    return (retval, output)

def parse_cmdline():
    Return a Namespace object of the parsed command line.

    The destdir attribute will be absolute and a home directory reference in it
    will be expanded.
    desc="Download a source rpm and unpack it."
    aparser = argparse.ArgumentParser(description=desc)
            help='The package to obtain the sources for (default) or an '
                    'URL to a SRPM to use if the "--url" option is specified.')
    aparser.add_argument('-d', '--destdir', default=DEFAULT_DESTDIR,
            help='Base directory where the sources will be unpacked. '
                    '(Default:%(default)s) A subdirectory named like '
                    'the package will be created there.')
    aparser.add_argument('-u', '--url', action='store_true', default=False,
            help='The argument is a SRPM URL obtained by other means.')
    cmdline = aparser.parse_args()
    cmdline.destdir = os.path.abspath(os.path.expanduser(cmdline.destdir))
    return cmdline

def get_source_url(pkg):
    Use yumdownloader to obtain and return the SRPM URL of the specified
    url_cmd = "yumdownloader --source --urls '%s'" % pkg
    retval, output= execute_shell_command(url_cmd)
    if retval:
        error("%s\n\n%s\n\nGetting the url with yumdownloader failed."
                % (output, url_cmd))
    url = output.split('\n')[-1].strip()
    if not url:
        error("%s\n\n%s\n\nGetting the url failed. Last line of "
                "output is empty." % (url_cmd, output))
    print("Got URL: " + url)
    return url

def get_package_info(url, destdir):
    Return a PackageInfo instance for the given arguments.
    STRIP_SUFFIX = '.src.rpm'
    srpm_filename = os.path.basename(url)
    if srpm_filename.endswith(STRIP_SUFFIX):
        pkg_name = srpm_filename[:-len(STRIP_SUFFIX)]
        error("url '%s' does not end with '%s'" % (url, STRIP_SUFFIX))
    pkg_topdir = os.path.join(destdir, pkg_name)
    srpm_path = os.path.join(pkg_topdir, 'SRPMS', srpm_filename)
    return PackageInfo(srpm_url=url, srpm_path=srpm_path, pkg_topdir=pkg_topdir)

def create_dir_structure(pkg_info):
    Create the directory structure of an SRPM build environment.
    pkg_topdir = pkg_info.pkg_topdir
    if os.path.exists(pkg_topdir):
        error("Destination exists: " + pkg_topdir)
    print("Creating directory structure at: " + pkg_topdir)
    for subdir in [''] + RPM_DIRS:
        os.makedirs(os.path.join(pkg_topdir, subdir))
    with open(os.path.join(pkg_topdir, PKGROOT_MARKER_FILE), "w") as fw:
        fw.write("this file marks an unpacked srpm root")

def download_srpm(pkg_info):
    Download the SRPM to its destined location
    urllib.urlretrieve(pkg_info.srpm_url, pkg_info.srpm_path)

def find_specfile(pkg_info):
    Return the one and only spec file or exit with an error.
    specdir = os.path.join(pkg_info.pkg_topdir, 'SPECS')
    entries = [e for e in os.listdir(specdir) if e.endswith('.spec')]
    if len(entries) != 1:
        error("Not exactly one spec file found\nrpmbuild --define "
                "'_topdir %s' -bp --target=$(uname -m) " %
    specfile = os.path.join(specdir, entries[0])
    return specfile

def unpack_srpm(pkg_info):
    Execute an rpm command that unpacks the SRPM into the current build
    unpack_cmd = ("rpm --define '_topdir %s' -Uvh %s" %
            (pkg_info.pkg_topdir, pkg_info.srpm_path))
    if os.system(unpack_cmd):
        error("%s\nUnpacking failed: %s" %
                (unpack_cmd, pkg_info.pkg_topdir))

def prep_srpm(pkg_info, specfile):
    Execute an rpmbuild command that prepares the sources for a build.
    # This will probably require root rights.  Just form the command and make
    # it part of an error message so that it can be pasted into a root console.
    builddep_cmd = 'yum-builddep %s' % pkg_info.srpm_path
    prep_cmd = ("rpmbuild --define '_topdir %s' -bp %s" %
            (pkg_info.pkg_topdir, specfile))
    if os.system(prep_cmd):
        error("%s\n%s\nPrepping failed: %s" % (builddep_cmd, prep_cmd,

def main():

    cmdline = parse_cmdline()

    if cmdline.url:
        url = cmdline.pkg
        url = get_source_url(cmdline.pkg)

    pkg_info = get_package_info(url, cmdline.destdir)




    specfile = find_specfile(pkg_info)

    prep_srpm(pkg_info, specfile)

    print("\nSources are at:\n" + pkg_info.pkg_topdir)

if __name__ == '__main__':

Executing rpm/rpmbuild Commands For The Current Package

When rpm and rpmbuild operate on Source-RPMs they always assume the operations should take place in the SRPM build environment location specified by the macro _toplevel. To use a separate location for each source package, the wrapper script drpm-shared, determines the root directory of the unpacked SRPM and adds an appropriate macro definition to the command line. Therefore, when you execute an rpm or rpmbuild command with that wrapper it will operate on the SRPM build environment that belongs to the current working directory.

drpm-shared should be called through any of the following symbolic links, which determine the behavior of the script:


The first two wrap rpm and rpmbuild respectively, and the last one additionally appends the path to the spec file to a rpmbuild command line. To install drpm-shared, save it into a directory in your $PATH and inside that directory, create the necessary symbolic links with the following commands:

ln -s drpm-shared drpm
ln -s drpm-shared drpmbuild
ln -s drpm-shared drpmbuildspec

This is the shell script drpm-shared:


# Config
# ======

# The name of the file that marks the root directory of an unpacked srpm.

# Internal
# ========

PROG="$(basename ${0})"

usage() {
    echo "USAGE: ${PROG} [-h|--help] <${WRAPPED_COMMAND}-sec-argv>... " >&2
    echo "Find the srpm root (${RPMROOT_MARKER_FILE}) of cwd and "\
            "execute the given ${WRAPPED_COMMAND} command with the "\
            "appropriate _toplevel macro definition."

error() {
    echo "ERROR: $@" >&2
    exit 1

# Print the canonical, absolute path to the current SRPM root directory.  Print
# an empty string if not inside of an RPM package.
__get_rpm_root() {
    local cwd_canon=`pwd -P`
    local lookout_path="${cwd_canon}"
    while [ ! -e "${lookout_path}/${RPMROOT_MARKER_FILE}" ] ; do
        local parent_dir=`dirname "${lookout_path}"`
        if [ "${parent_dir}" = "${lookout_path}" ] ; then
    echo -n ${lookout_path}

# Print the path to the one and only spec file or abort with an error.
__find_spec_path() {
    [ -n "${srpm_root}" ] || error "__find_spec_path: missing srpm_root argument"
    for candidate in "${srpm_root%/}"/SPECS/*.spec ; do
        [ -n "${spec_file}" ] && error "__find_spec_path: more than "\
                "one spec file available -- do not know which one to add" 
    echo -n $spec_file

# Sequence
# ========

case "$PROG" in
    # Call rpmbuild and append the path to the spec-file to the command
    # Call rpmbuild
    # Call rpm
        error "Unknown command name $PROG."

case "$1" in 
        exit 0

# Find the root directory of the unpacked srpm
if [ -z "${srpm_root}" ] ; then
    error "not inside an unpacked rpm package -- cannot "\
            "find \"${RPMROOT_MARKER_FILE}\""

# Set the spec-file argument if requested
if [ -n "${WANT_SPEC_FILE_PATH}" ] ; then
    SPEC_PATH=`__find_spec_path "${srpm_root}"`
    [ $? -eq 0 ] || error "$SPEC_PATH"

# Print and execute the command
echo "${WRAPPED_COMMAND}" --define "_topdir ${srpm_root}" "${@}" "${SPEC_PATH}"
"${WRAPPED_COMMAND}" --define "_topdir ${srpm_root}" "${@}" "${SPEC_PATH}"
exit $?

Fine, but I just want to build it

After unpacking and preparing the sources, an obvious task is to execute the build and install steps without undoing own modifications. That can be done with the following shell function:

drpm-compile () {
    drpmbuildspec -bc --short-circuit && drpmbuildspec -bi --short-circuit

Now, how to integrate it into my system?

The easiest way is to simply install to /usr/local with this command:

drpmbuildspec -bi --short-circuit --buildroot=/usr/local

This, of course, lacks all the advantages of the RPM system. To really integrate your changes into another RPM, you have to get a bit more familiar with RPM. Here are some helpful links:

How to create an RPM package
RPM Guide

Thursday, September 22, 2011

Mutt'n'Gmail: Show me the full thread

The 'Only One Copy of a Mail' scheme in Gmail makes really sense. I like it a lot. There was just one piece missing when I accessed my account with Mutt: Display the full thread of a particular message. Luckily, the full thread is available in the 'All Mail' folder. So, just extract the Message-ID of the current mail, and focus the thread with that id in the 'All Mail' folder.

BTW: I still cannot believe that there is no way to pass a mail through a filter in Mutt. I mean, to pipe a mail to a script and capture the output in a variable. If you know how to do that, please let me know.

This is the Python script that takes a mail on standard input, and writes the message-id of that mail into a temporary file:

#!/usr/bin/env python

import os, sys, re
from email import message_from_string, message_from_file
from email.header import Header, decode_header

RECORDFILE = "/tmp/mutt-store-for-message-id"
set my_message_id='%s'

class MessageIdException(Exception):
    The base exception of this module.

class HeaderNotAvailable(MessageIdException):
    catch this if the you are looking for is optional.

def get_header(mail, header):
    Return the requested header as one decoded string or throw
    header_raw = mail.get(header)
    if not header_raw:
        raise HeaderNotAvailable("looking for header %s in %s"
                % (header, mail.get('Subject')))
    header_parts = []
    for part, charset in decode_header(header_raw):
        header_parts.append(unicode(part, charset or 'ascii', 'replace'))
    return ''.join(header_parts).strip()

def escape_message_id(idstring):
    Return an escaped version of the message-id string.

    Some characters need to be escaped to not be evaluated in mutt.
    for c in CHARS_TO_ESCAPE:
        idstring = idstring.replace(c, '\\' + c)
    return idstring

def main():
    Read a mail piped on standard input, and write the escaped message-id
    string to a file.
    if os.isatty(file.fileno(sys.stdin)):
        raise MessageIdException("stdin is not a pipe")
    mail = message_from_file(sys.stdin)
    msg_id_header = 'Message-ID'
    raw_message_id = get_header(mail, msg_id_header)
    if not raw_message_id:
        raise MessageIdException("empty message-id")
    esc_message_id = escape_message_id(raw_message_id)
    with open(RECORDFILE, "w") as fw:
        fw.write(MUTTCMD % esc_message_id)

if __name__ == '__main__':

Now, the Mutt macro. You need to fill in the path to where you stored the above script -- called 'muttcmd-print-message-id' in this example:

macro index,pager ,T "\
<enter-command>set my_auto_tag=\$auto_tag auto_tag=no<Enter>\
<enter-command>set my_pipe_decode=\$pipe_decode pipe_decode=no<Enter>\
<enter-command>set my_wait_key=\$wait_key wait_key=no<Enter>\
<enter-command>set wait_key=\$my_wait_key<Enter>\
<enter-command>set pipe_decode=\$my_pipe_decode<Enter>\
<enter-command>set auto_tag=\$my_auto_tag<Enter>\
<enter-command>source /tmp/mutt-store-for-message-id<enter>\
<tag-pattern>~i \$my_message_id<enter>\
<limit>~(~i \$my_message_id)<enter>\